As pre­vi­ously reported, I tried the update to Android 3.2 on my Tab and was not happy about the new EMail app. At the week­end I had a lit­tle bit of time, so I tried to get the Email.apk from Android 3.1 into Android 3.2.

Long story short, I failed.

Tita­ni­um­Backup PRO was restor­ing or hours (the option to migrate from a dif­fer­ent ROM ver­sion was enabled) until I killed the app, and it did not get any­where (I just emailed their sup­port if I did some­thing com­pletely stu­pid, or of this is a bug in TB). And a copy by hand into /system/apps did not work (app fails to start).

Ideas welcome.

Recently we had a strange per­for­mance prob­lem at work. A web appli­ca­tion was hav­ing slow response times from time to time and users com­plained. We did not see an uncom­mon CPU/mem/swap usage on any involved machine. I gen­er­ated heat-maps from per­for­mance mea­sure­ments and there where no obvi­ous traces of slow behav­ior. We did not find any rea­son why the appli­ca­tion should be slow for clients, but obvi­ously it was.

Then some­one men­tioned two recent apache DoS prob­lems. Num­ber one — the cookie hash issue — did not seem to be the cause, we did not see a huge CPU or mem­ory con­sump­tion which we would expect to see with such an attack. The sec­ond one — the slow reads prob­lem (no max con­nec­tion dura­tion time­out in apache, can be exploited by a small receive win­dow for TCP) — looked like it could be an issue. The slow read DoS prob­lem can be detected by look­ing at the server-status page.

What you would see on the server-status page are a lot of worker threads in the ‘W’ (write data) state. This is sup­posed to be an indi­ca­tion of slow reads. We did see this.

As our site is behind a reverse proxy with some kind of IDS/IPS fea­ture, we took the reverse proxy out of the pic­ture to get a bet­ter view of who is doing what (we do not have X-Forwarded-For configured).

At this point we noticed still a lot of con­nec­tion in the ‘W’ state from the rev-proxy. This was strange, it was not sup­posed to do this. After restart­ing the rev-proxy (while the clients went directly to the web­servers) we had those ‘W’ entries still in the server-status. This was get­ting really strange. And to add to this, the dura­tion of the ‘W’ state from the rev-proxy tells that this state is active since sev­eral thou­sand sec­onds. Ugh. WTF?

Ok, next step: killing the offend­ers. First I ver­i­fied in the list of con­nec­tions in the server-status (extended-status is acti­vated) that all worker threads with the rev–proxy con­nec­tion of a given PID are in this strange state and no client request is active. Then I killed this par­tic­u­lar PID. I wanted to do this until I do not have those strange con­nec­tions any­more. Unfor­tu­nately I arrived at PIDs which were listed in the server-status (even after a refresh), but not avail­able in the OS. That is bad. Very bad.

So the next step was to move all clients away from one web­server, and then to reboot this web­server com­pletely to be sure the entire sys­tem is in a known good state for future mon­i­tor­ing (the big ham­mer approach).

As we did not know if this strange state was due to some kind of mis-administration of the sys­tem or not, we decided to have the rev-proxy again in front of the web­server and to mon­i­tor the systems.

We sur­vived about one and a half day. After that all worker threads on all web­servers where in this state. DoS. At this point we where sure there was some­thing mali­cious going on (some days later our man­age­ment showed us a mail from a com­pany which offered secu­rity con­sult­ing 2 months before to make sure we do not get hit by a DDoS dur­ing the hol­i­day sea­son… a coincidence?).

Next step, ver­i­fi­ca­tion of miss­ing secu­rity patches (unfor­tu­nately it is not us who decides which patches we apply to the sys­tems). What we noticed is, that the rev-proxy is miss­ing a patch for a DoS prob­lem, and for the web­servers a new fix­pack was sched­uled to be released not far in the future (as of this writ­ing: it is avail­able now).

Since we applied the DoS fix for the rev-proxy, we do not have a prob­lem any­more. This is not really con­clu­sive, as we do not really know if this fixed the prob­lem or if the attacker stopped attack­ing us.

From read­ing what the DoS patch fixes, we would assume we should see some con­tin­u­ous traf­fic going on between the rev-rpoxy and the web­server, but there was noth­ing when we observed the strange state.

We are still not allowed to apply patches as we think we should do, but at least we have a bet­ter mon­i­tor­ing in place to watch out for this par­tic­u­lar prob­lem (acti­vate the extended sta­tus in apache/IHS, look for lines with state ‘W’ and a long dura­tion (col­umn ‘SS’), raise an alert if the dura­tion is higher than the max. possible/expected/desired dura­tion for all pos­si­ble URLs).

The recent Phoronix bench­mark which com­pared a release can­di­date of FreeBSD 9 with Ora­cle Linux Server 6.1 cre­ated a huge dis­cus­sion in the FreeBSD mail­inglists. The rea­son was that some peo­ple think the num­bers pre­sented there give a wrong pic­ture of FreeBSD. Partly because not all bench­mark num­bers are pre­sented in the most promi­nent page (as linked above), but only at a dif­fer­ent place. This gives the impres­sion that FreeBSD is infe­rior in this bench­mark while it just puts the focus (for a rea­son, accord­ing to some peo­ple) on a dif­fer­ent part of the bench­mark (to be more spe­cific, blog­bench is doing disk reads and writes in par­al­lel, FreeBSD gives higher pri­or­ity to writes than to reads, FreeBSD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are pre­sented on the first page). Other com­plaints are that it is told that the default install was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The author of the Phoronix arti­cle par­tic­i­pated in parts of the dis­cus­sion and asked for spe­cific improve­ment sug­ges­tions. A FreeBSD com­mit­ter seems to be already work­ing to get some issues resolved. What I do not like per­son­ally, is that the arti­cle is not updated with a remark that some things pre­sented do not reflect the real­ity and a retest is necessary.

As there was much talk in the thread but not much obvi­ous activ­ity from our side to resolve some issues, I started to improve the FreeBSD wiki page about bench­mark­ing so that we are able to point to it in case some­one wants to bench­mark FreeBSD. Oth­ers already chimed in and improved some things too. It is far from per­fect, some more eyes — and more impor­tantly some more fin­gers which add con­tent — are needed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeBSD mail­inglist so that oth­ers can improve the wiki page).

What we need too, is a wiki page about FreeBSD tun­ing (a first step would be to take the man-page and con­vert it into a wiki page, then to improve it, and then to feed back the changes to the man-page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to improve the sit­u­a­tion. Do not talk, write some­thing. No mat­ter if it is an improve­ment to the bench­mark­ing page, tun­ing advise, or a tool which inspects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­name­Last­name account and ask a FreeBSD comit­ter for write access.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­matic FreeBSD bench­mark­ing. Unfor­tu­nately the author run out of time. The frame­work was able to install a FreeBSD sys­tem on a machine, run some spec­i­fied bench­mark (not much bench­marks where inte­grated), and then install another FreeBSD ver­sion to run the same bench­mark, or to rein­stall the same ver­sion to run another bench­mark. IIRC there was also some DB behind which col­lected the results and maybe there was even some way to com­pare them. It would be nice if some­one could get some time to talk with the author to get the frame­work and set it up some­where, so that we have a con­trolled envi­ron­ment where we can do our own bench­marks in an auto­matic and repeat­able fash­ion with sev­eral FreeBSD versions.

Things I do not like with Android, and what I would expect instead.

The EMail-App

 This is about the nor­mal EMail App on a stock Android 3.1, not about GMail. The con­nec­tion is via IMAP.

• By default the most recent EMail is on top. If I switch to oldest-first order, the old­est EMail is shown when enter­ing a folder. I want to see the most recent one. If I let the most-recent order, I see directly the newest mes­sages, but when I read the old­est unread mes­sage and delete it, the more old mes­sage is dis­played, instead of the more recent one. KMail is doing this bet­ter (it is able to go “up” instead of only “down” to go to the more recent mes­sage when the most recent mail is sorted on top), but in KMail I can not select mul­ti­ple mes­sages as easy as with the nor­mal EMail App (it seems I have to long-tap on a mail and then tell to select the EMail, and just then I can just quickly tab on the very small area on the left of the sub­ject to select more EMails).
• Select­ing mes­sages could also be improved. When I select sev­eral mes­sages, and then — by acci­dent — enter a mes­sage and want to go back, the first “go back” uns­e­lects all mes­sages and the sec­ond “go back” goes back to the folder-view. I would like to go back directly to the folder-view, instead of uns­e­lect­ing all messages.
• The folder-view itself is also not nice. I have a lot of fold­ers. IMAP sub­fold­ers are sep­a­rated by a dot, e.g. FreeBSD.arch and FreeBSD.mul­ti­me­dia are the sub­fold­ers arch and mul­ti­me­dia in the folder FreeBSD. Each desk­top EMail pro­gram I used so far was intel­li­gent enough to cre­ate a folder-hierarchy out of this, with the pos­si­bil­ity to col­lapse the dis­play of all sub­fold­ers of FreeBSD (I have a lot there, not only those 2) into one entry. If I want to field some­thing in my hier­ar­chi­cal folder struc­ture, I have to scroll a long list of fold­ers, instead of just (auto-)opening (dur­ing drag&drop) the cor­rect hierarchy.
• I do not find an option to tell that the App shall have a look at more than the default inbox to look for new mes­sages. If I want to know if there is a new mes­sage in one of the other fold­ers, I must have a look into the folder(s).
• It looks like I can only pro­duce TOFU-replies, I have not found a way to do a proper interleaved-style reply.
• The error mes­sages when set­ting up an out­go­ing (and I assume incom­ing) server are too brief for my taste. As a default it is OK, but there should be an option to enable more verbose/technical error mes­sages for those which are able to under­stand them.
• It does not option­ally save send mails to a spe­cific (con­fig­urable) folder.
• It does not allow to cryp­to­cally sign mes­sages (at the moment I do not really care if it is via S/MIME or via PGP).
• I also would like to tab with two fin­gers, and the text in-between is selected (not only in the EMail-App).

I have a lit­tle prob­lem find­ing a clean solu­tion to the fol­low­ing problem.

A machine with two net­work inter­faces and no default route. The first inter­face gets an IP at boot time and the cor­re­spond­ing sta­tic route is inserted dur­ing boot into the rout­ing table with­out prob­lems. The sec­ond inter­face only gets an IP address when the shared-IP zones on the machine are started, dur­ing boot the inter­face is plumbed but with­out any address. The net­works on those inter­faces are not con­nected and the machine is not a gate­way (this means we have a machine–admin­is­tra­tion net­work and a production-network). The sta­tic routes we want to have for the addresses of the zones are not added to the rout­ing table, because the next hop is not reach­able at the time the routing-setup is done. As soon as the zones are up (and the inter­face gets an IP), a re-run of the routing-setup adds the miss­ing sta­tic routes.

Unfor­tu­nately I can not tell Solaris to keep the sta­tic route even if the next hop is not reach­able ATM (at least I have not found an option to the route com­mand which does this).

One solu­tion to this prob­lem would be to add an address at boot to the inter­face which does not have an address at boot-time ATM (prob­a­bly with the dep­re­cated flag set). The prob­lem is, that this sub­net (/28) has not enough free addresses any­more, so this is not an option.

Another solu­tion is to use a script which re-runs the routing-setup after the zones are started. This is a prag­matic solu­tion, but not a clean solution.

As I under­stand the in.routed man-page in.routed is not an option with the default con­fig, because the machine shall not route between the net­works, and shall not change the rout­ing based upon RIP mes­sages from other machines. Unfor­tu­nately I do not know enough about it to be sure, and I do not get the time to play around with this. I have seen some inter­st­ing options regard­ing this in the man-page, but play­ing around with this and sniff­ing the net­work to see what hap­pens, is not an option ATM. Any­one with a config/tutorial for this “do not broad­cast any­thing, do not accept any­thing from outside”-case (if possible)?