A Solaris 9 brand­ed zone on Solaris 10

Last week we installed a Solaris 9 brand­ed zone. It is easy to do it, but in the end we rec­om­mend­ed to our client to use a real machine with a native Solaris 9 instal­la­tion. There is too much broken:

  • Start scripts (like the one for cron) use the -P 1 option to pkill. This means only kill if the par­ent PID is 1. In a zone this is not valid, and as such cron (and oth­er dae­mons) do not get stopped, respec­tive­ly a check if it is already run­ning at the start fails and cron can be start­ed mul­ti­ple times. There are already some patch­es which get auto­mat­i­cal­ly deployed dur­ing the first boot of the Solaris 9 brand­ed zone, but there are still sev­er­al easy to detect bugs around.
  • SNMP does not work out of the box. A col­league tried to get it run­ning, but he failed. Googling for the error mes­sage shows two hits. One hit is a link to the source, and anoth­er one is a report of a per­son which has the same prob­lem. MAybe we could get net-snmp up and run­ning, but we did­n’t test this, as we want to have our Solaris 9 sys­tems sim­i­lar (few­er changes and spe­cial cas­es for cfengine… it is already not easy to under­stand for some col­leagues as it is).
  • There is no doc­u­men­ta­tion how to han­dle the first set­up auto­mat­i­cal­ly. We think we can han­dle it by extract­ing all files from the Solaris 9 flar we down­loaded from Ora­cle for this instal­la­tion, mod­i­fy­ing the con­tents (e.g. adding a sysid.cfg), and cre­at­ing a flar again. We did not try to do it.

All this we detect­ed in less than half a day of the first play­ing around with it. To us Solaris 9 brand­ed zones are more like alpha or beta qual­i­ty fea­tures. We gave the rec­om­men­da­tion to our boss to tell the client that we do not think this is a pro­duc­tion ready fea­ture, and if the client insists on using a Solaris 9 brand­ed zone we can not real­ly accept to han­dle it with­in the nor­mal SLA.

So far we did not report any of the above prob­lems to Ora­cle. Those prob­lems are direct­ly vis­i­ble if you try to inte­grate a Solaris 9 brand­ed zone into a good enter­prise class envi­ron­ment, so it looks to me as there is no real pro­duc­tion qual­i­ty test­ing done on the side of Ora­cle. To me this means there is no real com­mer­cial inter­est. It also means that there are prob­a­bly a lot of uncov­ered prob­lems, which is a night­mare to han­dle in a pro­duc­tion environment.

ADSL RAM… the­o­ry and real­i­ty do not match

I am now wait­ing since Decem­ber that my ADSL line is switched to the rate adap­tive mode (RAM). The­o­ret­i­cal­ly it is pos­si­ble. Unfor­tu­nate­ly the real­i­ty does not agree to this (yet).

Luck­i­ly I am not a nor­mal cus­tomer, I know a tech­ni­cian which works for my ISP. He could switch the line with­out prob­lems, but the next update of the sys­tem (which hap­pens from time to time) would can­cel this again, as each update “resets” the sta­tus to what is record­ed in the DB. The prob­lem is, that he can not switch my line to RAM in the DB (actu­al­ly it is not him, he is a net­work tech­ni­cian not one of the sales peo­ple with access to the DB-interface). I am not the only cus­tomer where this is not pos­si­ble. So far they where not able to see a pattern.

Cur­rent­ly there are two col­leagues of him, a friend of him and me which he has as good exam­ples where it does not work (there are more, but those are “just” reg­u­lar cus­tomers). We are now his toys, he wants to find out how to con­vince the sys­tem to switch to RAM in those cas­es. This needs a while, as parts of this need to go the offi­cial way until he sees if it works or not.

I am very hap­py that I am not just a nor­mal cus­tomer. This way it is much more trans­par­ent for me.

HeatMaps again…

Today I stum­bled again over some HeatMaps from Bren­dan Gregg (of DTrace-fame). This time it was the PDF of his pre­sen­ta­tion at the LISA 2010 con­fer­ence. It shows nice­ly how he plans to evolve it from a single-machine (like in Ana­lyt­ics for Ora­cle Stor­age prod­ucts) to the cloud. It is a very good overview about what kind of intu­itive per­for­mance visu­al­iza­tion you can do with this.

I would love to see some­thing for FreeB­SD (or oth­er per­for­mance data). Maybe some­one could take the DTrac­eTaz­Tool as a base or hint and write some­thing gener­ic which works for a lot of things…

There are just too much nice and inter­est­ing things out there, and not enough time for all of them.

AQM/ECN in FreeBSD

After read­ing Jim Get­tys inves­ti­ga­tions about the prob­lems cur­rent buffer sizes of net­work equip­ment pro­voke (which may even have impli­ca­tions in the net neu­tral­i­ty debate), I had a look at which active queue man­age­ment (AQM) algo­rithms with or with­out explic­it con­ges­tion noti­fi­ca­tion (ECN) FreeB­SD supports.

It looks like there is not much imple­ment­ed (if the best solu­tion would be imple­ment­ed, it would not mat­ter how much there is, but unfor­tu­nate­ly there is no best solu­tion). Oth­er sys­tems offer more. RED is imple­ment­ed, but even the inventor/researcher of RED thinks the algo­rithm needs some improve­ments (he is in the process of prepar­ing a paper about this, as Jim Get­tys reveals). Blue/SFBlue is not imple­ment­ed (a more turnkey-solution than the cur­rent RED imple­men­ta­tion). PID con­troller (which may or may not be some­thing some­one wants to use in this case… no idea about its pros/cons in this regard, but it is ref­er­enced in the AQM arti­cle on Wikipedia) is also not implemented.

Regard­ing ECN for FreeB­SD you can find more or less no real doc­u­men­ta­tion in the net (at least with a sim­ple “ECN FreeB­SD” search). It is imple­ment­ed for the RED algo­rithm, but as the RED algo­rithm needs some tuning/setup, this is not a turnkey solu­tion. There is a ECN relat­ed sysctl, but I do not have the impres­sion that this is a turnkey-solution which mag­i­cal­ly gen­er­ates ECN mes­sages with­out using dum­mynet for AQM.

From my cur­rent under­stand­ing (but I think I do not know a lot about this top­ic) it looks like AQM is a fea­ture most peo­ple would like to have acti­vat­ed by default (with an appro­pri­ate algo­rithm which does not need tun­ing to pro­duce a good enough result). If this is cor­rect, it is a shame that FreeB­SD does not acti­vate AQM with an algo­rithm which is not bad for most cas­es by default (with the option to change the algo­rithm and to dis­able com­plete­ly). If my under­stand­ing is not cor­rect, I would like to get a hit with the clue bat please.

Why are game console/TV com­pa­nies not imple­ment­ing this?

At the week­end a friend vis­it­ed me. We have not seen since each oth­er since a long time. As we stud­ied both com­put­er sci­ence, parts of our dis­cus­sion where off course tech­nol­o­gy relat­ed. Parts of the dis­cus­sion where about cur­rent TV’s and game con­soles (he par­tic­i­pat­ed in the design of the PS3 CPU, so he is well aware about the tech­ni­cal lim­i­ta­tions of the hard­ware the cur­rent game con­soles use).

Dur­ing our dis­cus­sion we talked about the soft­ware lim­i­ta­tions of such hardware.

Cur­rent TV’s come for exam­ple with some pre­de­fined inter­net chan­nels, but not with a real web brows­er. We think that peo­ple which keep a TV for 10 years or longer (like for exam­ple our par­ents and prob­a­bly both of us too) this will result in a loss of fea­tures after some years, because those chan­nels will get less atten­tion of case to exist at all. There is also no way to switch to alter­na­tives then, except by buy­ing a new TV (we expect that there will be no firmware update in such a case). With a real web brows­er this would not be an issue (it may be more easy to enter URL’s with a real key­board than with a remote con­trol, but let us do small steps here). Game con­soles are a bit bet­ter in this regard, but there we have the prob­lem that some web­sites are too much mem­o­ry hun­gry (they do not include the user agent of the game con­sole browsers in the same class as smart phones or tablet PCs… from the size aspect they are not, but from the mem­o­ry and com­put­ing pow­er aspect they are more similar).

I would expect that the TV sta­tions do not want to have TVs with real­ly good browsers, because then you may not need a TV sta­tion any­more. But this is what users would use if it would be there.

Anoth­er deficit is that there is not a mail pro­gram in game con­soles and TV’s. For writ­ing mails you need a real key­board, but for a quick check if there is mail (e.g. X unread mails, or maybe even dis­play­ing the sub­ject line of the emails) or maybe to just read with­out answer­ing a solu­tion with­out a key­board con­nect­ed would already be enough.

I expect that con­sole man­u­fac­tur­ers do not want to spend mon­ey for some­thing peo­ple are not will­ing to give much mon­ey for, respec­tive­ly for some­thing where they can not make mon­ey with (an email ser­vice from the con­sole com­pa­ny would be anoth­er mail ser­vice addi­tion­al to the one for the PC and maybe addi­tion­al to the one of the smart phone… peo­ple do not need 10 email accounts, one is enough).

Anoth­er over­looked fea­ture is some kind of VoIP+Video fea­ture (at least for the game con­soles which have option­al­ly a cam­era, but IMO this is also pos­si­ble for the next gen­er­a­tion of TV’s with build-in web­cams). At least the offer­ings from Sony and Microsoft are pow­er­ful enough to come with some kind of video con­fer­enc­ing soft­ware. It does not mat­ter much if this is Skype or the Google ver­sion of this, or some oth­er wide­spread one (MS sure­ly wants to use their own stuff), it just has to be one which is in wide­spread use to be adopt­ed by the people.This does not need to be in HD, even a small video would already be much more than what is avail­able ATM.

Basi­cal­ly I gave the answer to my ques­tion (the title of this post­ing) myself (except for the video con­fer­enc­ing stuff)… but on the oth­er hand this would be some­thing which could set a prod­uct apart from oth­ers. For the PS3 this may be now one of the things which could show up in the Home­brew scene, now that the secu­ri­ty of the PS3 is com­pro­mised. For the Wii at least the email part could be eas­i­ly done. The rest… would have to catch up in case some­thing like this shows up for the PS3 and is used extensively.