A Sol­ar­is 9 branded zone on Sol­ar­is 10

Last week we in­stalled a Sol­ar­is 9 branded zone. It is easy to do it, but in the end we re­com­men­ded to our cli­ent to use a real ma­chine with a nat­ive Sol­ar­is 9 in­stall­a­tion. There is too much broken:

  • Start scripts (like the one for cron) use the -P 1 op­tion to pkill. This means only kill if the par­ent PID is 1. In a zone this is not val­id, and as such cron (and oth­er dae­mons) do not get stopped, re­spect­ively a check if it is already run­ning at the start fails and cron can be star­ted mul­tiple times. There are already some patches which get auto­mat­ic­ally de­ployed dur­ing the first boot of the Sol­ar­is 9 branded zone, but there are still sev­er­al easy to de­tect bugs around.
  • SNMP does not work out of the box. A col­league tried to get it run­ning, but he failed. Googling for the er­ror mes­sage shows two hits. One hit is a link to the source, and an­oth­er one is a re­port of a per­son which has the same prob­lem. MAybe we could get net-​snmp up and run­ning, but we didn’t test this, as we want to have our Sol­ar­is 9 sys­tems sim­il­ar (few­er changes and spe­cial cases for cfen­gine… it is already not easy to un­der­stand for some col­leagues as it is).
  • There is no doc­u­ment­a­tion how to handle the first setup auto­mat­ic­ally. We think we can handle it by ex­tract­ing all files from the Sol­ar­is 9 flar we down­loaded from Or­acle for this in­stall­a­tion, modi­fy­ing the con­tents (e.g. adding a sysid.cfg), and cre­at­ing a flar again. We did not try to do it.

All this we de­tec­ted in less than half a day of the first play­ing around with it. To us Sol­ar­is 9 branded zones are more like al­pha or beta qual­ity fea­tures. We gave the re­com­mend­a­tion to our boss to tell the cli­ent that we do not think this is a pro­duc­tion ready fea­ture, and if the cli­ent in­sists on us­ing a Sol­ar­is 9 branded zone we can not really ac­cept to handle it with­in the nor­mal SLA.

So far we did not re­port any of the above prob­lems to Or­acle. Those prob­lems are dir­ectly vis­ible if you try to in­teg­rate a Sol­ar­is 9 branded zone in­to a good en­ter­prise class en­vir­on­ment, so it looks to me as there is no real pro­duc­tion qual­ity test­ing done on the side of Or­acle. To me this means there is no real com­mer­cial in­terest. It also means that there are prob­ably a lot of un­covered prob­lems, which is a night­mare to handle in a pro­duc­tion en­vir­on­ment.

ADSL RAM… the­ory and real­ity do not match

I am now wait­ing since Decem­ber that my ADSL line is switched to the rate ad­apt­ive mode (RAM). The­or­et­ic­ally it is pos­sible. Un­for­tu­nately the real­ity does not agree to this (yet).

Luck­ily I am not a nor­mal cus­tom­er, I know a tech­ni­cian which works for my ISP. He could switch the line without prob­lems, but the next up­date of the sys­tem (which hap­pens from time to time) would can­cel this again, as each up­date “re­sets” the status to what is re­cor­ded in the DB. The prob­lem is, that he can not switch my line to RAM in the DB (ac­tu­ally it is not him, he is a net­work tech­ni­cian not one of the sales people with ac­cess to the DB–in­ter­face). I am not the only cus­tom­er where this is not pos­sible. So far they where not able to see a pat­tern.

Cur­rently there are two col­leagues of him, a friend of him and me which he has as good ex­amples where it does not work (there are more, but those are “just” reg­u­lar cus­tom­ers). We are now his toys, he wants to find out how to con­vince the sys­tem to switch to RAM in those cases. This needs a while, as parts of this need to go the of­fi­cial way un­til he sees if it works or not.

I am very happy that I am not just a nor­mal cus­tom­er. This way it is much more trans­par­ent for me.

Heat­Maps again…

Today I stumbled again over some Heat­Maps from Brendan Gregg (of DTrace-fame). This time it was the PDF of his present­a­tion at the LISA 2010 con­fer­ence. It shows nicely how he plans to evolve it from a single-​machine (like in Ana­lyt­ics for Or­acle Stor­age products) to the cloud. It is a very good over­view about what kind of in­tu­it­ive per­form­ance visu­al­iz­a­tion you can do with this.

I would love to see some­thing for FreeBSD (or oth­er per­form­ance data). Maybe someone could take the DTraceTazTool as a base or hint and write some­thing gen­er­ic which works for a lot of things…

There are just too much nice and in­ter­est­ing things out there, and not enough time for all of them.


After read­ing Jim Gettys in­vest­ig­a­tions about the prob­lems cur­rent buf­fer sizes of net­work equip­ment pro­voke (which may even have im­plic­a­tions in the net neut­ral­ity de­bate), I had a look at which act­ive queue man­age­ment (AQM) al­gorithms with or without ex­pli­cit con­ges­tion no­ti­fic­a­tion (ECN) FreeBSD sup­ports.

It looks like there is not much im­ple­men­ted (if the best solu­tion would be im­ple­men­ted, it would not mat­ter how much there is, but un­for­tu­nately there is no best solu­tion). Oth­er sys­tems of­fer more. RED is im­ple­men­ted, but even the inventor/​researcher of RED thinks the al­gorithm needs some im­prove­ments (he is in the pro­cess of pre­par­ing a pa­per about this, as Jim Gettys re­veals). Blue/​SFBlue is not im­ple­men­ted (a more turnkey-​solution than the cur­rent RED im­ple­ment­a­tion). PID con­trol­ler (which may or may not be some­thing someone wants to use in this case… no idea about its pros/​cons in this re­gard, but it is ref­er­enced in the AQM art­icle on Wiki­pe­dia) is also not im­ple­men­ted.

Re­gard­ing ECN for FreeBSD you can find more or less no real doc­u­ment­a­tion in the net (at least with a simple “ECN FreeBSD” search). It is im­ple­men­ted for the RED al­gorithm, but as the RED al­gorithm needs some tuning/​setup, this is not a turn­key solu­tion. There is a ECN re­lated sy­sctl, but I do not have the im­pres­sion that this is a turnkey-​solution which ma­gic­ally gen­er­ates ECN mes­sages without us­ing dummynet for AQM.

From my cur­rent un­der­stand­ing (but I think I do not know a lot about this top­ic) it looks like AQM is a fea­ture most people would like to have ac­tiv­ated by de­fault (with an ap­pro­pri­ate al­gorithm which does not need tun­ing to pro­duce a good enough res­ult). If this is cor­rect, it is a shame that FreeBSD does not ac­tiv­ate AQM with an al­gorithm which is not bad for most cases by de­fault (with the op­tion to change the al­gorithm and to dis­able com­pletely). If my un­der­stand­ing is not cor­rect, I would like to get a hit with the clue bat please.

Why are game console/​TV com­pan­ies not im­ple­ment­ing this?

At the week­end a friend vis­ited me. We have not seen since each oth­er since a long time. As we stud­ied both com­puter sci­ence, parts of our dis­cus­sion where off course tech­no­logy re­lated. Parts of the dis­cus­sion where about cur­rent TV’s and game con­soles (he par­ti­cip­ated in the design of the PS3 CPU, so he is well aware about the tech­nic­al lim­it­a­tions of the hard­ware the cur­rent game con­soles use).

Dur­ing our dis­cus­sion we talked about the soft­ware lim­it­a­tions of such hard­ware.

Cur­rent TV’s come for ex­ample with some pre­defined in­ter­net chan­nels, but not with a real web browser. We think that people which keep a TV for 10 years or longer (like for ex­ample our par­ents and prob­ably both of us too) this will res­ult in a loss of fea­tures after some years, be­cause those chan­nels will get less at­ten­tion of case to ex­ist at all. There is also no way to switch to al­tern­at­ives then, ex­cept by buy­ing a new TV (we ex­pect that there will be no firm­ware up­date in such a case). With a real web browser this would not be an is­sue (it may be more easy to enter URL’s with a real key­board than with a re­mote con­trol, but let us do small steps here). Game con­soles are a bit bet­ter in this re­gard, but there we have the prob­lem that some web­sites are too much memory hungry (they do not in­clude the user agent of the game con­sole browsers in the same class as smart phones or tab­let PCs… from the size as­pect they are not, but from the memory and com­put­ing power as­pect they are more sim­il­ar).

I would ex­pect that the TV sta­tions do not want to have TVs with really good browsers, be­cause then you may not need a TV sta­tion any­more. But this is what users would use if it would be there.

An­oth­er de­fi­cit is that there is not a mail pro­gram in game con­soles and TV’s. For writ­ing mails you need a real key­board, but for a quick check if there is mail (e.g. X un­read mails, or maybe even dis­play­ing the sub­ject line of the emails) or maybe to just read without an­swer­ing a solu­tion without a key­board con­nec­ted would already be enough.

I ex­pect that con­sole man­u­fac­tur­ers do not want to spend money for some­thing people are not will­ing to give much money for, re­spect­ively for some­thing where they can not make money with (an email ser­vice from the con­sole com­pany would be an­oth­er mail ser­vice ad­di­tion­al to the one for the PC and maybe ad­di­tion­al to the one of the smart phone… people do not need 10 email ac­counts, one is enough).

An­oth­er over­looked fea­ture is some kind of VoIP+Video fea­ture (at least for the game con­soles which have op­tion­ally a cam­era, but IMO this is also pos­sible for the next gen­er­a­tion of TV’s with build-​in web­cams). At least the of­fer­ings from Sony and Mi­crosoft are power­ful enough to come with some kind of video con­fer­en­cing soft­ware. It does not mat­ter much if this is Skype or the Google ver­sion of this, or some oth­er wide­spread one (MS surely wants to use their own stuff), it just has to be one which is in wide­spread use to be ad­op­ted by the people.This does not need to be in HD, even a small video would already be much more than what is avail­able ATM.

Ba­sic­ally I gave the an­swer to my ques­tion (the title of this post­ing) my­self (ex­cept for the video con­fer­en­cing stuff)… but on the oth­er hand this would be some­thing which could set a product apart from oth­ers. For the PS3 this may be now one of the things which could show up in the Homebrew scene, now that the se­cur­ity of the PS3 is com­prom­ised. For the Wii at least the email part could be eas­ily done. The rest… would have to catch up in case some­thing like this shows up for the PS3 and is used ex­tens­ively.

How big are the buf­fers in FreeBSD drivers?

Today I have read an in­ter­est­ing in­vest­ig­a­tion and prob­lem ana­lys­is from Jim Gettys.

It is a set of art­icles he wrote over sev­er­al months and is not fin­ished writ­ing as of this writ­ing (if you are deeply in­ter­ested in it go and read them, the most in­ter­est­ing ones are from Decem­ber and Janu­ary and the com­ments to the art­icles are also con­trib­ut­ing to the big pic­ture). Ba­sic­ally he is telling that a lot of net­work prob­lems users at home (with ADSL/​cable or WLAN) ex­per­i­ence  are be­cause buf­fers in the net­work hard­ware or in op­er­at­ing sys­tems are too big. He also pro­poses work­arounds un­til this prob­lem is at­tacked by OS vendors and equip­ment man­u­fac­tur­ers.

Ba­sic­ally he is telling the net­work con­ges­tion al­gorithms can not do their work good, be­cause the net­work buf­fers which are too big come in­to the way of their work (not re­port­ing pack­et loss timely enough re­spect­ively try to not lose pack­ets in situ­ations where pack­et loss would be bet­ter be­cause it would trig­ger ac­tion in the con­ges­tion al­gorithms).

He in­vest­ig­ated the be­ha­vi­or of Linux, OS X and Win­dows (the sys­tem he had avail­able). I wanted to have a quick look at the situ­ation in FreeBSD re­gard­ing this, but it seems at least with my net­work card I am not able to see/​find the cor­res­pond­ing size of the buf­fers in drivers in 30 seconds.

I think it would be very good if this is­sue is in­vest­ig­ated in FreeBSD, and apart from maybe tak­ing some ac­tion in the source also write some sec­tion for the hand­book which ex­plains the is­sue (one prob­lem here is, that there are situ­ations where you want/​need to have such big buf­fers and as such we can not just downs­ize them) and how to bench­mark and tune this.

Un­for­tu­nately I even have too much on my plate to even fur­ther look in­to this. 🙁 I hope one of the net­work people in FreeBSD is pick­ing up the ball and starts play­ing.