DTrace in GENERIC (-cur­rent)

In case you have not no­ticed yet, KDTRACE_​HOOKS is now in the GENERIC ker­nel in FreeBSD-cur­rent. This means you just need to load the DTrace mod­ules and can use DTrace with the GENERIC ker­nel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you in­vest read­ing it.

How to fix FreeBSD for cor­por­a­tions (and user with smal­ler in­stall­a­tions), the tools-​viewpoint

There is a huge dis­cus­sion go­ing on on hackers@ about how FreeBSD is not suit­able for large in­stall­a­tions (any­more?). As of this writ­ing, the dis­cus­sion seems to get some discussion-​clusters. We have some sub-​topics which could lead to some good im­prove­ments.

One sub­top­ic is the re­lease en­gin­eer­ing. Some changes like a more guided ap­proach of what should be merged to which branch, the fre­quency of re­leases and maybe some kind of long-term-branch(es). There is some dis­cus­sion to get maybe some joined-​funding in some way from in­ter­ested parties to pay someone to take care about long-term-branch(es).

An­oth­er sub­top­ic is the way bugs are handled in our old bugtrack­ing soft­ware and how patches go un­noticed there.

And both of them are con­nec­ted (parts more, parts less) by what can be done in a vo­lun­teer pro­ject.

To me it looks like the pro­pos­als “just” need some re­fine­ments and some “vo­lun­teers” to put value (this means man power and/​or money) to what they said.

What I want to dis­cuss here is, how tools could help with mak­ing PRs/​patches more vis­ible to de­velopers (there is already the pos­sib­il­ity to get emails from the small bugbuster-​team about patches in PR data­base, but you have to ask them to get them) and how to make it more easy to get patches in­to FreeBSD.

Mak­ing bugs more vis­ible to de­velopers

The ob­vi­ous first: We need a dif­fer­ent bugtrack­ing sys­tem. We already know about it. There is (or was…) even someone work­ing IIRC on an eval­u­ation of what could be done and how easy/​hard it would be. I am not aware of any out­come, des­pite the fact that it is months (or even a year) since this was an­nounced. I do not blame any­one here, I would like to get time to fin­ish some FreeBSD vo­lun­teer work my­self.

In my opin­ion this needs to be handled in a com­mer­cial way. Someone needs to be of­fi­cially paid (with a dead­line) to pro­duce a res­ult. Un­for­tu­nately there is the prob­lem that the re­quire­ments are in a way, that people do not have to change their workflows/​procedures.

IIRC people ask that they should be able to send a mail to the bugtrack­er without the need for au­then­tic­a­tion. Per­son­ally I think the bugtrack­ing is­sue is in a state where we need to change our workflows/​procedures. It is con­veni­ent to get mails from the bugtrack­er and only have to reply to the mail to add some­thing. On the oth­er hand, if I re­port bugs some­where, and if I really care about the prob­lem res­ol­u­tion, I am will­ing lo­gin to whatever in­ter­face to get this damn prob­lem solved.

Send­ing a prob­lem re­port from the sys­tem where I have the is­sue in an easy way is a very use­ful fea­ture. Cur­rently we have send-​pr for this and it uses emails. This means it re­quires a work­ing email setup. As an user I do not care if the tool uses email or HTTP or HTTPS, I just want to have an easy way to sub­mit the prob­lem. I would not mind if I first have to do a “send-​problem re­gister me@tld” (once), “send-​problem lo­gin me@tld” (once per system+user I want to send from) and then maybe a “send-​problem tem­plate write_template_here.txt” (to get some tem­plate text to fill out), edit the tem­plate file and then run “send-​problem send my_report.txt file1 file2 …”. That would be a dif­fer­ent work­flow, but still easy.

Email no­ti­fic­a­tions are surely needed, but if I really care about a prob­lem, I can be bothered to re­gister first. So in my opin­ion, we need a dif­fer­ent bugtrack­er des­per­ately enough that we need to drop our re­quire­ments re­gard­ing our cur­rent workflow/​procedures (even if it means we can not get a com­mand line way of sub­mit­ting bugs at all). The primary goal of the soft­ware needs to be to make it easy to track and re­solve bugs. The sub­mis­sion of bugs shall be not hard too. If I look at the state of the world as it is ATM, I would say a webin­ter­face with au­then­tic­a­tion is not a big bur­den to take if I really want to get my prob­lem fixed. Some com­mand line tool would be nice to have, but re­gard­ing the cur­rent state of our bugtrack­er it needs to be op­tion­al in­stead of a hard re­quire­ment.

Apart from mak­ing it easy to track and re­solve prob­lems, the soft­ware also needs to be able to make us aware of the biggest prob­lems. Now… you may ask what is a big prob­lem. Well… IMO it does not mat­ter to you what I think is big or small here. The per­son with a prob­lem needs to de­cide what is a big prob­lem to him. And people with the same prob­lem need to be able to tell that it is also a big prob­lem for them. So a fea­ture which al­lows to “vote” or “+1” or “AOL” (or how­ever you want to call it) would al­low to let users with prob­lems voice their opin­ion upon the rel­ev­ance of the prob­lem to our userbase. This also means there needs to be a way to see the highest voted prob­lems. An auto­mat­ic mail would be best, but as above this is op­tion­al. If I as a de­veloper really care about this, I can be bothered to lo­gin to a webin­ter­face (or maybe someone vo­lun­teers to make a copy & paste and send a mail… we need to be will­ing to re­think our pro­ced­ures).

Get­ting patches more easy in­to a FreeBSD branch

It looks to me that this top­ic is re­quires a little bit more in­volve­ment from mul­tiple tools. In my opin­ion we need to switch to a dis­trib­uted ver­sion con­trol sys­tem. One which al­lows to eas­ily cre­ate my own branch of FreeBSD on my own hard­ware, and which al­lows to let oth­er users use my branch eas­ily (if I want to al­low oth­er to branch from my branch). It also needs to be able to let me push my changes to­wards FreeBSD. Ob­vi­ously not dir­ectly in­to the of­fi­cial sources, but in­to some kind of sta­ging area. Oth­er people should be able to have a look at this sta­ging area and be able to re­view what I did. They need to be able to make some com­ments for oth­ers to see, or give some kind of (multi-dimensional?-)rating for the patch (code qual­ity /​ works for me /​ does not work /​ …). Based upon the review/​rating and maybe some auto­mated eval­u­ation (com­pile test /​ re­gres­sion test /​ bench­mark run) a com­mit­ter could push the patch in­to the of­fi­cial FreeBSD tree (ideal would be some auto­mated no­ti­fic­a­tion sys­tem, a push but­ton solu­tion for in­teg­ra­tion and so on, but as above we should not be afraid if we do not get all the bells and whistles).

If we would have some­thing like this in place, cre­at­ing some kind of long-​term-​release branch could be used more eas­ily in a col­ab­or­at­ive man­ner. Com­pan­ies which use the same long-​term-​release branch could sub­mit their back­ports of fixes/​features this way. They also could see if sim­il­ar branches (there could be re­lated but dif­fer­ent branches, like 9.4–se­cur­ity-fixes-​only <= 9.4-official-errata-only <= 9.4-bugfixes <= 9.4-bugfixes-and-driverupdates <= …) could be merged to their in-​house branch (and maybe con­sequently push-​back to the of­fi­cial branch they branched from if the patch comes from a dif­fer­ent branch).

It does not mat­ter here if we would cre­ate a fixed set of branches for each re­lease, or if we only cre­ate some special-​purpose branches based upon the phase of the moon (ideally we would cre­ate a lot of branches for every re­lease, companies/​users can cherry pick/​submit what they want, and the status of a long-​term-​branch is solely based upon the in­flow of patches and not by what the se­cur­ity team or re­lease man­ager or a ran­dom de­veloper thinks it should be… but the real­ity will prob­ably be some­where in the middle).

I do not know if tools ex­ists to make all this hap­pen, or which tools could be put to­geth­er to make it hap­pen. I also did not men­tion on pur­pose tools I am aware of which already provide (small) parts of this. These are just some ideas to think about. In­ter­ested parties are in­vited to join the dis­cus­sion on hackers@ (which is far away from dis­cuss­ing spe­cif­ic tools or fea­tures), but you are also free to add some com­ments here.

A phoronix bench­mark cre­ates a huge bench­mark­ing dis­cus­sion

The re­cent Phoronix bench­mark which com­pared a re­lease can­did­ate of FreeBSD 9 with Or­acle Linux Serv­er 6.1 cre­ated a huge dis­cus­sion in the FreeBSD mailing­lists. The reas­on was that some people think the num­bers presen­ted there give a wrong pic­ture of FreeBSD. Partly be­cause not all bench­mark num­bers are presen­ted in the most prom­in­ent page (as linked above), but only at a dif­fer­ent place. This gives the im­pres­sion that FreeBSD is in­feri­or in this bench­mark while it just puts the fo­cus (for a reas­on, ac­cord­ing to some people) on a dif­fer­ent part of the bench­mark (to be more spe­cif­ic, blo­g­bench is do­ing disk reads and writes in par­al­lel, FreeBSD gives high­er pri­or­ity to writes than to reads, FreeBSD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are presen­ted on the first page). Oth­er com­plaints are that it is told that the de­fault in­stall was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The au­thor of the Phoronix art­icle par­ti­cip­ated in parts of the dis­cus­sion and asked for spe­cif­ic im­prove­ment sug­ges­tions. A FreeBSD com­mit­ter seems to be already work­ing to get some is­sues re­solved. What I do not like per­son­ally, is that the art­icle is not up­dated with a re­mark that some things presen­ted do not re­flect the real­ity and a retest is ne­ces­sary.

As there was much talk in the thread but not much ob­vi­ous activ­ity from our side to re­solve some is­sues, I star­ted to im­prove the FreeBSD wiki page about bench­mark­ing so that we are able to point to it in case someone wants to bench­mark FreeBSD. Oth­ers already chimed in and im­proved some things too. It is far from per­fect, some more eyes – and more im­port­antly some more fin­gers which add con­tent – are needed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeBSD mailing­list so that oth­ers can im­prove the wiki page).

What we need too, is a wiki page about FreeBSD tun­ing (a first step would be to take the man-​page and con­vert it in­to a wiki page, then to im­prove it, and then to feed back the changes to the man-​page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to im­prove the situ­ation. Do not talk, write some­thing. No mat­ter if it is an im­prove­ment to the bench­mark­ing page, tun­ing ad­vise, or a tool which in­spects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­nameLast­name ac­count and ask a FreeBSD comit­ter for write ac­cess.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­mat­ic FreeBSD bench­mark­ing. Un­for­tu­nately the au­thor run out of time. The frame­work was able to in­stall a FreeBSD sys­tem on a ma­chine, run some spe­cified bench­mark (not much bench­marks where in­teg­rated), and then in­stall an­oth­er FreeBSD ver­sion to run the same bench­mark, or to re­in­stall the same ver­sion to run an­oth­er bench­mark. IIRC there was also some DB be­hind which col­lec­ted the res­ults and maybe there was even some way to com­pare them. It would be nice if someone could get some time to talk with the au­thor to get the frame­work and set it up some­where, so that we have a con­trolled en­vir­on­ment where we can do our own bench­marks in an auto­mat­ic and re­peat­able fash­ion with sev­er­al FreeBSD ver­sions.

Video4Linux2 sup­port in FreeBSD (linuxu­lat­or)

I com­mit­ted the v4l2 sup­port in­to the lin­ux­u­la­tor (in 9–cur­rent). Part of this was the im­port of the v4l2 head­er from linux. We have the per­mis­sion to use it (like the v4l one), it is not li­censed via GPL. This means we can use it in FreeBSD nat­ive dri­vers, and they are even al­lowed to be com­piled in­to GENERIC (but I doubt we have a dri­ver which could pro­vide the v4l2 inter­face in GENERIC).

The code I com­mit­ted is “just” the glue-​code which al­lows to use FreeBSD nat­ive devices which pro­vide a v4l2 inter­face (e.g. multimedia/​pwcbsd or multimedia/​webcamd) from linux pro­grams.

Thanks to nox@ for writ­ing the glue code.

VDR ports docs

After a quick dis­cus­sion with nox@ I made a copy&paste of his “VDR is com­mit­ted now”-mail in­to the FreeBSD wiki. I also re-​styled some small parts of it to fit bet­ter in­to the wiki. It is not per­fect, but already us­able. Now in­ter­ested people can go and im­prove the docs there.

Thanks to Juer­gen for all his work in this area!

Heat­Maps again…

Today I stumbled again over some Heat­Maps from Brendan Gregg (of DTrace-fame). This time it was the PDF of his present­a­tion at the LISA 2010 con­fer­ence. It shows nicely how he plans to evolve it from a single-​machine (like in Ana­lyt­ics for Or­acle Stor­age products) to the cloud. It is a very good over­view about what kind of in­tu­it­ive per­form­ance visu­al­iz­a­tion you can do with this.

I would love to see some­thing for FreeBSD (or oth­er per­form­ance data). Maybe someone could take the DTraceTazTool as a base or hint and write some­thing gen­er­ic which works for a lot of things…

There are just too much nice and in­ter­est­ing things out there, and not enough time for all of them.

Un­der­stand­ing latency

Brendan Gregg of Sun Or­acle fame made a good ex­plan­a­tion how to visu­al­ize latency to get a bet­ter un­der­stand­ing of what is go­ing on (and as such about how to solve bot­tle­necks). I have seen all this already in vari­ous posts in his blog and in the Ana­lyt­ics pack­age in an Open­Stor­age present­a­tion, but the ACM art­icle sum­mar­izes it very good.

Un­for­tu­nately Ana­lyt­ics is AFAIK not avail­able in OpenSol­ar­is, so we can not go out and ad­apt it for FreeBSD (which would prob­ably re­quire to port/​implement some ad­di­tion­al dtrace stuff/​probes). I am sure some­thing like this would be very in­ter­est­ing to all those com­pan­ies which use FreeBSD in an ap­pli­ance (re­gard­less if it is a stor­age ap­pli­ance like Net­App, or a net­work ap­pli­ance like a Cisco/​Juniper router, or any­thing else which has to per­form good).