A phoronix bench­mark cre­ates a huge bench­mark­ing discussion

The recent Phoronix bench­mark which com­pared a release can­di­date of FreeB­SD 9 with Ora­cle Lin­ux Serv­er 6.1 cre­at­ed a huge dis­cus­sion in the FreeB­SD mail­inglists. The rea­son was that some peo­ple think the num­bers pre­sent­ed there give a wrong pic­ture of FreeB­SD. Part­ly because not all bench­mark num­bers are pre­sent­ed in the most promi­nent page (as linked above), but only at a dif­fer­ent place. This gives the impres­sion that FreeB­SD is infe­ri­or in this bench­mark while it just puts the focus (for a rea­son, accord­ing to some peo­ple) on a dif­fer­ent part of the bench­mark (to be more spe­cif­ic, blog­bench is doing disk reads and writes in par­al­lel, FreeB­SD gives high­er pri­or­i­ty to writes than to reads, FreeB­SD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are pre­sent­ed on the first page). Oth­er com­plaints are that it is told that the default install was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The author of the Phoronix arti­cle par­tic­i­pat­ed in parts of the dis­cus­sion and asked for spe­cif­ic improve­ment sug­ges­tions. A FreeB­SD com­mit­ter seems to be already work­ing to get some issues resolved. What I do not like per­son­al­ly, is that the arti­cle is not updat­ed with a remark that some things pre­sent­ed do not reflect the real­i­ty and a retest is necessary.

As there was much talk in the thread but not much obvi­ous activ­i­ty from our side to resolve some issues, I start­ed to improve the FreeB­SD wiki page about bench­mark­ing so that we are able to point to it in case some­one wants to bench­mark FreeB­SD. Oth­ers already chimed in and improved some things too. It is far from per­fect, some more eyes – and more impor­tant­ly some more fin­gers which add con­tent – are need­ed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeB­SD mail­inglist so that oth­ers can improve the wiki page).

What we need too, is a wiki page about FreeB­SD tun­ing (a first step would be to take the man-page and con­vert it into a wiki page, then to improve it, and then to feed back the changes to the man-page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to improve the sit­u­a­tion. Do not talk, write some­thing. No mat­ter if it is an improve­ment to the bench­mark­ing page, tun­ing advise, or a tool which inspects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­name­Last­name account and ask a FreeB­SD comit­ter for write access.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­mat­ic FreeB­SD bench­mark­ing. Unfor­tu­nate­ly the author run out of time. The frame­work was able to install a FreeB­SD sys­tem on a machine, run some spec­i­fied bench­mark (not much bench­marks where inte­grat­ed), and then install anoth­er FreeB­SD ver­sion to run the same bench­mark, or to rein­stall the same ver­sion to run anoth­er bench­mark. IIRC there was also some DB behind which col­lect­ed the results and maybe there was even some way to com­pare them. It would be nice if some­one could get some time to talk with the author to get the frame­work and set it up some­where, so that we have a con­trolled envi­ron­ment where we can do our own bench­marks in an auto­mat­ic and repeat­able fash­ion with sev­er­al FreeB­SD versions.