A phoronix bench­mark cre­ates a huge bench­mark­ing discussion

The recent Phoronix bench­mark which com­pared a release can­di­date of FreeB­SD 9 with Ora­cle Lin­ux Serv­er 6.1 cre­at­ed a huge dis­cus­sion in the FreeB­SD mail­inglists. The rea­son was that some peo­ple think the num­bers pre­sent­ed there give a wrong pic­ture of FreeB­SD. Part­ly because not all bench­mark num­bers are pre­sent­ed in the most promi­nent page (as linked above), but only at a dif­fer­ent place. This gives the impres­sion that FreeB­SD is infe­ri­or in this bench­mark while it just puts the focus (for a rea­son, accord­ing to some peo­ple) on a dif­fer­ent part of the bench­mark (to be more spe­cif­ic, blog­bench is doing disk reads and writes in par­al­lel, FreeB­SD gives high­er pri­or­i­ty to writes than to reads, FreeB­SD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are pre­sent­ed on the first page). Oth­er com­plaints are that it is told that the default install was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The author of the Phoronix arti­cle par­tic­i­pat­ed in parts of the dis­cus­sion and asked for spe­cif­ic improve­ment sug­ges­tions. A FreeB­SD com­mit­ter seems to be already work­ing to get some issues resolved. What I do not like per­son­al­ly, is that the arti­cle is not updat­ed with a remark that some things pre­sent­ed do not reflect the real­i­ty and a retest is necessary.

As there was much talk in the thread but not much obvi­ous activ­i­ty from our side to resolve some issues, I start­ed to improve the FreeB­SD wiki page about bench­mark­ing so that we are able to point to it in case some­one wants to bench­mark FreeB­SD. Oth­ers already chimed in and improved some things too. It is far from per­fect, some more eyes – and more impor­tant­ly some more fin­gers which add con­tent – are need­ed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeB­SD mail­inglist so that oth­ers can improve the wiki page).

What we need too, is a wiki page about FreeB­SD tun­ing (a first step would be to take the man-page and con­vert it into a wiki page, then to improve it, and then to feed back the changes to the man-page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to improve the sit­u­a­tion. Do not talk, write some­thing. No mat­ter if it is an improve­ment to the bench­mark­ing page, tun­ing advise, or a tool which inspects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­name­Last­name account and ask a FreeB­SD comit­ter for write access.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­mat­ic FreeB­SD bench­mark­ing. Unfor­tu­nate­ly the author run out of time. The frame­work was able to install a FreeB­SD sys­tem on a machine, run some spec­i­fied bench­mark (not much bench­marks where inte­grat­ed), and then install anoth­er FreeB­SD ver­sion to run the same bench­mark, or to rein­stall the same ver­sion to run anoth­er bench­mark. IIRC there was also some DB behind which col­lect­ed the results and maybe there was even some way to com­pare them. It would be nice if some­one could get some time to talk with the author to get the frame­work and set it up some­where, so that we have a con­trolled envi­ron­ment where we can do our own bench­marks in an auto­mat­ic and repeat­able fash­ion with sev­er­al FreeB­SD versions.

2 thoughts on “A phoronix bench­mark cre­ates a huge bench­mark­ing discussion”

  1. Tomasz says:

    Don’t give Phoronix “bench­mark” any cred­it. It presents wrong pic­ture of any oper­at­ing sys­tem. For exam­ple, they use gzip com­pres­sion as filesys­tem test… or com­pile own ver­sion of httpd com­plete­ly dis­re­gard­ing what opart­ing sys­tem ships.

  2. Erik Cederstrand says:

    Regard­ing the bench­mark­ing frame­work – that’s me. Yes, I’m most­ly out of time right now, but I’d like to offer help if some­one wants to pick up the project. It con­sists of a tinderbox-like build script for a build serv­er, a script to run bench­marks on at least one slave, and a data­base and web front­ed to aggre­gate and visu­al­ize results. The frame­work is designed specif­i­cal­ly to track per­for­mance over time by com­pare revi­sions of FreeB­SD, every­thing else being equal.
    The frame­work does work as-is, but it real­ly needs to be updat­ed. CVS -> SVN, bet­ter visu­al­iza­tion on the web front­ed, and a bench­mark­ing script that is eas­i­er to extend. Also, I don’t have hard­ware avail­able to run the benchmarks.
    If you’re inter­est­ed, just Google for my email address.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.