A phoronix bench­mark cre­ates a huge bench­mark­ing dis­cus­sion

The re­cent Phoronix bench­mark which com­pared a re­lease can­did­ate of FreeBSD 9 with Or­acle Linux Serv­er 6.1 cre­ated a huge dis­cus­sion in the FreeBSD mailing­lists. The reas­on was that some people think the num­bers presen­ted there give a wrong pic­ture of FreeBSD. Partly be­cause not all bench­mark num­bers are presen­ted in the most prom­in­ent page (as linked above), but only at a dif­fer­ent place. This gives the im­pres­sion that FreeBSD is in­feri­or in this bench­mark while it just puts the fo­cus (for a reas­on, ac­cord­ing to some people) on a dif­fer­ent part of the bench­mark (to be more spe­cif­ic, blo­g­bench is do­ing disk reads and writes in par­al­lel, FreeBSD gives high­er pri­or­ity to writes than to reads, FreeBSD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are presen­ted on the first page). Oth­er com­plaints are that it is told that the de­fault in­stall was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The au­thor of the Phoronix art­icle par­ti­cip­ated in parts of the dis­cus­sion and asked for spe­cif­ic im­prove­ment sug­ges­tions. A FreeBSD com­mit­ter seems to be already work­ing to get some is­sues re­solved. What I do not like per­son­ally, is that the art­icle is not up­dated with a re­mark that some things presen­ted do not re­flect the real­ity and a retest is ne­ces­sary.

As there was much talk in the thread but not much ob­vi­ous activ­ity from our side to re­solve some is­sues, I star­ted to im­prove the FreeBSD wiki page about bench­mark­ing so that we are able to point to it in case someone wants to bench­mark FreeBSD. Oth­ers already chimed in and im­proved some things too. It is far from per­fect, some more eyes – and more im­port­antly some more fin­gers which add con­tent – are needed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeBSD mailing­list so that oth­ers can im­prove the wiki page).

What we need too, is a wiki page about FreeBSD tun­ing (a first step would be to take the man-​page and con­vert it in­to a wiki page, then to im­prove it, and then to feed back the changes to the man-​page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to im­prove the situ­ation. Do not talk, write some­thing. No mat­ter if it is an im­prove­ment to the bench­mark­ing page, tun­ing ad­vise, or a tool which in­spects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­nameLast­name ac­count and ask a FreeBSD comit­ter for write ac­cess.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­mat­ic FreeBSD bench­mark­ing. Un­for­tu­nately the au­thor run out of time. The frame­work was able to in­stall a FreeBSD sys­tem on a ma­chine, run some spe­cified bench­mark (not much bench­marks where in­teg­rated), and then in­stall an­oth­er FreeBSD ver­sion to run the same bench­mark, or to re­in­stall the same ver­sion to run an­oth­er bench­mark. IIRC there was also some DB be­hind which col­lec­ted the res­ults and maybe there was even some way to com­pare them. It would be nice if someone could get some time to talk with the au­thor to get the frame­work and set it up some­where, so that we have a con­trolled en­vir­on­ment where we can do our own bench­marks in an auto­mat­ic and re­peat­able fash­ion with sev­er­al FreeBSD ver­sions.

2 thoughts on “A phoronix bench­mark cre­ates a huge bench­mark­ing dis­cus­sion”

  1. Don’t give Phoronix “bench­mark” any cred­it. It presents wrong pic­ture of any op­er­at­ing sys­tem. For ex­ample, they use gzip com­pres­sion as filesys­tem test… or com­pile own ver­sion of ht­tpd com­pletely dis­reg­ard­ing what opart­ing sys­tem ships.

  2. Re­gard­ing the bench­mark­ing frame­work – that’s me. Yes, I’m mostly out of time right now, but I’d like to of­fer help if someone wants to pick up the pro­ject. It con­sists of a tinderbox-​like build script for a build serv­er, a script to run bench­marks on at least one slave, and a data­base and web fron­ted to ag­greg­ate and visu­al­ize res­ults. The frame­work is de­signed spe­cific­ally to track per­form­ance over time by com­pare re­vi­sions of FreeBSD, everything else be­ing equal.
    The frame­work does work as-​is, but it really needs to be up­dated. CVS -> SVN, bet­ter visu­al­iz­a­tion on the web fron­ted, and a bench­mark­ing script that is easi­er to ex­tend. Also, I don’t have hard­ware avail­able to run the bench­marks.
    If you’re in­ter­ested, just Google for my email ad­dress.

Leave a Reply

Your email address will not be published. Required fields are marked *