Alexander Leidinger

Just another weblog

Dec
21

A phoronix bench­mark cre­ates a huge bench­mark­ing discussion

The recent Phoronix bench­mark which com­pared a release can­di­date of FreeBSD 9 with Ora­cle Linux Server 6.1 cre­ated a huge dis­cus­sion in the FreeBSD mail­inglists. The rea­son was that some peo­ple think the num­bers pre­sented there give a wrong pic­ture of FreeBSD. Partly because not all bench­mark num­bers are pre­sented in the most promi­nent page (as linked above), but only at a dif­fer­ent place. This gives the impres­sion that FreeBSD is infe­rior in this bench­mark while it just puts the focus (for a rea­son, accord­ing to some peo­ple) on a dif­fer­ent part of the bench­mark (to be more spe­cific, blog­bench is doing disk reads and writes in par­al­lel, FreeBSD gives higher pri­or­ity to writes than to reads, FreeBSD 9 out­per­forms OLS 6.1 in the writes while OLS 6.1 shines with the reads, and only the reads are pre­sented on the first page). Other com­plaints are that it is told that the default install was used (in this case UFS as the FS), when it was not (ZFS as the FS).

The author of the Phoronix arti­cle par­tic­i­pated in parts of the dis­cus­sion and asked for spe­cific improve­ment sug­ges­tions. A FreeBSD com­mit­ter seems to be already work­ing to get some issues resolved. What I do not like per­son­ally, is that the arti­cle is not updated with a remark that some things pre­sented do not reflect the real­ity and a retest is necessary.

As there was much talk in the thread but not much obvi­ous activ­ity from our side to resolve some issues, I started to improve the FreeBSD wiki page about bench­mark­ing so that we are able to point to it in case some­one wants to bench­mark FreeBSD. Oth­ers already chimed in and improved some things too. It is far from per­fect, some more eyes — and more impor­tantly some more fin­gers which add con­tent — are needed. Please go to the wiki page and try to help out (if you are afraid to write some­thing in the wiki, please at least tell your sug­ges­tions on a FreeBSD mail­inglist so that oth­ers can improve the wiki page).

What we need too, is a wiki page about FreeBSD tun­ing (a first step would be to take the man-page and con­vert it into a wiki page, then to improve it, and then to feed back the changes to the man-page while keep­ing the wiki page to be able to cross ref­er­ence parts from the bench­mark­ing page).

I already told about this in the thread about the Phoronix bench­mark: every­one is wel­come to improve the sit­u­a­tion. Do not talk, write some­thing. No mat­ter if it is an improve­ment to the bench­mark­ing page, tun­ing advise, or a tool which inspects the sys­tem and sug­gests some tun­ing. If you want to help in the wiki, cre­ate a First­name­Last­name account and ask a FreeBSD comit­ter for write access.

A while ago (IIRC we have to think in months or even years) there was some frame­work for auto­matic FreeBSD bench­mark­ing. Unfor­tu­nately the author run out of time. The frame­work was able to install a FreeBSD sys­tem on a machine, run some spec­i­fied bench­mark (not much bench­marks where inte­grated), and then install another FreeBSD ver­sion to run the same bench­mark, or to rein­stall the same ver­sion to run another bench­mark. IIRC there was also some DB behind which col­lected the results and maybe there was even some way to com­pare them. It would be nice if some­one could get some time to talk with the author to get the frame­work and set it up some­where, so that we have a con­trolled envi­ron­ment where we can do our own bench­marks in an auto­matic and repeat­able fash­ion with sev­eral FreeBSD versions.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…
Share/Save

Tags: , , , , , , , , ,

2 Responses to “A phoronix bench­mark cre­ates a huge bench­mark­ing discussion”

  1. Tomasz Says:

    Don’t give Phoronix “bench­mark” any credit. It presents wrong pic­ture of any oper­at­ing sys­tem. For exam­ple, they use gzip com­pres­sion as filesys­tem test… or com­pile own ver­sion of httpd com­pletely dis­re­gard­ing what opart­ing sys­tem ships.

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  2. Erik Cederstrand Says:

    Regard­ing the bench­mark­ing frame­work — that’s me. Yes, I’m mostly out of time right now, but I’d like to offer help if some­one wants to pick up the project. It con­sists of a tinderbox-like build script for a build server, a script to run bench­marks on at least one slave, and a data­base and web fronted to aggre­gate and visu­al­ize results. The frame­work is designed specif­i­cally to track per­for­mance over time by com­pare revi­sions of FreeBSD, every­thing else being equal.
    The frame­work does work as-is, but it really needs to be updated. CVS -> SVN, bet­ter visu­al­iza­tion on the web fronted, and a bench­mark­ing script that is eas­ier to extend. Also, I don’t have hard­ware avail­able to run the bench­marks.
    If you’re inter­ested, just Google for my email address.

    GD Star Rating
    loading...
    GD Star Rating
    loading...

Leave a Reply