ZFS and NFS / on-disk-cache

In the FreeB­SD mail­inglists I stum­bled over  a post which refers to a blog-post which describes why ZFS seems to be slow (on Solaris).

In short: ZFS guar­an­tees that the NFS client does not expe­ri­ence silent cor­rup­tion of data (NFS serv­er crash and loss of data which is sup­posed to be already on disk for the client). A rec­om­men­da­tion is to enable the disk-cache for disks which are com­plete­ly used by ZFS, as ZFS (unlike UFS) is aware of disk-caches. This increas­es the per­for­mance to what UFS is deliv­er­ing in the NFS case.

There is no in-deep descrip­tion of what it means that ZFS is aware of disk-caches, but I think this is a ref­er­ence to the fact that ZFS is send­ing a flush com­mand to the disk at the right moments. Let­ting aside the fact that there are disks out there which lie to you about this (they tell the flush com­mand fin­ished when it is not), this would mean that this is sup­port­ed in FreeB­SD too.

So every­one who is cur­rent­ly dis­abling the ZIL to get bet­ter NFS per­for­mance (and accept silent data cor­rup­tion on the client side): move your zpool to ded­i­cat­ed (no oth­er real FS than ZFS, swap and dump devices are OK) disks (hon­est ones) and enable the disk-caches instead of dis­abling the ZIL.

I also rec­om­mend that peo­ple which have ZFS already on ded­i­cat­ed (and hon­est) disks have a look if the disk-caches are enabled.

Send to Kin­dle

6 thoughts on “ZFS and NFS / on-disk-cache”

    1. Depends upon your hard­ware and OS ver­sion.

      Here in my 9‑current with ATA via CAM I can use e.g. “cam­con­trol iden­ti­fy 0:0:0” to see the sta­tus (“write cache”). I do not know of a way to change the write cache set­ting from the com­mand line, but I think the loader tun­able “hw.ata.wc” is hon­ored by ATA via CAM too (set it to 0 to dis­able the write cache for all disks man­aged by the ATA part).

      For my USB mem­o­ry stick I can get a list of “mod­e­pages” via “cam­con­trol mod­e­page 4:0:0 ‑l”. It shows 0×08 as the “Caching Page”, so “cam­con­trol mod­e­page 4:0:0 ‑m 0×08” gives me the sta­tus for it (WCE means “Write Cache Enable”). To change this mod­e­page add “-e” to the pre­vi­ous com­mand line (I doubt any consumer-grade USB mem­o­ry stick has a write cache, but you can use this for USB attached disks).

      For ATA hard­ware not con­trolled via CAM, have a look at the atacontrol(8) man-page how to deter­mine the cur­rent sta­tus of your disks.

  1. Disk caches are always enabled on FreeB­SD, unless you man­u­al­ly dis­able them. Due to the way GEOM works, this is even true for non-dedicated disks (ie par­ti­tioned disks) using ZFS.

    IOW, this blog post is not rel­e­vant for FreeB­SD ZFS users. 🙂

    1. I beg to dif­fer. This post is high­ly rel­e­vant for peo­ple which work with FreeB­SD since a not so short while and know about the rec­om­men­da­tion to dis­able the write caches of disk dri­ves to have an accept­able behav­ior in case of a power-failure. There are many peo­ple out there which know about this and dis­able the disk-cache as one of the first things.

      BTW: the han­dling of the disk-caches os not relat­ed to GEOM at all, this is some­thing the ATA/CAM sub­sys­tems are respon­si­ble for. There was even a short moment in time where sos@ switched the default in the ATA dri­ver to dis­able the caches by default (IIRC he had to revert it because too much peo­ple had the opin­ion that performance(-reviews) are more impor­tant than data-consistency). AFAIK CAM (the SCSI side of it) does not even touch this set­ting, as it is a prop­er­ty of a dri­ve (a set­ting in the dri­ve), so you get what­ev­er the dri­ve ven­dor has set as factory-defaults (most prob­a­bly the write cache is enabled).

  2. Again, the caches are enabled by ATA/CAM and not by ZFS/GEOM. ZFS is able to send the flush-cache com­mand in all cas­es, but this only mat­ters if the cache is enabled. For ATA the caching is enabled by default, for SCSI disks it depends on the disks, see my cam­con­trol expla­na­tion above.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.