Alexander Leidinger

Just another weblog

Sep
23

ZFS and NFS / on-disk-cache

In the FreeBSD mail­inglists I stum­bled over  a post which refers to a blog-post which describes why ZFS seems to be slow (on Solaris).

In short: ZFS guar­an­tees that the NFS client does not expe­ri­ence silent cor­rup­tion of data (NFS server crash and loss of data which is sup­posed to be already on disk for the client). A rec­om­men­da­tion is to enable the disk-cache for disks which are com­pletely used by ZFS, as ZFS (unlike UFS) is aware of disk–caches. This increases the per­for­mance to what UFS is deliv­er­ing in the NFS case.

There is no in-deep descrip­tion of what it means that ZFS is aware of disk-caches, but I think this is a ref­er­ence to the fact that ZFS is send­ing a flush com­mand to the disk at the right moments. Let­ting aside the fact that there are disks out there which lie to you about this (they tell the flush com­mand fin­ished when it is not), this would mean that this is sup­ported in FreeBSD too.

So every­one who is cur­rently dis­abling the ZIL to get bet­ter NFS per­for­mance (and accept silent data cor­rup­tion on the client side): move your zpool to ded­i­cated (no other real FS than ZFS, swap and dump devices are OK) disks (hon­est ones) and enable the disk-caches instead of dis­abling the ZIL.

I also rec­om­mend that peo­ple which have ZFS already on ded­i­cated (and hon­est) disks have a look if the disk-caches are enabled.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…
Share

Tags: , , , , , , , , ,

6 Responses to “ZFS and NFS / on-disk-cache”

  1. andrnils Says:

    So where and how are the disk-caches turned on?

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  2. netchild Says:

    Depends upon your hard­ware and OS version.

    Here in my 9-current with ATA via CAM I can use e.g. “cam­con­trol iden­tify 0:0:0″ to see the sta­tus (“write cache”). I do not know of a way to change the write cache set­ting from the com­mand line, but I think the loader tun­able “hw.ata.wc” is hon­ored by ATA via CAM too (set it to 0 to dis­able the write cache for all disks man­aged by the ATA part).

    For my USB mem­ory stick I can get a list of “mod­e­pages” via “cam­con­trol mod­e­page 4:0:0 –l”. It shows 0×08 as the “Caching Page”, so “cam­con­trol mod­e­page 4:0:0 –m 0×08″ gives me the sta­tus for it (WCE means “Write Cache Enable”). To change this mod­e­page add “-e” to the pre­vi­ous com­mand line (I doubt any consumer-grade USB mem­ory stick has a write cache, but you can use this for USB attached disks).

    For ATA hard­ware not con­trolled via CAM, have a look at the atacontrol(8) man-page how to deter­mine the cur­rent sta­tus of your disks.

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  3. Freddie Says:

    Disk caches are always enabled on FreeBSD, unless you man­u­ally dis­able them. Due to the way GEOM works, this is even true for non-dedicated disks (ie par­ti­tioned disks) using ZFS.

    IOW, this blog post is not rel­e­vant for FreeBSD ZFS users. :)

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  4. netchild Says:

    I beg to dif­fer. This post is highly rel­e­vant for peo­ple which work with FreeBSD since a not so short while and know about the rec­om­men­da­tion to dis­able the write caches of disk dri­ves to have an accept­able behav­ior in case of a power-failure. There are many peo­ple out there which know about this and dis­able the disk-cache as one of the first things.

    BTW: the han­dling of the disk-caches os not related to GEOM at all, this is some­thing the ATA/CAM sub­sys­tems are respon­si­ble for. There was even a short moment in time where sos@ switched the default in the ATA dri­ver to dis­able the caches by default (IIRC he had to revert it because too much peo­ple had the opin­ion that performance(-reviews) are more impor­tant than data-consistency). AFAIK CAM (the SCSI side of it) does not even touch this set­ting, as it is a prop­erty of a drive (a set­ting in the drive), so you get what­ever the drive ven­dor has set as factory-defaults (most prob­a­bly the write cache is enabled).

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  5. Freddie Says:

    Straight from the horse’s mouth:
    http://opensolaris.org/jive/thread.jspa?threadID=27794

    See Pawel’s reply about three mes­sages in. Due to the way GEOM works, the cache is always enabled for ZFS, even when using slices/partitions.

    GD Star Rating
    loading...
    GD Star Rating
    loading...
  6. netchild Says:

    Again, the caches are enabled by ATA/CAM and not by ZFS/GEOM. ZFS is able to send the flush-cache com­mand in all cases, but this only mat­ters if the cache is enabled. For ATA the caching is enabled by default, for SCSI disks it depends on the disks, see my cam­con­trol expla­na­tion above.

    GD Star Rating
    loading...
    GD Star Rating
    loading...

Leave a Reply