In the FreeBSD mailinglists I stumbled over a post which refers to a blog-post which describes why ZFS seems to be slow (on Solaris).
In short: ZFS guarantees that the NFS client does not experience silent corruption of data (NFS server crash and loss of data which is supposed to be already on disk for the client). A recommendation is to enable the disk-cache for disks which are completely used by ZFS, as ZFS (unlike UFS) is aware of disk-caches. This increases the performance to what UFS is delivering in the NFS case.
There is no in-deep description of what it means that ZFS is aware of disk-caches, but I think this is a reference to the fact that ZFS is sending a flush command to the disk at the right moments. Letting aside the fact that there are disks out there which lie to you about this (they tell the flush command finished when it is not), this would mean that this is supported in FreeBSD too.
So everyone who is currently disabling the ZIL to get better NFS performance (and accept silent data corruption on the client side): move your zpool to dedicated (no other real FS than ZFS, swap and dump devices are OK) disks (honest ones) and enable the disk-caches instead of disabling the ZIL.
I also recommend that people which have ZFS already on dedicated (and honest) disks have a look if the disk-caches are enabled.
So where and how are the disk-caches turned on?
Depends upon your hardware and OS version.
Here in my 9‑current with ATA via CAM I can use e.g. “camcontrol identify 0:0:0” to see the status (“write cache”). I do not know of a way to change the write cache setting from the command line, but I think the loader tunable “hw.ata.wc” is honored by ATA via CAM too (set it to 0 to disable the write cache for all disks managed by the ATA part).
For my USB memory stick I can get a list of “modepages” via “camcontrol modepage 4:0:0 ‑l”. It shows 0×08 as the “Caching Page”, so “camcontrol modepage 4:0:0 ‑m 0×08” gives me the status for it (WCE means “Write Cache Enable”). To change this modepage add “-e” to the previous command line (I doubt any consumer-grade USB memory stick has a write cache, but you can use this for USB attached disks).
For ATA hardware not controlled via CAM, have a look at the atacontrol(8) man-page how to determine the current status of your disks.
Disk caches are always enabled on FreeBSD, unless you manually disable them. Due to the way GEOM works, this is even true for non-dedicated disks (ie partitioned disks) using ZFS.
IOW, this blog post is not relevant for FreeBSD ZFS users. 🙂
I beg to differ. This post is highly relevant for people which work with FreeBSD since a not so short while and know about the recommendation to disable the write caches of disk drives to have an acceptable behavior in case of a power-failure. There are many people out there which know about this and disable the disk-cache as one of the first things.
BTW: the handling of the disk-caches os not related to GEOM at all, this is something the ATA/CAM subsystems are responsible for. There was even a short moment in time where sos@ switched the default in the ATA driver to disable the caches by default (IIRC he had to revert it because too much people had the opinion that performance(-reviews) are more important than data-consistency). AFAIK CAM (the SCSI side of it) does not even touch this setting, as it is a property of a drive (a setting in the drive), so you get whatever the drive vendor has set as factory-defaults (most probably the write cache is enabled).
Straight from the horse’s mouth:
http://opensolaris.org/jive/thread.jspa?threadID=27794
See Pawel’s reply about three messages in. Due to the way GEOM works, the cache is always enabled for ZFS, even when using slices/partitions.
Again, the caches are enabled by ATA/CAM and not by ZFS/GEOM. ZFS is able to send the flush-cache command in all cases, but this only matters if the cache is enabled. For ATA the caching is enabled by default, for SCSI disks it depends on the disks, see my camcontrol explanation above.