Peri­od­ic scrub­bing of ZFS pools

I noticed that we do not have some auto­mat­ic way of scrub­bing a ZFS pool peri­od­i­cal­ly. A quick poll on fs@ revealed, that there is inter­est in some­thing like this. So I took a lit­tle bit of time to write a peri­od­ic dai­ly script which checks if the last scrub is X days ago and scrubs a pool accord­ing­ly. The script has options to scrub all pools, or just a spe­cif­ic sub­set. It also allows to spec­i­fy a time-interval between scrubs for each pool with dif­fer­ent lev­els of fall-back (if no pool-specific inter­val is set, the default inter­val is used, which is set to 30 days if no oth­er default inter­val is specified).

The dis­cus­sion about this is hap­pen­ing over at fs@, so go there and have a look for the CFT (with a link to the WIP of the script) and the dis­cus­sion if you are interested.

So far there are some minor details to sort out (and a lit­tle bit of doc­u­men­ta­tion to write) before I can com­mit it… prob­a­bly next week.

Under­stand­ing latency

Bren­dan Gregg of Sun Ora­cle fame made a good expla­na­tion how to visu­al­ize laten­cy to get a bet­ter under­stand­ing of what is going on (and as such about how to solve bot­tle­necks). I have seen all this already in var­i­ous posts in his blog and in the Ana­lyt­ics pack­age in an Open­Stor­age pre­sen­ta­tion, but the ACM arti­cle sum­ma­rizes it very good.

Unfor­tu­nate­ly Ana­lyt­ics is AFAIK not avail­able in Open­So­laris, so we can not go out and adapt it for FreeB­SD (which would prob­a­bly require to port/implement some addi­tion­al dtrace stuff/probes). I am sure some­thing like this would be very inter­est­ing to all those com­pa­nies which use FreeB­SD in an appli­ance (regard­less if it is a stor­age appli­ance like NetApp, or a net­work appli­ance like a Cisco/Juniper router, or any­thing else which has to per­form good).

Direct, indi­rect and explic­it depen­den­cies in progams/ports

The dis­cus­sion about direct and indi­rect depen­den­cies is com­ing up again on the FreeB­SD mail­inglists. Seems I should make some blog post about it, maybe it makes this top­ic more find­able than my post­ings in the mailinglists.

Some def­i­n­i­tions:

  • A direct depen­den­cy from A to B is when program/port A uses sym­bols from library/port B.
  • An indi­rect depen­den­cy from A to C is when program/port A uses sym­bols from library/port B but no sym­bols from library/port C, and library/port B uses sym­bols from library/port C.
  • An explic­it depen­den­cy from A to C is when it is a direct or indi­rect depen­den­cy A to C, and when the compiler-time-linker added an explic­it ref­er­ence to C to the program/lib of A.

Ide­al­ly we have no indi­rect depen­den­cies in the explic­it depen­den­cies, only direct depen­den­cies. Unfor­tu­nate­ly in real­i­ty we also have indi­rect depen­den­cies there. This has at least two causes:

  1. libtool (at least 1.x) does not (or was not) come with a hint on FreeB­SD, which tells that the run-time-linker is recur­sive­ly resolv­ing dependencies.
  2. Some pkg-config setups list indi­rect depen­den­cies as explic­it depen­den­cies (IIRC it depends if Requires.private and/or Libs.private is used in the .pc file or not; if it is used, there should be no indi­rect depen­den­cy appear from this soft­ware, but I am not 100% sure about this).

Three years ago I wrote /usr/ports/Tools/scripts/explicit_lib_depends.sh, it looks at the files of a giv­en port (it needs to be installed), and prints out explic­it depen­den­cies. Because of the indi­rect depen­den­cies which could be list­ed there, this list is not a list of ports which are real depen­den­cies from a source code point of view, but it reflects the link-time real­i­ty. If a port C shows up there, the port which is checked needs to be rebuild in case the ABI of library/port C changes.

LAME updat­ed in the FreeB­SD ports collection

After all the big-impact com­mits (Gnome/gettext/KDE/X11/…) have set­tled now, I took the time to update audio/lame (I iden­ti­fied more than 100 ports with an (implic­it) depen­den­cy on lame, 45 of them need­ed a portre­vi­sion bump; if I forgot/overlooked some, bump the revi­sion your­self or noti­fy me please). That is the first update of my ports where miwi@ did not beat me in com­mit­ting an update since a year (he has implic­it approval to do any­thing he wants with my ports).

I can be hap­py that he is/was this fast (and that we have such a pro­duc­tive and effi­cient com­mit­ter), or I can be sad that I do not have the time any­more to be faster than I am with such things… or both. Hmmm… I think I will go the hap­py way. 😉