Peri­od­ic scrub­bing of ZFS pools

I no­ticed that we do not have some auto­mat­ic way of scrub­bing a ZFS pool peri­od­ic­ally. A quick poll on fs@ re­vealed, that there is in­terest in some­thing like this. So I took a little bit of time to write a peri­od­ic daily script which checks if the last scrub is X days ago and scrubs a pool ac­cord­ingly. The script has op­tions to scrub all pools, or just a spe­cif­ic sub­set. It also al­lows to spe­cify a time–in­ter­val between scrubs for each pool with dif­fer­ent levels of fall-​back (if no pool-​specific in­ter­val is set, the de­fault in­ter­val is used, which is set to 30 days if no oth­er de­fault in­ter­val is spe­cified).

The dis­cus­sion about this is hap­pen­ing over at fs@, so go there and have a look for the CFT (with a link to the WIP of the script) and the dis­cus­sion if you are in­ter­ested.

So far there are some minor de­tails to sort out (and a little bit of doc­u­ment­a­tion to write) be­fore I can com­mit it… prob­ably next week.

Un­der­stand­ing latency

Brendan Gregg of Sun Or­acle fame made a good ex­plan­a­tion how to visu­al­ize latency to get a bet­ter un­der­stand­ing of what is go­ing on (and as such about how to solve bot­tle­necks). I have seen all this already in vari­ous posts in his blog and in the Ana­lyt­ics pack­age in an Open­Stor­age present­a­tion, but the ACM art­icle sum­mar­izes it very good.

Un­for­tu­nately Ana­lyt­ics is AFAIK not avail­able in OpenSol­ar­is, so we can not go out and ad­apt it for FreeBSD (which would prob­ably re­quire to port/​implement some ad­di­tion­al dtrace stuff/​probes). I am sure some­thing like this would be very in­ter­est­ing to all those com­pan­ies which use FreeBSD in an ap­pli­ance (re­gard­less if it is a stor­age ap­pli­ance like Net­App, or a net­work ap­pli­ance like a Cisco/​Juniper router, or any­thing else which has to per­form good).

Dir­ect, in­dir­ect and ex­pli­cit de­pend­en­cies in progams/​ports

The dis­cus­sion about dir­ect and in­dir­ect de­pend­en­cies is com­ing up again on the FreeBSD mailing­lists. Seems I should make some blog post about it, maybe it makes this top­ic more find­able than my post­ings in the mailing­lists.

Some defin­i­tions:

  • A dir­ect de­pend­ency from A to B is when program/​port A uses sym­bols from library/​port B.
  • An in­dir­ect de­pend­ency from A to C is when program/​port A uses sym­bols from library/​port B but no sym­bols from library/​port C, and library/​port B uses sym­bols from library/​port C.
  • An ex­pli­cit de­pend­ency from A to C is when it is a dir­ect or in­dir­ect de­pend­ency A to C, and when the compiler-​time–linker ad­ded an ex­pli­cit ref­er­ence to C to the program/​lib of A.

Ideally we have no in­dir­ect de­pend­en­cies in the ex­pli­cit de­pend­en­cies, only dir­ect de­pend­en­cies. Un­for­tu­nately in real­ity we also have in­dir­ect de­pend­en­cies there. This has at least two causes:

  1. lib­tool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-​time-​linker is re­curs­ively resolv­ing de­pend­en­cies.
  2. Some pkg–con­fig setups list in­dir­ect de­pend­en­cies as ex­pli­cit de­pend­en­cies (IIRC it de­pends if Requires.private and/​or Libs.private is used in the .pc file or not; if it is used, there should be no in­dir­ect de­pend­ency ap­pear from this soft­ware, but I am not 100% sure about this).

Three years ago I wrote /​usr/​ports/​Tools/​scripts/, it looks at the files of a giv­en port (it needs to be in­stalled), and prints out ex­pli­cit de­pend­en­cies. Be­cause of the in­dir­ect de­pend­en­cies which could be lis­ted there, this list is not a list of ports which are real de­pend­en­cies from a source code point of view, but it re­flects the link-​time real­ity. If a port C shows up there, the port which is checked needs to be re­build in case the ABI of library/​port C changes.

LAME up­dated in the FreeBSD ports col­lec­tion

After all the big-​impact com­mits (Gnome/​gettext/​KDE/​X11/​…) have settled now, I took the time to up­date audio/​lame (I iden­ti­fied more than 100 ports with an (im­pli­cit) de­pend­ency on lame, 45 of them needed a port­re­vi­sion bump; if I forgot/​overlooked some, bump the re­vi­sion your­self or no­ti­fy me please). That is the first up­date of my ports where miwi@ did not beat me in com­mit­ting an up­date since a year (he has im­pli­cit ap­prov­al to do any­thing he wants with my ports).

I can be happy that he is/​was this fast (and that we have such a pro­duct­ive and ef­fi­cient com­mit­ter), or I can be sad that I do not have the time any­more to be faster than I am with such things… or both. Hmmm… I think I will go the happy way. 😉