FreeNAS & Sen­sors for FreeBSD

This WE I was told that FreeNAS seems to want to move from FreeB­SD to Lin­ux (since then it seems there could be a lin­ux and a FreeB­SD ver­sion). One of the rea­sons seems to be a miss­ing sen­sors framework.

As I was com­mit­ting a port of the OpenB­SD sen­sors frame­work (pro­duced as part of the Google Sum­mer of Code 2007) to FreeB­SD and had to remove it after­wards because one com­mit­ter com­plained very loud­ly, I was asked what the sta­tus of this is.

The short sta­tus is: Nobody is doing some­thing about it.

Before I explain the long sta­tus, I give  a short overview what this sen­sors frame­work is:

  • a ker­nel API which allows to add sensors
  • an inter­face for the user­land to query the sen­sor data
  • some basic user­land code to show and log the sen­sor info

The API and the query inter­face are more or less inde­pen­dent. For the user­land code it was more a log­ging infra­struc­ture than a real mon­i­tor­ing solu­tion. The rea­son was the real mon­i­tor­ing solu­tions already exist (Nagios, snm­pd, …) and can be adapt­ed to query the sen­sors. Ide­al­ly a query in user­land should be han­dled by a library instead of direct­ly access­ing the sysctl inter­face, this way the kernel<->userland inter­face would be abstract­ed away (and could b replaced as needs arise). This was not done, it was some­thing to be done lat­er (Rome was not build in a day).

The user­land inter­face also only cared about dumb sen­sors (those which you need to query man­u­al­ly to get the infor­ma­tion), smart sen­sors (those which are able to send events them­self) where not tak­en care about in the sense of real­ly send­ing sensor-triggered events, but the ker­nel API allowed to add such sen­sors. The sysctl inter­face has no way of send­ing events, but FreeB­SD already has an event inter­face (devd is tak­ing care about it). It would have been not a prob­lem to send events via this chan­nel and let an user­land library take care about the deliv­ery togeth­er with oth­er sensor-data in userland.

And now the long sta­tus is:

PHK com­plained loud­ly about it. First he said he did not look at it but he com­plained that is not good regard­less. After a lot of nag­ging from me he had a look at it and was not hap­py about the time stuff in it (short: the FreeB­SD time­counter code is bet­ter). This was not a prob­lem in my opin­ion, we could have dis­abled this part with­out prob­lems. After such an offer from me, he com­plained that the sen­sors frame­work uses the sysctl inter­face instead of an entry in /dev.

At this point in time already sev­er­al user­land util­i­ties used the sysctl frame­work to query for sta­tus data in the ker­nel. So there was already prece­dence for such an use of it. Lat­er some more such uses where added too (e.g. the proc­stat stuff by core team mem­ber Robert Watson).

I saved some of the cor­re­spond­ing mails (to pub­lic mail­ing lists) in a mbox file, read the mess your­self if you want.

The bot­tom line is: Sev­er­al com­mit­ters (even some which we could call high pro­file com­mit­ters) told me that they do not see a prob­lem in the use of the sysctl inter­face. They do not seem to want to tell it in pub­lic (nobody of them voiced their opin­ion in the thread, so do not ask me who those peo­ple are). I am not inter­est­ed in invest­ing more of my spare time into fight­ing wind­mills (it looks like this to me).

So, if some­one is inter­st­ed in the code, r172631 has it. In the per­force repos­i­to­ry you can maybe find some sen­sors. I think most of it can still be used with­out much changes.

If some­one tries it with a more recent FreeB­SD, please drop me a note if it just applies fine, or a patch (or an URL to it) if it needs some mod­i­fi­ca­tions. Who knows, maybe in a future project it may be use­ful for me.

If there is enough inter­est by sev­er­al peo­ple, I can even put up a wiki page where those peo­ple can coor­di­nate, but that is most prob­a­bly all I am will­ing to invest fur­ther into this (at least in my unpaid time).

Video4Linux sup­port in FreeBSD

Yes­ter­day I com­mit­ted the v4l sup­port into the lin­ux­u­la­tor (in 9‑current). Part of this was the import of the v4l head­er from lin­ux. We have the per­mis­sion to use it, it is not licensed via GPL. This means we can use it in FreeB­SD native dri­vers, and they are even allowed to be com­piled into GENERIC (but I doubt we have a dri­ver which could pro­vide the v4l inter­face in GENERIC).

The code I com­mit­ted is “just” the glue-code which allows to use FreeB­SD native devices which pro­vide a v4l inter­face (e.g. multimedia/pwcbsd) from lin­ux programs.

If some­one is will­ing to write the glue-code for the v4l2 inter­face please con­tact me. We have the per­mis­sion to use the v4l2 head­er too, we just need some­one doing the coding.

In a sim­i­lar way, if some­one is will­ing to add v4l2 inter­face sup­port to FreeB­SD native dri­vers (I do not know any FreeB­SD dri­ver which pro­vides a v4l2 inter­face) , just tell me and I import the v4l2 head­er into FreeBSD.

And if some­one wants to add v4l sup­port to FreeB­SD native dri­vers but does not know where to start, feel free to con­tact me too.

Regard­ing the code which is in FreeB­SD ATM: it is not com­plete­ly fin­ished yet (some clip­ping relat­ed stuff is being worked on), but the not fin­ished part can not even be test­ed, as we do not know about a FreeB­SD device which pro­vides this functionality.

There is no MFC planned yet, but the more suc­cess sto­ries and test sce­nar­ios are being told about on the emu­la­tion or mul­ti­me­dia mail­inglists, the more like­ly I will do a MFC soon­er than later.

Video for lin­ux (v4l) emu­la­tion com­ing to the linuxulator

I am in the process of prepar­ing the import of code which makes v4l devices usable in the lin­ux­u­la­tor. Basi­cal­ly this means you can use your web­cam in skype (test­ed by the sub­mit­ter of the patch on amd64).

This is not a “apply patch and com­mit” thing, because the orig­i­nal videodev.h (with some mod­i­fi­ca­tions) is used. I was seek­ing the OK from core@ for this. As there is no license in the head­er, and the orig­i­nal author (Alan Cox, the lin­ux one, not our FreeB­SD one) gave per­mis­sions to use it, core@ is OK with the import.

I intent to do a ven­dor import of the lin­ux head­er (pre­pared today, togeth­er with some readme which explains where it comes from and some stuff to show that we are on the safe side regard­ing legal stuff), and then I want to copy this over to the lin­ux­u­la­tor as linux_videodev.h and com­mit the patch (prob­a­bly a lit­tle bit mod­i­fied in a few places). My plan is to com­mit it this week. Peo­ple which already want to play around with it now can have a look at the emu­la­tion mail­inglist, a link to the patch is post­ed there.

With the head­er being in a ven­dor branch, inter­est­ed peo­ple could then start to sub­mit new BSD licensed dri­vers or mod­i­fy exist­ing dri­vers which make use of the v4l inter­face, but I let the import of the head­er into the FreeB­SD include direc­to­ry up to the per­son which wants to com­mit the first native FreeBSD-v4l support.

When such native FreeBSD-v4l sup­port is com­mit­ted, the lin­ux­u­la­tor code needs to be revised.

Tarsnap usage statistics

The more time pass­es with tarsnap, the more impres­sive it is.

Fol­low­ing is a list of all my pri­vate­ly used sys­tems (2 machines which only host jails – here named Prison1 and Prison2 – and sev­er­al jails – here named accord­ing to their func­tion­al­i­ty) togeth­er with some tarsnap sta­tis­tics. For each back­up tarsnap prints out some sta­tis­tics. The amount of uncom­pressed stor­age space of all archives of this machine, the com­pressed stor­age space of all archives, the unique uncom­pressed stor­age space of all archives, the unique com­pressed stor­age space of all archives, and the same mount of info for the cur­rent archive. The unique stor­age space is after dedu­pli­ca­tion. The most inter­est­ing infor­ma­tion is the unique and com­pressed one. For a spe­cif­ic archive it shows the amount of data which is dif­fer­ent to all oth­er archives, and for the total amount it tells how much stor­age space is used on the tarsnap serv­er. I do not back­up all data in tarsnap. I do a full back­up on exter­nal stor­age (zfs snap­shot + zfs send | zfs receive) once in a while and tarsnap is only for the stuff which could change dai­ly or is very small (my mails belong to the first group, the con­fig of appli­ca­tions or the sys­tem to the sec­ond group). At the end of the post there is also an overview of the mon­ey I have spend so far in tarsnap for the backups.

Atten­tion: the fol­low­ing graphs are dis­play­ing small val­ues in KB, while the text is telling about sizes in MB or even GB!

Prison1

The back­up of one day cov­ers 1.1 GB of uncom­pressed data, the sub­trees I back­up are /etc, /usr/local/etc, /home, /root, /var/db/pkg, /var/db/mergemaster.mtree, /space/jails/flavours and a sub­ver­sion check­out of /usr/src (exclud­ing the ker­nel com­pile direc­to­ry; I back­up this as I have local mod­i­fi­ca­tions to FreeB­SD). If I want to have all days uncom­pressed on my hard­disk, I would have to pro­vide 10 GB of stor­age space. Com­pressed this comes down to 2.4 GB, unique uncom­pressed this is 853 MB, and unique com­pressed this is 243 MB. The fol­low­ing graph splits this up into all the back­ups I have as of this writ­ting. I only show the unique val­ues, as includ­ing the total val­ues would make the unique val­ues dis­ap­pear in the graph (val­ues too small).

chart


In this graph we see that I have a con­stant rate of new data. I think this is most­ly ref­er­ences to already stored data (/usr/src being the most like­ly cause of this, noth­ing changed in those directories).

Internal-DNS

One day cov­ers 7 MB of uncom­pressed data, all archives take 56 MB uncom­pressed, unique and com­pressed this comes down to 1.3 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/named, and /var/db/mergemaster.mtree.

chart


This graph is strange. I have no idea why there is so much data for the sec­ond and the last day. Noth­ing changed.

Outgoing-Postfix

One day cov­ers 8 MB of uncom­pressed data, all archives take 62 MB uncom­pressed, unique and com­pressed this comes down to 1.5 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/spool/postfix, and /var/db/mergemaster.mtree.

chart


This looks not bad. I was send­ing a lot of mails on the 25th. And the days in the mid­dle I was not send­ing much.

IMAP

One day cov­ers about 900 MB of uncom­pressed data, all archives take 7.2 GB uncom­pressed, unique and com­pressed this comes down to 526 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree, /home (mail fold­ers) and /usr/local/share/courier-imap.

chart


Obvi­ous­ly I have a not so small amount of change in my mail­box. As my spam­fil­ter is work­ing nice­ly this is direct­ly cor­re­lat­ed to mails from var­i­ous mail­inglists (most­ly FreeBSD).

MySQL (for the Horde web­mail interface)

One day cov­ers 100 MB of uncom­pressed data, all archives take 801 MB uncom­pressed, unique and com­pressed this comes down to 19 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mysql and /var/db/mergemaster.mtree.

chart


This is cor­re­lat­ed with the use of my web­mail inter­face, and as such is also cor­re­lat­ed with the amount of mails I get and send. Obvi­ous­ly I did not use my web­mail inter­face at the week­end (as the back­up cov­ers the change of the pre­vi­ous day).

Web­mail

One day cov­ers 121 MB of uncom­pressed data, all archives take 973 MB uncom­pressed, unique and com­pressed this comes down to 33 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree, /usr/local/www/horde and /home.

chart


This one is strange again. Noth­ing in the data changed.

Sam­ba

One day cov­ers 10 MB of uncom­pressed data, all archives take 72 MB uncom­pressed, unique and com­pressed this comes down to 1.9 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree and /var/db/samba.

chart


Here we see the changes to /var/db/samba, this should be most­ly my Wii access­ing mul­ti­me­dia files there.

Proxy

One day cov­ers 31 MB of uncom­pressed data, all archives take 223 MB uncom­pressed, unique and com­pressed this comes down to 6.6 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg and /var/db/mergemaster.mtree.

chart


This is also a strange graph. Again, noth­ing changed there (the cache direc­to­ry is not in the backup).

php­MyAd­min

One day cov­ers 44 MB of uncom­pressed data, all archives take 310 uncom­pressed, unique and com­pressed this comes down to 11 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree, /home and /usr/local/www/phpMyAdmin.

chart


And again a strange graph. No changes in the FS.

Gallery

One day cov­ers 120 MB of uncom­pressed data, all archives take 845 MB uncom­pressed, unique and com­pressed this comes down to 25 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree, /usr/local/www/gallery2 and /home/gallery (exclud­ing some parts of /home/gallery).

chart


This one is OK. Friends and Fam­i­ly access­ing the pictures.

Prison2

One day cov­ers 7 MB of uncom­pressed data, all archives take 28 MB uncom­pressed, unique and com­pressed this comes down to 1.3 MB. This cov­ers /etc, /usr/local/etc, /root, /var/db/pkg, /var/db/mergemaster.mtree, /space/jails/flavours and /home.

chart


This one looks strange to me again. Same rea­sons as with the pre­vi­ous graphs.

Incoming-Postfix

One day cov­ers 56 MB of uncom­pressed data, all archives take 225 MB uncom­pressed, unique and com­pressed this  comes down to 5.4 MB. This cov­ers /etc, /usr/local/etc, /usr/local/www/postfixadmin, /root/, /var/db/pkg, /var/db/mysql, /var/spool/postfix and /var/db/mergemaster.mtree.

chart


This graph looks OK to me.

Blog-and-XMPP

One day cov­ers 59 MB of uncom­pressed data, all archives take 478 MB uncom­pressed, unique and com­pressed this comes down to 14 MB. This cov­ers /etc, /usr/local/etc, /root, /home, /var/db/pkg, /var/db/mergemaster.mtree, /var/db/mysql and /var/spool/ejabberd (yes, no back­up of the web-data, I have it in anoth­er jail, no need to back­up it again). 

chart


With the MySQL and XMPP data­bas­es in the back­up, I do not think this graph is wrong.

Totals

The total amount of stored data per sys­tem is: 

chart


Costs

Since I use tarsnap (8 days), I have spend 38 cents, most of this is band­width cost for the trans­fer of the ini­tial back­up (29.21 cents). Accord­ing to the graphs, I am cur­rent­ly at about 8 – 14 cents per week (or about half a dol­lar per month) for my back­ups (I still have a machine to add, and this may increase the amount in a sim­i­lar way than the Prison1 sys­tem with 2 – 3 jails). The amount of mon­ey spend in US-cents (round­ed!) per day is: 

chart


ZFS & power-failure: stable

At the week­end there was a power-failure at our disaster-recovery-site. As every­thing should be con­nect­ed to the UPS, this should not have had an impact… unfor­tu­nate­ly the guys respon­si­ble for the cabling seem to have not pro­vid­ed enough pow­er con­nec­tions from the UPS. Result: one of our stor­age sys­tems (all vol­umes in sev­er­al RAID5 vir­tu­al disks) for the test sys­tems lost pow­er, 10 hard­disks switched into failed state when the pow­er was sta­ble again (I was told there where sev­er­al small power-failures that day). After telling the soft­ware to have a look at the dri­ves again, all phys­i­cal disks where accepted.

All vol­umes on one of the vir­tu­al disks where dam­aged (actu­al­ly, one of the vir­tu­al disks was dam­aged) beyond repair and we had to recov­er from backup.

All ZFS based mount­points on the good vir­tu­al disks did not show bad behav­ior (zfs clear + zfs scrub for those which showed check­sum errors to make us feel bet­ter). For the UFS based ones… some caused a pan­ic after reboot and we had to run fsck on them before try­ing a sec­ond boot.

We spend a lot more time to get UFS back online, than get­ting ZFS back online. After this expe­ri­ence it looks like our future Solaris 10u8 installs will be with root on ZFS (our work­sta­tions are already like this, but our servers are still at Solaris 10u6).