I try to build mono on FreeBSD-cur­rent (it is a depen­dency of some GNOME pro­gram). Unfor­tu­nately this does not work correctly.

What I see are hangs of the build. If I stop the build when it hangs and restart it, it will con­tinue and suc­ceed to process the build steps a lit­tle bit fur­ther, but then it hangs again.

If I ktrace the hang­ing process, I see that there is a call to wait return­ing with the error mes­sage that the child does not exist. Then there is a call to nanosleep.

It looks to me like this process missed some SIGCLD (or is wait­ing for some­thing which did not exist at all), and a loop is wait­ing for a child to exit. This loop prob­a­bly has no proper con­di­tion for the fact that there is no such child (any­more). As such it will stay for­ever in this loop.

So I grepped a litte bit around in mono and found the fol­low­ing code in <mono-src-dir>/mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs:

public void WaitForExit ()
{
    int status;
    int r;
    do {
        r = Native.Syscall.waitpid (pid, out status, (Native.WaitOptions) 0);
    } while (UnixMarshal.ShouldRetrySyscall (r));
    UnixMarshal.ThrowExceptionForLastErrorIf (r);
}

This does look a lit­tle bit as it could be related to the prob­lem I see, but Shoul­dRetrySyscall only returns true if the errno is EINTR. So this looks cor­rect.

I looked a lit­tle bit more at this file and it looks like either I do not under­stand the seman­tic of this lan­guage, or Get­ProcessSta­tus does return the return­value of the wait­pid call instead of the sta­tus (which is not what it shall return to my under­stand­ing). If I am cor­rect, it can not really detect the sta­tus of a process. It would be very bad if such a fun­da­men­tal thing went unno­ticed in mono…  which does not put a good light on the unit-tests (if any) or the gen­eral test­ing of mono. For this rea­son I hope I am wrong.

I did not stop there, as this part does not look like it is the prob­lem. I found the fol­low­ing in mono/io-layer/processes.c:

static gboolean waitfor_pid (gpointer test, gpointer user_data)
{
...
    do {
        ret = waitpid (process->id, &status, WNOHANG);
    } while (errno == EINTR);

    if (ret <= 0) {
        /* Process not ready for wait */
#ifdef DEBUG
        g_message ("%s: Process %d not ready for waiting for: %s",
                   __func__, process->id, g_strerror (errno));
#endif

        return (FALSE);
    }

#ifdef DEBUG
    g_message ("%s: Process %d finished", __func__, ret);
#endif

    process->waited = TRUE;
...
}

And here we have the prob­lem, I think. I changed the (ret <= 0) to  (ret == 0 || (ret < 0 && errno != ECHILD)). This will not really give the cor­rect sta­tus, but at least it should not block any­more and I should be able to see the dif­fer­ence dur­ing the build.

And now after test­ing, I see a dif­fer­ence, but the prob­lem is still there. The wait with ECHILD is gone in the loop, but there is still some loop with a sem­a­phore operation:

62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)
62960 mono     RET   nanosleep 0
62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)

OK, there is more going on. I think some­one with more knowl­edge about mono should have a look at this (do not only look at this semop thing, but also look why it loses a child).

I exper­i­mented a lit­tle bit with the order of direc­to­ries to backup in tarsnap.

Cur­rently I use the fol­low­ing sort­ing algo­rithm:

1. least fre­quently changed directory-trees first
   Every change — even in meta-data — will affect the fol­low­ing data, as tarsnap is doing the de-duplication in fixed-width blocks (AFAIR 64k).
2. for those directory-trees which change with the same fre­quency: list the big­ger ones first
   Implic­itly I assume that the smaller ones are much smaller than the big­ger ones so that the smaller part which will be backed up will not be noticed because of the big­ger change. For my use cases of tarsnap this is true.
3. if changes in a directory-tree are much much big­ger than any­thing else, but the directory-tree has a medium change-frequency, put it even before less-frequently chang­ing stuff
   I do not want that a small change trig­gers a big backup, but a big backup can con­tain the remain­ing small part.
4. if you backup home direc­to­ries (even root’s one) and they do not con­tain much data, put them before directory-trees which change a lot daily
   I do not want that a login trig­gers the trans­fer of data in other directory-trees which have not changed.

Unfor­tu­nately you can not chose…

So, I am now on the sofa, cov­ered a lot (a flu, I even have no voice any­more; before I left work a female coworker told that her hus­band would prob­a­bly be happy if this would hap­pen to her…  ) and med­ica­tion and water are not far away on the table.

The good thing with the cur­rent tech­nol­ogy is, that you can still be a lit­tle bit pro­duc­tive (depend­ing on the illness).

As you can read this, it means I have my net­book with me, so that I can take care about some sim­ple things.

I am in the process of prepar­ing the import of code which makes v4l devices usable in the lin­ux­u­la­tor. Basi­cally this means you can use your web­cam in skype (tested by the sub­mit­ter of the patch on amd64).

This is not a “apply patch and com­mit” thing, because the orig­i­nal videodev.h (with some mod­i­fi­ca­tions) is used. I was seek­ing the OK from core@ for this. As there is no license in the header, and the orig­i­nal author (Alan Cox, the linux one, not our FreeBSD one) gave per­mis­sions to use it, core@ is OK with the import.

I intent to do a ven­dor import of the linux header (pre­pared today, together with some readme which explains where it comes from and some stuff to show that we are on the safe side regard­ing legal stuff), and then I want to copy this over to the lin­ux­u­la­tor as linux_videodev.h and com­mit the patch (prob­a­bly a lit­tle bit mod­i­fied in a few places). My plan is to com­mit it this week. Peo­ple which already want to play around with it now can have a look at the emu­la­tion mail­inglist, a link to the patch is posted there.

With the header being in a ven­dor branch, inter­ested peo­ple could then start to sub­mit new BSD licensed dri­vers or mod­ify exist­ing dri­vers which make use of the v4l inter­face, but I let the import of the header into the FreeBSD include direc­tory up to the per­son which wants to com­mit the first native FreeBSD-v4l support.

When such native FreeBSD-v4l sup­port is com­mit­ted, the lin­ux­u­la­tor code needs to be revised.

A lot of peo­ple do not do back­ups at home. Unfor­tu­nately this includes me. So far I was lucky that noth­ing bad hap­pened (or that I was able to get back all via manip­u­lat­ing on-disk-data by hand). For me this is mostly because of the price and com­plex­ity (num­ber of backup media involved) for local back­ups. It also means that I only need a backup for the case of a disk-crash, not because I need to recover a file because I acci­dently deleted it (yes, I am very good at not loos­ing data at home… maybe this is also the rea­son why I have so much data).

I have about 1 TB of raw disk space at home (two times raidz1). Not all of this needs a backup (or is used at all), for exam­ple the base sys­tem files, /usr/local, the ports dis­t­files and pack­ages all do not need a backup, but my cam­era fotos (cur­rently 8 GB) should be included in a backup, my home should be included in a backup, my local stag­ing area for the con­tent of my web­server should be included in a backup, my mails (cur­rently 800 MB) should be included in the backup, local patches to FreeBSD should be included in the backup, …

So let us cal­cu­late with about 200 GB of more (mails, fotos, pri­vate videos) or less (MP3s from my own CDs) impor­tant data for a full backup. Most of it will not change often (MP3s, pri­vate videos, fotos), so once a month should be ok, and some (e.g. my mails) change daily, so an incre­men­tal each day and a full once a week would be inter­est­ing for me. When I do such a backup, I do not want to shuf­fle around tapes a lot. Maybe one or two times (s0 2 – 3 tapes) would be ok. So I am talk­ing about 80 – 200 GB per tape. The prices for this range from 700 EUR to 1300 EUR just for the tape drive. I have also seen a tape drive from Iomega for about 400 EUR (120 GB per tape), but some­how this sounds like a not so trust­worty solu­tion to me (I do not know why, it is just a feel­ing, maybe I am a lit­tle bit biased because of the high-end solu­tion at work, if some­one knows more about those Iomega dri­ves and tapes in a long term or high-useage envi­ron­ment, please write a comment).

When I take the cur­rent price of tarsnap into account, I come to about $60 per month for the stor­age, and again $60 for the ini­tial trans­fer of 200 GB. This assumes the 200 GB are not very com­press­able. With each incre­men­tal backup (changed files mat­ter, not a diff between the files), I assume about 10 GB per month of change (= $3 per month). Yes, this is most prob­a­bly too much, but I try to cal­cu­late a worst case sce­nario. So this sums up to $60 once and $63 per month. I do not take into account my too slow upstream band­with. When I assume a 1500 EUR tape drive plus tapes plus clean­ing car­tridge, we are talk­ing about some­thing like 2 – 3 years of stor­ing the backup in tarsnap (with the Iomega drive this would be just one year).

When I assume that I over­es­ti­mated every­thing with a fac­tor of 2, this means 5 – 6 years of using tarsnap vs. buy­ing an expen­sive tape drive. Now the ques­tion is, will the tape drive sur­vive this long, will I be able to use such a drive in new hard­ware  in 6 years, will tarsnap sur­vive this long, and will it stay at the same or bet­ter price level (this is also influ­enced by the price of the stor­age provider).

Another solu­tion would be to go with a mixed set­ting, e.g. an 1 TB hard-disk (less than 150 EUR  for the hard-disk (with 7 years war­ranty) and an exter­nal case) to do the long term stor­age of high-volume but long-term-stable stuff (fotos, videos, music), and use tarsnap for the low-volume fast chang­ing stuff (con­fig files, mails). This way the amount of money for the really impor­tant things is not much, I would expect less than 1 GB com­pressed (= less than an EUR per month). This also sounds like a solu­tion which allows me to be lazy (the impor­tant stuff can be auto­mated, the nice to have stuff can be done from time to time as needed).

I think I should ask Collin if he has a way to receive SWIFT (IBAN/BIC) trans­fers for tarsnap…