Today I had again some energy to look at why mono fails to build on FreeBSD-cur­rent.

I decided to do a debug-build of mono. This did not work ini­tially, I had to pro­duce some patches.

Does this mean nobody is doing debug builds of mono on FreeBSD?

I have to say, this expe­ri­ence with lang/mono is com­pletely unsatisfying.

Ok, bot­tom line, either the debug build seems to pre­vent a race con­di­tion in most cases (I had a lot less lock­ups for each of the two builds I did).

What­ever it is, I do not care ATM (if the con­fig­ure stuff is look­ing at the archi­tec­ture of the sys­tem, it may be the case that the i386-portbld-freebsdX does not enable some impor­tant stuff which would be enabled when run with i486-portbld-freebsdX or bet­ter). Here are the patches I used in case some­one is inter­ested (warn­ing, copy&paste con­verted tabs to spaces, you also have to apply the map.c (a gen­er­ated file… maybe a touch of the right file would allow to apply this patch in the nor­mal patch stage) related stuff when the build fails, else there is some parser error in mono):

--- mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs.orig       2010-01-29 11:34:00.592323482 +0100
+++ mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs    2010-01-29 11:34:18.540607357 +0100
@@ -57,7 +57,7 @@ namespace Mono.Unix {
 int r = Native.Syscall.waitpid (pid, out status,
 Native.WaitOptions.WNOHANG | Native.WaitOptions.WUNTRACED);
 UnixMarshal.ThrowExceptionForLastErrorIf (r);
-                       return r;
+                       return status;
 }

 public int ExitCode {

--- mono/io-layer/processes.c.orig    2010-01-29 11:36:08.904331535 +0100
+++ mono/io-layer/processes.c 2010-01-29 11:42:21.819159544 +0100
@@ -160,7 +160,7 @@ static gboolean waitfor_pid (gpointer te
 ret = waitpid (process->id, &status, WNOHANG);
 } while (errno == EINTR);

-       if (ret <= 0) {
+       if (ret == 0 || (ret < 0 && errno != ECHILD)) {
 /* Process not ready for wait */
 #ifdef DEBUG
 g_message ("%s: Process %d not ready for waiting for: %s",
@@ -169,6 +169,17 @@ static gboolean waitfor_pid (gpointer te

 return (FALSE);
 }
+
+       if (ret < 0 && errno == ECHILD) {
+#ifdef DEBUG
+               g_message ("%s: Process %d does not exist (anymore)", __func__,
+                          process->id);
+#endif
+               /* Faking the return status. I do not know if it is correct
+                * to assume a successful exit.
+                */
+               status = 0;
+       }

 #ifdef DEBUG
 g_message ("%s: Process %d finished", __func__, ret);

--- mono/metadata/mempool.c.orig      2010-01-29 11:58:16.871052861 +0100
+++ mono/metadata/mempool.c   2010-01-29 12:30:45.143367454 +0100
@@ -212,12 +212,14 @@ mono_backtrace (int size)

         EnterCriticalSection (&mempool_tracing_lock);
         g_print ("Allocating %d bytesn", size);
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         symbols = backtrace (array, BACKTRACE_DEPTH);
         names = backtrace_symbols (array, symbols);
         for (i = 1; i < symbols; ++i) {
                 g_print ("t%sn", names [i]);
         }
         free (names);
+#endif
         LeaveCriticalSection (&mempool_tracing_lock);
 }

--- mono/metadata/metadata.c.orig     2010-01-29 11:59:38.552316989 +0100
+++ mono/metadata/metadata.c  2010-01-29 12:00:43.957337476 +0100
@@ -3673,12 +3673,16 @@ mono_backtrace (int limit)
         void *array[limit];
         char **names;
         int i;
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         backtrace (array, limit);
         names = backtrace_symbols (array, limit);
         for (i =0; i < limit; ++i) {
                 g_print ("t%sn", names [i]);
         }
         g_free (names);
+#else
+       g_print ("No backtrace available.n");
+#endif
 }
 #endif

--- support/map.c.orig        2010-01-29 12:05:22.374653708 +0100
+++ support/map.c 2010-01-29 12:10:29.024412452 +0100
@@ -216,7 +216,7 @@
 #define _cnm_dump(to_t, from) do {} while (0)
 #endif /* def _CNM_DUMP */

-#ifdef DEBUG
+#if defined(DEBUG) && !defined(__FreeBSD__)
 #define _cnm_return_val_if_overflow(to_t,from,val)  G_STMT_START {   
         int     uns = _cnm_integral_type_is_unsigned (to_t);             
         gint64  min = (gint64)  _cnm_integral_type_min (to_t);           

I try to build mono on FreeBSD-cur­rent (it is a depen­dency of some GNOME pro­gram). Unfor­tu­nately this does not work correctly.

What I see are hangs of the build. If I stop the build when it hangs and restart it, it will con­tinue and suc­ceed to process the build steps a lit­tle bit fur­ther, but then it hangs again.

If I ktrace the hang­ing process, I see that there is a call to wait return­ing with the error mes­sage that the child does not exist. Then there is a call to nanosleep.

It looks to me like this process missed some SIGCLD (or is wait­ing for some­thing which did not exist at all), and a loop is wait­ing for a child to exit. This loop prob­a­bly has no proper con­di­tion for the fact that there is no such child (any­more). As such it will stay for­ever in this loop.

So I grepped a litte bit around in mono and found the fol­low­ing code in <mono-src-dir>/mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs:

public void WaitForExit ()
{
    int status;
    int r;
    do {
        r = Native.Syscall.waitpid (pid, out status, (Native.WaitOptions) 0);
    } while (UnixMarshal.ShouldRetrySyscall (r));
    UnixMarshal.ThrowExceptionForLastErrorIf (r);
}

This does look a lit­tle bit as it could be related to the prob­lem I see, but Shoul­dRetrySyscall only returns true if the errno is EINTR. So this looks cor­rect.

I looked a lit­tle bit more at this file and it looks like either I do not under­stand the seman­tic of this lan­guage, or Get­ProcessSta­tus does return the return­value of the wait­pid call instead of the sta­tus (which is not what it shall return to my under­stand­ing). If I am cor­rect, it can not really detect the sta­tus of a process. It would be very bad if such a fun­da­men­tal thing went unno­ticed in mono…  which does not put a good light on the unit-tests (if any) or the gen­eral test­ing of mono. For this rea­son I hope I am wrong.

I did not stop there, as this part does not look like it is the prob­lem. I found the fol­low­ing in mono/io-layer/processes.c:

static gboolean waitfor_pid (gpointer test, gpointer user_data)
{
...
    do {
        ret = waitpid (process->id, &status, WNOHANG);
    } while (errno == EINTR);

    if (ret <= 0) {
        /* Process not ready for wait */
#ifdef DEBUG
        g_message ("%s: Process %d not ready for waiting for: %s",
                   __func__, process->id, g_strerror (errno));
#endif

        return (FALSE);
    }

#ifdef DEBUG
    g_message ("%s: Process %d finished", __func__, ret);
#endif

    process->waited = TRUE;
...
}

And here we have the prob­lem, I think. I changed the (ret <= 0) to  (ret == 0 || (ret < 0 && errno != ECHILD)). This will not really give the cor­rect sta­tus, but at least it should not block any­more and I should be able to see the dif­fer­ence dur­ing the build.

And now after test­ing, I see a dif­fer­ence, but the prob­lem is still there. The wait with ECHILD is gone in the loop, but there is still some loop with a sem­a­phore operation:

62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)
62960 mono     RET   nanosleep 0
62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)

OK, there is more going on. I think some­one with more knowl­edge about mono should have a look at this (do not only look at this semop thing, but also look why it loses a child).

At work we have to use a proxy which requires autho­riza­tion. With pre­vi­ous ver­sions (fire­fox 3.0.x and 3.5.y for each valid x and y) I had the prob­lem that each tab requested to enter the mas­ter pass­word when start­ing fire­fox, to be able to fill in the proxy-auth data (short­cut: fill in only the first request, and for all oth­ers just hit return/OK). So for each tab I had to do some­thing for the master-password, and after that for each tab I also had to con­firm the proxy-auth stuff.

Very annoy­ing! Oh, I should maybe men­tion that as of this writ­ing I have 31 tabs open. Some­times there are more, some­times there are less.

Now with fire­fox 3.6 this is not the case any­more. Yeah! Great! Finally only one time the mas­ter pass­word stuff, and then one time the proxy-auth stuff, and then all tabs proceed.

It took a long time since my first report about this, but now it is finally there. This is the best improve­ment in 3.6 for me.

I exper­i­mented a lit­tle bit with the order of direc­to­ries to backup in tarsnap.

Cur­rently I use the fol­low­ing sort­ing algo­rithm:

1. least fre­quently changed directory-trees first
   Every change — even in meta-data — will affect the fol­low­ing data, as tarsnap is doing the de-duplication in fixed-width blocks (AFAIR 64k).
2. for those directory-trees which change with the same fre­quency: list the big­ger ones first
   Implic­itly I assume that the smaller ones are much smaller than the big­ger ones so that the smaller part which will be backed up will not be noticed because of the big­ger change. For my use cases of tarsnap this is true.
3. if changes in a directory-tree are much much big­ger than any­thing else, but the directory-tree has a medium change-frequency, put it even before less-frequently chang­ing stuff
   I do not want that a small change trig­gers a big backup, but a big backup can con­tain the remain­ing small part.
4. if you backup home direc­to­ries (even root’s one) and they do not con­tain much data, put them before directory-trees which change a lot daily
   I do not want that a login trig­gers the trans­fer of data in other directory-trees which have not changed.

After putting the disks of the 7–sta­ble sys­tem which exhib­ited sta­bil­ity prob­lems into a com­pletely dif­fer­ent sys­tem (it is a rented root-server, not our own hard­ware), the sys­tem now sur­vived more than a day (and still no trace of prob­lems) with the UFS setup. Pre­vi­ously it would crash after some minutes.

The ZFS setup with the changed hard­ware had a prob­lem dur­ing the night before (like always after all my ZFS related changes on this machine), but on this machine I changed all locks in ZFS from shared locks to exclu­sive locks (this extended the uptime from 4 – 6 hours to “until I rebooted the morn­ing after because of hang­ing processes”), so this may be because of this. I do not know yet if we will test the ZFS setup with the pure 7-stable source we use now or not (the goal was to get back a sta­ble sys­tem, instead of play­ing around with unre­lated stuff).

It looks like some kind of hard­ware prob­lem was uncov­ered by updat­ing from 7.1 to 7.2 (and 7-stable sub­se­quently). This new machine has a com­pletely dif­fer­ent chipset, a new CPU and RAM and PSU and … so I do not really know what caused this (but the fact that the pre­vi­ous sys­tem did not rec­og­nize the CPU after replac­ing it with a big­ger one and the obser­va­tion that only shared locks with a spe­cific usage pat­tern where affected lets me point towards miss­ing microc­ode updates…).