The dis­cus­sion about direct and indi­rect depen­den­cies is com­ing up again on the FreeBSD mail­inglists. Seems I should make some blog post about it, maybe it makes this topic more find­able than my post­ings in the mailinglists.

Some def­i­n­i­tions:

• A direct depen­dency from A to B is when program/port A uses sym­bols from library/port B.
• An indi­rect depen­dency from A to C is when program/port A uses sym­bols from library/port B but no sym­bols from library/port C, and library/port B uses sym­bols from library/port C.
• An explicit depen­dency from A to C is when it is a direct or indi­rect depen­dency A to C, and when the compiler-time-linker added an explicit ref­er­ence to C to the program/lib of A.

Ide­ally we have no indi­rect depen­den­cies in the explicit depen­den­cies, only direct depen­den­cies. Unfor­tu­nately in real­ity we also have indi­rect depen­den­cies there. This has at least two causes:

1. libtool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-time-linker is recur­sively resolv­ing dependencies.
2. Some pkg-config setups list indi­rect depen­den­cies as explicit depen­den­cies (IIRC it depends if Requires.private and/or Libs.private is used in the .pc file or not; if it is used, there should be no indi­rect depen­dency appear from this soft­ware, but I am not 100% sure about this).

Three years ago I wrote /usr/ports/Tools/scripts/explicit_lib_depends.sh, it looks at the files of a given port (it needs to be installed), and prints out explicit depen­den­cies. Because of the indi­rect depen­den­cies which could be listed there, this list is not a list of ports which are real depen­den­cies from a source code point of view, but it reflects the link-time real­ity. If a port C shows up there, the port which is checked needs to be rebuild in case the ABI of library/port C changes.

After all the big-impact com­mits (Gnome/gettext/KDE/X11/…) have set­tled now, I took the time to update audio/lame (I iden­ti­fied more than 100 ports with an (implicit) depen­dency on lame, 45 of them needed a portre­vi­sion bump; if I forgot/overlooked some, bump the revi­sion your­self or notify me please). That is the first update of my ports where miwi@ did not beat me in com­mit­ting an update since a year (he has implicit approval to do any­thing he wants with my ports).

I can be happy that he is/was this fast (and that we have such a pro­duc­tive and effi­cient com­mit­ter), or I can be sad that I do not have the time any­more to be faster than I am with such things… or both. Hmmm… I think I will go the happy way.

I have an old sys­tem (only the hard­ware, it runs –cur­rent) which reboots itself from time to time (mostly dur­ing the daily periodic(8) run, but also dur­ing a lot of com­pil­ing (por­tup­grade)). There is no obvi­ous rea­son (no panic) why it is doing this. It could be that there is some hard­ware defect, or some­thing else. It is not impor­tant enough to get a high enough pri­or­ity that I try hard to ana­lyze the prob­lem with this machine. The annoy­ing part is, that some­times after a restart apache does not start. So if this hap­pens, the solu­tion is to login and start the web­server. If the web­server would start each time, nearly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My prag­matic solu­tion (for ser­vices started via a good rc.d script which has a work­ing sta­tus com­mand) is a crontab entry which checks peri­od­i­cally if it is run­ning and which restarts the ser­vice if not. As an exam­ple for apache and an inter­val of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a prob­lem with the ser­vice, a mail with the restart out­put would arrive each time it runs, else only after a reboot for which the ser­vice did not restart.

Today I had again some energy to look at why mono fails to build on FreeBSD–cur­rent.

I decided to do a debug-build of mono. This did not work ini­tially, I had to pro­duce some patches.

Does this mean nobody is doing debug builds of mono on FreeBSD?

I have to say, this expe­ri­ence with lang/mono is com­pletely unsatisfying.

Ok, bot­tom line, either the debug build seems to pre­vent a race con­di­tion in most cases (I had a lot less lock­ups for each of the two builds I did).

What­ever it is, I do not care ATM (if the con­fig­ure stuff is look­ing at the archi­tec­ture of the sys­tem, it may be the case that the i386-portbld-freebsdX does not enable some impor­tant stuff which would be enabled when run with i486-portbld-freebsdX or bet­ter). Here are the patches I used in case some­one is inter­ested (warn­ing, copy&paste con­verted tabs to spaces, you also have to apply the map.c (a gen­er­ated file… maybe a touch of the right file would allow to apply this patch in the nor­mal patch stage) related stuff when the build fails, else there is some parser error in mono):

--- mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs.orig       2010-01-29 11:34:00.592323482 +0100
+++ mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs    2010-01-29 11:34:18.540607357 +0100
@@ -57,7 +57,7 @@ namespace Mono.Unix {
 int r = Native.Syscall.waitpid (pid, out status,
 Native.WaitOptions.WNOHANG | Native.WaitOptions.WUNTRACED);
 UnixMarshal.ThrowExceptionForLastErrorIf (r);
-                       return r;
+                       return status;
 }
 public int ExitCode {
--- mono/io-layer/processes.c.orig    2010-01-29 11:36:08.904331535 +0100
+++ mono/io-layer/processes.c 2010-01-29 11:42:21.819159544 +0100
@@ -160,7 +160,7 @@ static gboolean waitfor_pid (gpointer te
 ret = waitpid (process->id, &status, WNOHANG);
 } while (errno == EINTR);
-       if (ret <= 0) {
+       if (ret == 0 || (ret < 0 && errno != ECHILD)) {
 /* Process not ready for wait */
 #ifdef DEBUG
 g_message ("%s: Process %d not ready for waiting for: %s",
@@ -169,6 +169,17 @@ static gboolean waitfor_pid (gpointer te
 return (FALSE);
 }
+
+       if (ret < 0 && errno == ECHILD) {
+#ifdef DEBUG
+               g_message ("%s: Process %d does not exist (anymore)", __func__,
+                          process->id);
+#endif
+               /* Faking the return status. I do not know if it is correct
+                * to assume a successful exit.
+                */
+               status = 0;
+       }
 #ifdef DEBUG
 g_message ("%s: Process %d finished", __func__, ret);
--- mono/metadata/mempool.c.orig      2010-01-29 11:58:16.871052861 +0100
+++ mono/metadata/mempool.c   2010-01-29 12:30:45.143367454 +0100
@@ -212,12 +212,14 @@ mono_backtrace (int size)
         EnterCriticalSection (&mempool_tracing_lock);
         g_print ("Allocating %d bytesn", size);
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         symbols = backtrace (array, BACKTRACE_DEPTH);
         names = backtrace_symbols (array, symbols);
         for (i = 1; i < symbols; ++i) {
                 g_print ("t%sn", names [i]);
         }
         free (names);
+#endif
         LeaveCriticalSection (&mempool_tracing_lock);
 }
--- mono/metadata/metadata.c.orig     2010-01-29 11:59:38.552316989 +0100
+++ mono/metadata/metadata.c  2010-01-29 12:00:43.957337476 +0100
@@ -3673,12 +3673,16 @@ mono_backtrace (int limit)
         void *array[limit];
         char **names;
         int i;
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         backtrace (array, limit);
         names = backtrace_symbols (array, limit);
         for (i =0; i < limit; ++i) {
                 g_print ("t%sn", names [i]);
         }
         g_free (names);
+#else
+       g_print ("No backtrace available.n");
+#endif
 }
 #endif
--- support/map.c.orig        2010-01-29 12:05:22.374653708 +0100
+++ support/map.c 2010-01-29 12:10:29.024412452 +0100
@@ -216,7 +216,7 @@
 #define _cnm_dump(to_t, from) do {} while (0)
 #endif /* def _CNM_DUMP */
-#ifdef DEBUG
+#if defined(DEBUG) && !defined(__FreeBSD__)
 #define _cnm_return_val_if_overflow(to_t,from,val)  G_STMT_START {  
         int     uns = _cnm_integral_type_is_unsigned (to_t);            
         gint64  min = (gint64)  _cnm_integral_type_min (to_t);           

I try to build mono on FreeBSD–cur­rent (it is a depen­dency of some GNOME pro­gram). Unfor­tu­nately this does not work correctly.

What I see are hangs of the build. If I stop the build when it hangs and restart it, it will con­tinue and suc­ceed to process the build steps a lit­tle bit fur­ther, but then it hangs again.

If I ktrace the hang­ing process, I see that there is a call to wait return­ing with the error mes­sage that the child does not exist. Then there is a call to nanosleep.

It looks to me like this process missed some SIGCLD (or is wait­ing for some­thing which did not exist at all), and a loop is wait­ing for a child to exit. This loop prob­a­bly has no proper con­di­tion for the fact that there is no such child (any­more). As such it will stay for­ever in this loop.

So I grepped a litte bit around in mono and found the fol­low­ing code in <mono-src-dir>/mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs:

public void WaitForExit ()
{
    int status;
    int r;
    do {
        r = Native.Syscall.waitpid (pid, out status, (Native.WaitOptions) 0);
    } while (UnixMarshal.ShouldRetrySyscall (r));
    UnixMarshal.ThrowExceptionForLastErrorIf (r);
}

This does look a lit­tle bit as it could be related to the prob­lem I see, but Shoul­dRetrySyscall only returns true if the errno is EINTR. So this looks cor­rect.

I looked a lit­tle bit more at this file and it looks like either I do not under­stand the seman­tic of this lan­guage, or Get­ProcessSta­tus does return the return­value of the wait­pid call instead of the sta­tus (which is not what it shall return to my under­stand­ing). If I am cor­rect, it can not really detect the sta­tus of a process. It would be very bad if such a fun­da­men­tal thing went unno­ticed in mono…  which does not put a good light on the unit-tests (if any) or the gen­eral test­ing of mono. For this rea­son I hope I am wrong.

I did not stop there, as this part does not look like it is the prob­lem. I found the fol­low­ing in mono/io-layer/processes.c:

static gboolean waitfor_pid (gpointer test, gpointer user_data)
{
...
    do {
        ret = waitpid (process->id, &status, WNOHANG);
    } while (errno == EINTR);
    if (ret <= 0) {
        /* Process not ready for wait */
#ifdef DEBUG
        g_message ("%s: Process %d not ready for waiting for: %s",
                   __func__, process->id, g_strerror (errno));
#endif
        return (FALSE);
    }
#ifdef DEBUG
    g_message ("%s: Process %d finished", __func__, ret);
#endif
    process->waited = TRUE;
...
}

And here we have the prob­lem, I think. I changed the (ret <= 0) to  (ret == 0 || (ret < 0 && errno != ECHILD)). This will not really give the cor­rect sta­tus, but at least it should not block any­more and I should be able to see the dif­fer­ence dur­ing the build.

And now after test­ing, I see a dif­fer­ence, but the prob­lem is still there. The wait with ECHILD is gone in the loop, but there is still some loop with a sem­a­phore operation:

62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)
62960 mono     RET   nanosleep 0
62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)

OK, there is more going on. I think some­one with more knowl­edge about mono should have a look at this (do not only look at this semop thing, but also look why it loses a child).