The techniques in the paper can not be applied to the irregexp engine, but maybe could help to speed up awk, egrep and similar programs.

I think it would be interesting to compare those recent developments to what we have in FreeBSD, and if they are faster, to see if it is possible to improve the FreeBSD implementation based upon them (either by writing new code, or by importing existing code, depending on the corresponding license and the language the code is written in).

Maybe a candidate for the GSoC?

I had the impression that it is about the invalidation of at least parts of one of the patents. Maybe they have a client which was sued for infringement. Unfortunately for them I have absolutely no clue what is inside the MP3-patents (I am/was taking care about the “glue” in LAME) and the phone call we had was just some hours before I went into holiday. I referred him to two other developers of the LAME-project which not only should have better knowledge about the parts the lawyer is interested in, but also where probably not in holiday.

We also had a little chat about patents in general, and my opinion was that software patents are not that useful. In the IT world 3 years is a lot of time, technology is already overtaken by new developments most of the time after this time. When assuming that developing something new depending on some technology seen at another place takes at least about 1 year (do not hit me because of this rough estimation without specifying the size of the project or the quality requirements), a software patent which is valid longer than 5 years is more than enough in my opinion. Any company which was not able to make some money with it during this time made something wrong, and blocking the competition because of this is not really a good idea from my point of an user of technology. As an user I want advancements. And as an open source developer I try to produce my own advancements when I can not get them from somewhere else. In this light software patents are not doing good for the “advancement of the human race”.

The lawyer did not try to convince me to the opposite. Either has was too polite, did not care about it, or he silently agrees. He  told me he wants to stay in touch with me in some way regarding Open Source and patents. I did not object to this.

As I was curious about the state of this, I contacted the lawyer about it, and the current outcome is not bad. Previously a lot of tries (by other lawyers in the same German court) failed to fight against the particular patent. This time the court did not follow his previous rulings but told that this issue needs to be investigated again (at least this is how I understand this — beware, I am not a lawyer). Maybe we can see a result this year.

The disks are connected to an ICH 5 controller, so an obvious improvement would be to either buy a controller for the PCI slot which is able to do NCQ with the SATA disks (a siis(4) based one is not cheap), or to buy a new system which comes with a chipset which knows how to do NCQ (this would mean new RAM, new CPU, new MB and maybe even a new PSU). A new controller is a little bit expensive for the old system which I want to tune. A new system would be nice, and reading about the specs of new systems lets me want to get a Core i5 system. The problem is that I think the current offers of mainboards for this are far from good. The system should be a little bit future proof, as I would like to use it for about 5 years or more (the current system is somewhere between 5–6 years old). This means it should have SATA-3 and USB 3, but when I look at what is offered currently it looks like there are only beta-versions of hardware with SATA-3 and USB 3 support available on the marked (according to tests there is a lot of variance of the max speed the controllers are able to achieve, bugs in the BIOS, or the  controllers are attached to a slow bus which prevents to use the full bandwidth). So it will not be a new system soon.

As I had a 1GB USB-stick around, I decided to attach it to the one of the EHCI USB ports and use it as a cache device for ZFS. If someone wants to try this too, be careful with the USB ports. My mainboard has only 2 USB ports connected to an EHCI, the rest are UHCI ones. This means that only 2 USB ports are fast (sort of… 40 MBit/s), the rest is only usable for slow things like a mouse, keyboard or a serial line.

Be warned, this will not give you a lot of bandwidth (if you have a fast USB stick, the 40MBit/s of the EHCI are the limit which prevent a big streaming bandwidth), but the latency of the cache device is great when doing small random IO. When I do a gstat and have a look how long a read operation takes for each involved device, I see something between 3 msec and 20 msec for the harddisks (depending if they are reading something at the current head position, or if the harddisk needs to seek around a lot). For the cache device (the USB stick) I see something between around 1 mssec and 5 msec. That is 1/3th to 1/4th of the latency of the harddisks.

With a “zfs send” I see about 300 IOops per harddisk (3 disks in a RAIDZ). Obviously this is an optimum streaming case where the disks do not need to seek around a lot. You see this in the low latency, it is about 2 msec in this case. In the random-read case, like for example when you run a find, the disks can not keep this amount of IOops, as they need to seek around. And here the USB-stick shines. I’ve seen upto 1600 IOops on it during running a find (if the corresponding data is in the cache, off course). This was with something between 0.5 and 0.8 msec of latency.

This is the machine at home which is taking care about my mails (incoming and outgoing SMTP, IMAP and Webmail), has a squid proxy and acts as a file server. There are not many users (just me and my wife) and there is no regular usage pattern for all those services. Because of this I did not do any benchmark to see how much time I can gain with various workloads (and I am not interested in some artificial performance numbers of my webmail session, as the browsing experience is highly subjective in this case). For this system a 1 GB USB stick (which was just collecting dust before) seems to be a cheap way to improve the response time for often used small data. When I use the webmail interface now, my subjective impression is, that it is faster. I am talking about listing emails (subject, date, sender, size) and displaying the content of some emails. FYI, my maildir storage has 849 MB with 35000 files in 91 folders.

Bottom line is: do not expect a lot of bandwidth increase with this, but if you have a workload which generates random read requests and you want to decrease the read latency, it could be a cheap solution to add a (big) USB stick as a cache device.

One application (not off-the-shelf software) is not performing good. The problem is that the design of the application is far from good (auto-commit is used, and the Oracle DB is doing too much writes for what the application is supposed to do because of this). During helping our DBAs in their performance analysis (the vendor of the application is telling our hardware is not fast enough and I had to provide some numbers to show that this is not the case and they need to improve the software as it does not comply to the performance requirements they got before developing the application) I noticed that the filesystem where the DB and the application are located (a ZFS if someone is interested) is doing sometimes 1.200 IO (write) operations per second (to write about 100 MB). Yeah, that is a lot of IOops our SAN is able to do! Unfortunately too expensive to buy for use at home.

Another application (nagios 3.0) was generating a lot of major faults (caused by a lot of fork()s for the checks). It is a SunFire V890, and the highest number of MF per second I have seen on this machine was about 27.000. It never went below 10.000. On average maybe somewhere between 15.000 and 20.000. My Solaris–Desktop (an Ultra 20) is generating maybe several hundred MF if a lot is going on (most of the time is does not generate much). Nobody can say the V890 is not used… Oh, yes, I suggested to enable the nagios config setting for large sites, now the major faults are around 0–10.000 and the machine is not that stressed anymore. The next step is probably to have a look at the ancient probes (migrated from the big brother setup which was there several years before) and reduce the number of forks they do.

I decided to do a debug-build of mono. This did not work initially, I had to produce some patches.

Does this mean nobody is doing debug builds of mono on FreeBSD?

I have to say, this experience with lang/mono is completely unsatisfying.

Ok, bottom line, either the debug build seems to prevent a race condition in most cases (I had a lot less lockups for each of the two builds I did).

Whatever it is, I do not care ATM (if the configure stuff is looking at the architecture of the system, it may be the case that the i386-portbld-freebsdX does not enable some important stuff which would be enabled when run with i486-portbld-freebsdX or better). Here are the patches I used in case someone is interested (warning, copy&paste converted tabs to spaces, you also have to apply the map.c (a generated file… maybe a touch of the right file would allow to apply this patch in the normal patch stage) related stuff when the build fails, else there is some parser error in mono):

--- mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs.orig       2010-01-29 11:34:00.592323482 +0100
+++ mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs    2010-01-29 11:34:18.540607357 +0100
@@ -57,7 +57,7 @@ namespace Mono.Unix {
 int r = Native.Syscall.waitpid (pid, out status,
 Native.WaitOptions.WNOHANG | Native.WaitOptions.WUNTRACED);
 UnixMarshal.ThrowExceptionForLastErrorIf (r);
-                       return r;
+                       return status;
 }

 public int ExitCode {

--- mono/io-layer/processes.c.orig    2010-01-29 11:36:08.904331535 +0100
+++ mono/io-layer/processes.c 2010-01-29 11:42:21.819159544 +0100
@@ -160,7 +160,7 @@ static gboolean waitfor_pid (gpointer te
 ret = waitpid (process->id, &status, WNOHANG);
 } while (errno == EINTR);

-       if (ret <= 0) {
+       if (ret == 0 || (ret < 0 && errno != ECHILD)) {
 /* Process not ready for wait */
 #ifdef DEBUG
 g_message ("%s: Process %d not ready for waiting for: %s",
@@ -169,6 +169,17 @@ static gboolean waitfor_pid (gpointer te

 return (FALSE);
 }
+
+       if (ret < 0 && errno == ECHILD) {
+#ifdef DEBUG
+               g_message ("%s: Process %d does not exist (anymore)", __func__,
+                          process->id);
+#endif
+               /* Faking the return status. I do not know if it is correct
+                * to assume a successful exit.
+                */
+               status = 0;
+       }

 #ifdef DEBUG
 g_message ("%s: Process %d finished", __func__, ret);

--- mono/metadata/mempool.c.orig      2010-01-29 11:58:16.871052861 +0100
+++ mono/metadata/mempool.c   2010-01-29 12:30:45.143367454 +0100
@@ -212,12 +212,14 @@ mono_backtrace (int size)

         EnterCriticalSection (&mempool_tracing_lock);
         g_print ("Allocating %d bytesn", size);
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         symbols = backtrace (array, BACKTRACE_DEPTH);
         names = backtrace_symbols (array, symbols);
         for (i = 1; i < symbols; ++i) {
                 g_print ("t%sn", names [i]);
         }
         free (names);
+#endif
         LeaveCriticalSection (&mempool_tracing_lock);
 }

--- mono/metadata/metadata.c.orig     2010-01-29 11:59:38.552316989 +0100
+++ mono/metadata/metadata.c  2010-01-29 12:00:43.957337476 +0100
@@ -3673,12 +3673,16 @@ mono_backtrace (int limit)
         void *array[limit];
         char **names;
         int i;
+#if defined(HAVE_BACKTRACE_SYMBOLS)
         backtrace (array, limit);
         names = backtrace_symbols (array, limit);
         for (i =0; i < limit; ++i) {
                 g_print ("t%sn", names [i]);
         }
         g_free (names);
+#else
+       g_print ("No backtrace available.n");
+#endif
 }
 #endif

--- support/map.c.orig        2010-01-29 12:05:22.374653708 +0100
+++ support/map.c 2010-01-29 12:10:29.024412452 +0100
@@ -216,7 +216,7 @@
 #define _cnm_dump(to_t, from) do {} while (0)
 #endif /* def _CNM_DUMP */

-#ifdef DEBUG
+#if defined(DEBUG) && !defined(__FreeBSD__)
 #define _cnm_return_val_if_overflow(to_t,from,val)  G_STMT_START {   
         int     uns = _cnm_integral_type_is_unsigned (to_t);             
         gint64  min = (gint64)  _cnm_integral_type_min (to_t);           

What I see are hangs of the build. If I stop the build when it hangs and restart it, it will continue and succeed to process the build steps a little bit further, but then it hangs again.

If I ktrace the hanging process, I see that there is a call to wait returning with the error message that the child does not exist. Then there is a call to nanosleep.

It looks to me like this process missed some SIGCLD (or is waiting for something which did not exist at all), and a loop is waiting for a child to exit. This loop probably has no proper condition for the fact that there is no such child (anymore). As such it will stay forever in this loop.

So I grepped a litte bit around in mono and found the following code in <mono-src-dir>/mcs/class/Mono.Posix/Mono.Unix/UnixProcess.cs:

public void WaitForExit ()
{
    int status;
    int r;
    do {
        r = Native.Syscall.waitpid (pid, out status, (Native.WaitOptions) 0);
    } while (UnixMarshal.ShouldRetrySyscall (r));
    UnixMarshal.ThrowExceptionForLastErrorIf (r);
}

This does look a little bit as it could be related to the problem I see, but ShouldRetrySyscall only returns true if the errno is EINTR. So this looks correct.

I looked a little bit more at this file and it looks like either I do not understand the semantic of this language, or GetProcessStatus does return the returnvalue of the waitpid call instead of the status (which is not what it shall return to my understanding). If I am correct, it can not really detect the status of a process. It would be very bad if such a fundamental thing went unnoticed in mono…  which does not put a good light on the unit-tests (if any) or the general testing of mono. For this reason I hope I am wrong.

I did not stop there, as this part does not look like it is the problem. I found the following in mono/io-layer/processes.c:

static gboolean waitfor_pid (gpointer test, gpointer user_data)
{
...
    do {
        ret = waitpid (process->id, &status, WNOHANG);
    } while (errno == EINTR);

    if (ret <= 0) {
        /* Process not ready for wait */
#ifdef DEBUG
        g_message ("%s: Process %d not ready for waiting for: %s",
                   __func__, process->id, g_strerror (errno));
#endif

        return (FALSE);
    }

#ifdef DEBUG
    g_message ("%s: Process %d finished", __func__, ret);
#endif

    process->waited = TRUE;
...
}

And here we have the problem, I think. I changed the (ret <= 0) to  (ret == 0 || (ret < 0 && errno != ECHILD)). This will not really give the correct status, but at least it should not block anymore and I should be able to see the difference during the build.

And now after testing, I see a difference, but the problem is still there. The wait with ECHILD is gone in the loop, but there is still some loop with a semaphore operation:

62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)
62960 mono     RET   nanosleep 0
62960 mono     CALL  clock_gettime(0xd,0xbf9feef8)
62960 mono     RET   clock_gettime 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  semop(0x20c0000,0xbf9feef6,0x1)
62960 mono     RET   semop 0
62960 mono     CALL  nanosleep(0xbf9fef84,0)

OK, there is more going on. I think someone with more knowledge about mono should have a look at this (do not only look at this semop thing, but also look why it loses a child).

Very annoying! Oh, I should maybe mention that as of this writing I have 31 tabs open. Sometimes there are more, sometimes there are less.

Now with firefox 3.6 this is not the case anymore. Yeah! Great! Finally only one time the master password stuff, and then one time the proxy-auth stuff, and then all tabs proceed.

It took a long time since my first report about this, but now it is finally there. This is the best improvement in 3.6 for me.

Currently I use the following sorting algorithm:

1. least frequently changed directory-trees first
   Every change — even in meta-data — will affect the following data, as tarsnap is doing the de-duplication in fixed-width blocks (AFAIR 64k).
2. for those directory-trees which change with the same frequency: list the bigger ones first
   Implicitly I assume that the smaller ones are much smaller than the bigger ones so that the smaller part which will be backed up will not be noticed because of the bigger change. For my use cases of tarsnap this is true.
3. if changes in a directory-tree are much much bigger than anything else, but the directory-tree has a medium change-frequency, put it even before less-frequently changing stuff
   I do not want that a small change triggers a big backup, but a big backup can contain the remaining small part.
4. if you backup home directories (even root’s one) and they do not contain much data, put them before directory-trees which change a lot daily
   I do not want that a login triggers the transfer of data in other directory-trees which have not changed.

The ZFS setup with the changed hardware had a problem during the night before (like always after all my ZFS related changes on this machine), but on this machine I changed all locks in ZFS from shared locks to exclusive locks (this extended the uptime from 4–6 hours to “until I rebooted the morning after because of hanging processes”), so this may be because of this. I do not know yet if we will test the ZFS setup with the pure 7-stable source we use now or not (the goal was to get back a stable system, instead of playing around with unrelated stuff).

It looks like some kind of hardware problem was uncovered by updating from 7.1 to 7.2 (and 7-stable subsequently). This new machine has a completely different chipset, a new CPU and RAM and PSU and … so I do not really know what caused this (but the fact that the previous system did not recognize the CPU after replacing it with a bigger one and the observation that only shared locks with a specific usage pattern where affected lets me point towards missing microcode updates…).