Google’s new RE engine

I stum­bled over Google’s new RE engine. Unfor­tu­nate­ly it is not han­dling back­ref­er­ences, so it is not a drop-in replace­ment for the reg­u­lar expres­sions code in FreeB­SD. It has a POSIX mode, but this only seems to be enough for the egrep syn­tax. For peo­ple which need back­ref­er­ences, they refer to the Google Chrome’s RE engine irreg­exp which in turn ref­er­ences a paper from 2007 which is titled Reg­u­lar Expres­sion Match­ing Can Be Sim­ple And Fast.

The tech­niques in the paper can not be applied to the irreg­exp engine, but maybe could help to speed up awk, egrep and sim­i­lar programs.

I think it would be inter­est­ing to com­pare those recent devel­op­ments to what we have in FreeB­SD, and if they are faster, to see if it is pos­si­ble to improve the FreeB­SD imple­men­ta­tion based upon them (either by writ­ing new code, or by import­ing exist­ing code, depend­ing on the cor­re­spond­ing license and the lan­guage the code is writ­ten in).

Maybe a can­di­date for the GSoC?

FreeNAS & Sen­sors for FreeBSD

This WE I was told that FreeNAS seems to want to move from FreeB­SD to Lin­ux (since then it seems there could be a lin­ux and a FreeB­SD ver­sion). One of the rea­sons seems to be a miss­ing sen­sors framework.

As I was com­mit­ting a port of the OpenB­SD sen­sors frame­work (pro­duced as part of the Google Sum­mer of Code 2007) to FreeB­SD and had to remove it after­wards because one com­mit­ter com­plained very loud­ly, I was asked what the sta­tus of this is.

The short sta­tus is: Nobody is doing some­thing about it.

Before I explain the long sta­tus, I give  a short overview what this sen­sors frame­work is:

  • a ker­nel API which allows to add sensors
  • an inter­face for the user­land to query the sen­sor data
  • some basic user­land code to show and log the sen­sor info

The API and the query inter­face are more or less inde­pen­dent. For the user­land code it was more a log­ging infra­struc­ture than a real mon­i­tor­ing solu­tion. The rea­son was the real mon­i­tor­ing solu­tions already exist (Nagios, snm­pd, …) and can be adapt­ed to query the sen­sors. Ide­al­ly a query in user­land should be han­dled by a library instead of direct­ly access­ing the sysctl inter­face, this way the kernel<->userland inter­face would be abstract­ed away (and could b replaced as needs arise). This was not done, it was some­thing to be done lat­er (Rome was not build in a day).

The user­land inter­face also only cared about dumb sen­sors (those which you need to query man­u­al­ly to get the infor­ma­tion), smart sen­sors (those which are able to send events them­self) where not tak­en care about in the sense of real­ly send­ing sensor-triggered events, but the ker­nel API allowed to add such sen­sors. The sysctl inter­face has no way of send­ing events, but FreeB­SD already has an event inter­face (devd is tak­ing care about it). It would have been not a prob­lem to send events via this chan­nel and let an user­land library take care about the deliv­ery togeth­er with oth­er sensor-data in userland.

And now the long sta­tus is:

PHK com­plained loud­ly about it. First he said he did not look at it but he com­plained that is not good regard­less. After a lot of nag­ging from me he had a look at it and was not hap­py about the time stuff in it (short: the FreeB­SD time­counter code is bet­ter). This was not a prob­lem in my opin­ion, we could have dis­abled this part with­out prob­lems. After such an offer from me, he com­plained that the sen­sors frame­work uses the sysctl inter­face instead of an entry in /dev.

At this point in time already sev­er­al user­land util­i­ties used the sysctl frame­work to query for sta­tus data in the ker­nel. So there was already prece­dence for such an use of it. Lat­er some more such uses where added too (e.g. the proc­stat stuff by core team mem­ber Robert Watson).

I saved some of the cor­re­spond­ing mails (to pub­lic mail­ing lists) in a mbox file, read the mess your­self if you want.

The bot­tom line is: Sev­er­al com­mit­ters (even some which we could call high pro­file com­mit­ters) told me that they do not see a prob­lem in the use of the sysctl inter­face. They do not seem to want to tell it in pub­lic (nobody of them voiced their opin­ion in the thread, so do not ask me who those peo­ple are). I am not inter­est­ed in invest­ing more of my spare time into fight­ing wind­mills (it looks like this to me).

So, if some­one is inter­st­ed in the code, r172631 has it. In the per­force repos­i­to­ry you can maybe find some sen­sors. I think most of it can still be used with­out much changes.

If some­one tries it with a more recent FreeB­SD, please drop me a note if it just applies fine, or a patch (or an URL to it) if it needs some mod­i­fi­ca­tions. Who knows, maybe in a future project it may be use­ful for me.

If there is enough inter­est by sev­er­al peo­ple, I can even put up a wiki page where those peo­ple can coor­di­nate, but that is most prob­a­bly all I am will­ing to invest fur­ther into this (at least in my unpaid time).

Some more WP plugins

Addi­tion­al­ly to the WP plu­g­ins I already talked about, I installed some more since then:

  • aLinks: auto­mat­ic link gen­er­a­tion based upon rules/modules/keywords
  • All in one SEO Pack: Search Engine Opti­miza­tion, e.g. auto­mat­ic meta tag gen­er­a­tion and more
  • Glob­al Trans­la­tor: auto­mat­ic machine-translation of your posts into oth­er lan­guages (see the coun­try flags in the sidebar)
  • Google XML Sitemaps: auto­mat­i­cal­ly gen­er­ates a sitemap and announces the update (after a post) to sev­er­al searchengines
  • http:BL word­press plu­g­in: checks vis­i­tors against the Project Hon­ey Pot black­list (email har­vest­ing) and rejects access
  • InfoLink: adds a but­ton to look up marked text in the edi­tor and link the markup to the first result if desired
  • One-Time Pass­word: allows to use one-time pass­words (RFC 2289) for the WP login
  • Smartlink­er: anoth­er but­ton for the edi­tor regard­ing auto­mat­ic link­ing (gives oth­er auto­mat­ic links than the oth­er auto­mat­ic link­ing plugins)
  • Update Noti­fi­er: sends me a mail when there is a update for WP
  • WP-Print: print­er friend­ly page of the post­ing (see the link below the title of this post), I had to patch it to not print the links from the WP AJAX Edit Com­ments plugin
  • WP AJAX Edit Com­ments: pro­vides an AJAX edit inter­face for comments

Catch­ing up… GSoC 2007

We got a lot of good pro­pos­als. Google is will­ing to give us a very nice amount of stu­dents. We did­n’t expect­ed this much. Thanks!

Now we need to rate the stu­dent appli­ca­tions and find suit­able men­tors… not that easy. It’s easy for the strongest pro­pos­als, but for the rest I expect that there will be some shuf­fling around until the very end.

Inter­est­ing Sum­mer for FreeBSD

Now that we have (or “soon will have”) an offi­cial FreeB­SD blog, I decid­ed to give this kind of elec­tron­ic and pub­lic diary a try. At least to report the sta­tus and progress of some FreeB­SD relat­ed projects I’m involved in.

A lot hap­pened in the last days. We’re past the dead­line of the Google Sum­mer of Code rat­ing process and now the lucky stu­dents are cho­sen. It was­n’t easy. We had more than 120 pro­pos­als (out of ~6400). From those we where will­ing to men­tor 40 – 50 (based upon our resources and the qual­i­ty of the pro­pos­als). Google grant­ed 14 to us (a big thank you to Google!). I expect an offi­cial announce­ment “soon”. Hope­ful­ly some of those stu­dents Google was­n’t able to fund are will­ing to work with us regard­less of the mon­ey, there are a lot of very nice pro­pos­als (I will add some items to the ideas list based upon them lat­er). We’re at least will­ing to pro­vide the same amount of men­tor­ship as if they where selected.

I will work with two stu­dents. One of them will work on sync­ing the OSS API from recent releas­es from 4Front with our sound sys­tem. The oth­er one will work on improv­ing the lin­ux­o­la­tor. I will announce this on the cor­re­spond­ing mail­inglists after the offi­cial announce­ment. ATM we need to do some admin­is­tra­tive stuff (hand­ing out access to the wiki and per­force, set­ting up email aliases, …).

I already mailed some gen­er­al guides to the stu­dents (note to men­tors: it’s in the wiki on the “hid­den” and restrict­ed to men­tors SoC page). For the sound stuff I don’t have a nice TODO list, but luck­i­ly the stu­dent already inves­ti­gat­ed the 4Front stuff for the pro­pos­al and is eager to start the work (basi­cal­ly this gives us the user­land inter­face for multi-channel mix­er sup­port). Regard­ing the lin­ux­o­la­tor improve­ments the stu­dent has a lit­tle bit less luck. I com­piled a nice TODO list which is based upon his own pro­pos­al and all the var­i­ous things I know about the lin­ux­o­la­tor (PR’s, mes­sages on emulation@, pri­vate con­ver­sa­tions, things we noticed while work­ing on the update of the lin­ux base to a recent Fedo­ra Core, …). I will be impressed if he man­ages to do every­thing on the TODO list (don’t wor­ry, he knows that not every­thing has to be done to declare “suc­cess” to Google).

Both of them may be can­di­dates for a com­mit bit (not with­in the SoC, but maybe lat­er), both are eager to do the work, inter­est­ed, moti­vat­ed, and don’t need hand-holding.

Fur­ther news in the area of the sound sys­tem: Ariff told me he is work­ing on multi-channel and endi­aness issues/support, and I may be able to com­mit two more sound dri­vers to the tree. One dri­ver is the emu10kx dri­ver cur­rent­ly avail­able in the Ports Col­lec­tion, and the oth­er one is a dri­ver for some envy24 chips. Cur­rent­ly I’m in con­tact with the author of the emu10kx dri­ver and a vol­un­teer who wants to improve an exist­ing envy24 dri­ver (author of the dri­ver con­tact­ed; since it’s BSD licensed, this is a “don’t be evil” action).

And some news regard­ing the user­land part of the lin­ux­o­la­tor: We’re wait­ing for a repo-copy form linux_base-fc3 to linux_base-fc4. It seems FC4 is com­pat­i­ble enough so that we can dire­clty move from Red­Hat 8 to FC 4. FC 5 is out of the game for a while, it does­n’t want to run with the old lin­ux ker­nel ver­sion our lin­ux­o­la­tor announces. Chang­ing the ver­sion via the sysctl does­n’t help (prob­a­bly because of the changed seman­tic of the lin­ux clone syscall between 2.4.x and 2.6.x), so it has to wait until the lin­ux­o­la­tor SoC finishes.