Google’s new RE en­gine

I stumbled over Google’s new RE en­gine. Un­for­tu­nately it is not hand­ling back­refer­ences, so it is not a drop-​in re­place­ment for the reg­u­lar ex­pres­sions code in FreeBSD. It has a POSIX mode, but this only seems to be enough for the egrep syn­tax. For people which need back­refer­ences, they refer to the Google Chrome’s RE en­gine ir­reg­exp which in turn ref­er­ences a pa­per from 2007 which is titled Reg­u­lar Ex­pres­sion Match­ing Can Be Sim­ple And Fast.

The tech­niques in the pa­per can not be ap­plied to the ir­reg­exp en­gine, but may­be could help to speed up awk, egrep and sim­il­ar pro­grams.

I think it would be in­ter­est­ing to com­pare those re­cent de­vel­op­ments to what we have in FreeBSD, and if they are faster, to see if it is pos­sible to im­prove the FreeBSD im­ple­ment­a­tion based upon them (either by writ­ing new code, or by im­port­ing ex­ist­ing code, de­pend­ing on the cor­res­pond­ing li­cense and the lan­guage the code is writ­ten in).

May­be a can­did­ate for the GSoC?

FreeNAS & Sensors for FreeBSD

This WE I was told that FreeNAS seems to want to move from FreeBSD to Linux (since then it seems there could be a linux and a FreeBSD ver­sion). One of the reas­ons seems to be a miss­ing sensors frame­work.

As I was com­mit­ting a port of the OpenBSD sensors frame­work (pro­duced as part of the Google Sum­mer of Code 2007) to FreeBSD and had to re­move it af­ter­wards be­cause one com­mit­ter com­plained very loudly, I was asked what the status of this is.

The short status is: Nobody is do­ing some­thing about it.

Be­fore I ex­plain the long status, I give  a short over­view what this sensors frame­work is:

  • a ker­nel API which al­lows to add sensors
  • an in­ter­face for the user­land to query the sensor data
  • some ba­sic user­land code to show and log the sensor in­fo

The API and the query in­ter­face are more or less in­de­pend­ent. For the user­land code it was more a log­ging in­fra­struc­ture than a real mon­it­or­ing solu­tion. The reas­on was the real mon­it­or­ing solu­tions already ex­ist (Nagios, sn­m­pd, …) and can be ad­ap­ted to query the sensors. Ideally a query in user­land should be handled by a lib­rary in­stead of dir­ectly ac­cess­ing the sy­sctl in­ter­face, this way the ker­nel<->user­land in­ter­face would be ab­strac­ted away (and could b re­placed as needs arise). This was not done, it was some­thing to be done later (Rome was not build in a day).

The user­land in­ter­face also only cared about dumb sensors (those which you need to query manu­ally to get the in­form­a­tion), smart sensors (those which are able to send events them­self) where not taken care about in the sense of really send­ing sensor-​triggered events, but the ker­nel API al­lowed to add such sensors. The sy­sctl in­ter­face has no way of send­ing events, but FreeBSD already has an event in­ter­face (devd is tak­ing care about it). It would have been not a prob­lem to send events via this chan­nel and let an user­land lib­rary take care about the de­liv­ery to­geth­er with oth­er sensor-​data in user­land.

And now the long status is:

PHK com­plained loudly about it. First he said he did not look at it but he com­plained that is not good re­gard­less. Af­ter a lot of nag­ging from me he had a look at it and was not happy about the time stuff in it (short: the FreeBSD time­coun­ter code is bet­ter). This was not a prob­lem in my opin­ion, we could have dis­abled this part without prob­lems. Af­ter such an of­fer from me, he com­plained that the sensors frame­work uses the sy­sctl in­ter­face in­stead of an entry in /​dev.

At this point in time already sev­er­al user­land util­it­ies used the sy­sctl frame­work to query for status data in the ker­nel. So there was already pre­ced­ence for such an use of it. Later some more such uses where ad­ded too (e.g. the proc­stat stuff by core team mem­ber Robert Wat­son).

I saved some of the cor­res­pond­ing mails (to pub­lic mail­ing lists) in a mbox file, read the mess your­self if you want.

The bot­tom line is: Sev­er­al com­mit­ters (even some which we could call high pro­file com­mit­ters) told me that they do not see a prob­lem in the use of the sy­sctl in­ter­face. They do not seem to want to tell it in pub­lic (nobody of them voiced their opin­ion in the thread, so do not ask me who those people are). I am not in­ter­ested in in­vest­ing more of my spare time in­to fight­ing wind­mills (it looks like this to me).

So, if someone is in­tersted in the code, r172631 has it. In the per­for­ce re­pos­it­ory you can may­be find some sensors. I think most of it can still be used without much changes.

If someone tries it with a more re­cent FreeBSD, please drop me a note if it just ap­plies fine, or a patch (or an URL to it) if it needs some modi­fic­a­tions. Who knows, may­be in a fu­ture pro­ject it may be use­ful for me.

If there is enough in­terest by sev­er­al people, I can even put up a wiki page where those people can co­ordin­ate, but that is most prob­ably all I am will­ing to in­vest fur­ther in­to this (at least in my un­paid time).

Some more WP plu­gins

Ad­di­tion­ally to the WP plu­gins I already talked about, I in­stalled some more since then:

Catch­ing up… GSoC 2007

We got a lot of good pro­pos­als. Google is will­ing to give us a very nice amount of stu­dents. We didn’t ex­pec­ted this much. Thanks!

Now we need to rate the stu­dent ap­plic­a­tions and find suit­able ment­ors… not that easy. It’s easy for the strongest pro­pos­als, but for the rest I ex­pect that there will be some shuff­ling around un­til the very end.

In­ter­est­ing Sum­mer for FreeBSD

Now that we have (or “soon will have”) an of­fi­cial FreeBSD blog, I de­cided to give this kind of elec­tron­ic and pub­lic di­ary a try. At least to re­port the status and pro­gress of some FreeBSD re­lated pro­jects I’m in­volved in.

A lot happened in the last days. We’re past the dead­line of the Google Sum­mer of Code rat­ing pro­cess and now the lucky stu­dents are chosen. It wasn’t easy. We had more than 120 pro­pos­als (out of ~6400). From those we where will­ing to ment­or 40 – 50 (based upon our re­sources and the qual­ity of the pro­pos­als). Google gran­ted 14 to us (a big thank you to Google!). I ex­pect an of­fi­cial an­nounce­ment “soon”. Hope­fully some of those stu­dents Google wasn’t able to fund are will­ing to work with us re­gard­less of the money, there are a lot of very nice pro­pos­als (I will add some items to the ideas list based upon them later). We’re at least will­ing to provide the same amount of ment­or­ship as if they where se­lec­ted.

I will work with two stu­dents. One of them will work on syncing the OSS API from re­cent re­leases from 4Front with our sound sys­tem. The oth­er one will work on im­prov­ing the linuxolat­or. I will an­nounce this on the cor­res­pond­ing mailing­lists af­ter the of­fi­cial an­nounce­ment. ATM we need to do some ad­min­is­trat­ive stuff (hand­ing out ac­cess to the wiki and per­for­ce, set­ting up email ali­ases, …).

I already mailed some gen­er­al guides to the stu­dents (note to ment­ors: it’s in the wiki on the “hid­den” and re­stric­ted to ment­ors SoC page). For the sound stuff I don’t have a nice TODO list, but luck­ily the stu­dent already in­vest­ig­ated the 4Front stuff for the pro­pos­al and is eager to start the work (ba­sic­ally this gives us the user­land in­ter­face for multi-​channel mix­er sup­port). Re­gard­ing the linuxolat­or im­prove­ments the stu­dent has a little bit less luck. I com­piled a nice TODO list which is based upon his own pro­pos­al and all the vari­ous things I know about the linuxolat­or (PR’s, mes­sages on emulation@, private con­ver­sa­tions, things we no­ticed while work­ing on the up­date of the linux base to a re­cent Fe­dora Core, …). I will be im­pressed if he man­ages to do everything on the TODO list (don’t worry, he knows that not everything has to be done to de­clare “suc­cess” to Google).

Both of them may be can­did­ates for a com­mit bit (not with­in the SoC, but may­be later), both are eager to do the work, in­ter­ested, mo­tiv­ated, and don’t need hand-​holding.

Fur­ther news in the area of the sound sys­tem: Ar­iff told me he is work­ing on multi-​channel and en­di­aness issues/​support, and I may be able to com­mit two more sound drivers to the tree. One driver is the emu10kx driver cur­rently avail­able in the Ports Col­lec­tion, and the oth­er one is a driver for some envy24 chips. Cur­rently I’m in con­tact with the au­thor of the emu10kx driver and a vo­lun­teer who wants to im­prove an ex­ist­ing envy24 driver (au­thor of the driver con­tac­ted; since it’s BSD li­censed, this is a “don’t be evil” ac­tion).

And some news re­gard­ing the user­land part of the linuxolat­or: We’re wait­ing for a repo-​copy form linux_​base-​fc3 to linux_​base-​fc4. It seems FC4 is com­pat­ible enough so that we can dir­e­clty move from Red­Hat 8 to FC 4. FC 5 is out of the game for a while, it doesn’t want to run with the old linux ker­nel ver­sion our linuxolat­or an­nounces. Chan­ging the ver­sion via the sy­sctl doesn’t help (prob­ably be­cause of the changed se­mantic of the linux clone sy­scall between 2.4.x and 2.6.x), so it has to wait un­til the linuxolat­or SoC fin­ishes.