Google’s new RE en­gine

I stumbled over Google’s new RE en­gine. Un­for­tu­nately it is not hand­ling back­refer­ences, so it is not a drop-​in re­place­ment for the reg­u­lar ex­pres­sions code in FreeBSD. It has a POSIX mode, but this only seems to be enough for the egrep syn­tax. For people which need back­refer­ences, they refer to the Google Chrome’s RE en­gine ir­reg­exp which in turn ref­er­ences a pa­per from 2007 which is titled Reg­u­lar Ex­pres­sion Match­ing Can Be Simple And Fast.

The tech­niques in the pa­per can not be ap­plied to the ir­reg­exp en­gine, but maybe could help to speed up awk, egrep and sim­ilar pro­grams.

I think it would be in­ter­est­ing to com­pare those re­cent de­vel­op­ments to what we have in FreeBSD, and if they are faster, to see if it is pos­sible to im­prove the FreeBSD im­ple­ment­a­tion based upon them (either by writ­ing new code, or by im­port­ing ex­ist­ing code, de­pend­ing on the cor­res­pond­ing li­cense and the lan­guage the code is writ­ten in).

Maybe a can­did­ate for the GSoC?

StumbleUponXINGBalatarinBox.netDiggGoogle GmailNetvouzPlurkSiteJotTypePad PostYahoo BookmarksVKSlashdotPocketHacker NewsDiigoBuddyMarksRedditLinkedInBibSonomyBufferEmailHatenaLiveJournalNewsVinePrintViadeoYahoo MailAIMBitty BrowserCare2 NewsEvernoteMail.RuPrintFriendlyWaneloYahoo MessengerYoolinkWebnewsStumpediaProtopage BookmarksOdnoklassnikiMendeleyInstapaperFarkCiteULikeBlinklistAOL MailTwitterGoogle+PinterestTumblrAmazon Wish ListBlogMarksDZoneDeliciousFlipboardFolkdJamespotMeneameMixiOknotiziePushaSvejoSymbaloo FeedsWhatsAppYouMobdiHITTWordPressRediff MyPageOutlook.comMySpaceDesign FloatBlogger PostApp.netDiary.RuKindle ItNUjijSegnaloTuentiWykopTwiddlaSina WeiboPinboardNetlogLineGoogle BookmarksDiasporaBookmarks.frBaiduFacebookGoogle ClassroomKakaoQzoneSMSTelegramRenrenKnownYummlyShare/​Save

FreeNAS & Sensors for FreeBSD

This WE I was told that FreeNAS seems to want to move from FreeBSD to Linux (since then it seems there could be a linux and a FreeBSD ver­sion). One of the reas­ons seems to be a miss­ing sensors frame­work.

As I was com­mit­ting a port of the OpenBSD sensors frame­work (pro­duced as part of the Google Sum­mer of Code 2007) to FreeBSD and had to re­move it af­ter­wards be­cause one com­mit­ter com­plained very loudly, I was asked what the status of this is.

The short status is: Nobody is do­ing some­thing about it.

Be­fore I ex­plain the long status, I give  a short over­view what this sensors frame­work is:

  • a ker­nel API which al­lows to add sensors
  • an in­ter­face for the user­land to query the sensor data
  • some ba­sic user­land code to show and log the sensor info

The API and the query in­ter­face are more or less in­de­pend­ent. For the user­land code it was more a log­ging in­fra­struc­ture than a real mon­it­or­ing solu­tion. The reason was the real mon­it­or­ing solu­tions already ex­ist (Nagios, sn­mpd, …) and can be ad­ap­ted to query the sensors. Ideally a query in user­land should be handled by a lib­rary in­stead of dir­ectly ac­cess­ing the sy­sctl in­ter­face, this way the kernel<->userland in­ter­face would be ab­strac­ted away (and could b re­placed as needs arise). This was not done, it was some­thing to be done later (Rome was not build in a day).

The user­land in­ter­face also only cared about dumb sensors (those which you need to query manu­ally to get the in­form­a­tion), smart sensors (those which are able to send events them­self) where not taken care about in the sense of really send­ing sensor-​triggered events, but the ker­nel API al­lowed to add such sensors. The sy­sctl in­ter­face has no way of send­ing events, but FreeBSD already has an event in­ter­face (devd is tak­ing care about it). It would have been not a prob­lem to send events via this chan­nel and let an user­land lib­rary take care about the de­liv­ery to­gether with other sensor-​data in user­land.

And now the long status is:

PHK com­plained loudly about it. First he said he did not look at it but he com­plained that is not good re­gard­less. After a lot of nag­ging from me he had a look at it and was not happy about the time stuff in it (short: the FreeBSD time­counter code is bet­ter). This was not a prob­lem in my opin­ion, we could have dis­abled this part without prob­lems. After such an of­fer from me, he com­plained that the sensors frame­work uses the sy­sctl in­ter­face in­stead of an entry in /​dev.

At this point in time already sev­eral user­land util­it­ies used the sy­sctl frame­work to query for status data in the ker­nel. So there was already pre­ced­ence for such an use of it. Later some more such uses where ad­ded too (e.g. the proc­stat stuff by core team mem­ber Robert Wat­son).

I saved some of the cor­res­pond­ing mails (to pub­lic mail­ing lists) in a mbox file, read the mess your­self if you want.

The bot­tom line is: Sev­eral com­mit­ters (even some which we could call high pro­file com­mit­ters) told me that they do not see a prob­lem in the use of the sy­sctl in­ter­face. They do not seem to want to tell it in pub­lic (nobody of them voiced their opin­ion in the thread, so do not ask me who those people are). I am not in­ter­ested in in­vest­ing more of my spare time into fight­ing wind­mills (it looks like this to me).

So, if someone is in­tersted in the code, r172631 has it. In the per­force re­pos­it­ory you can maybe find some sensors. I think most of it can still be used without much changes.

If someone tries it with a more re­cent FreeBSD, please drop me a note if it just ap­plies fine, or a patch (or an URL to it) if it needs some modi­fic­a­tions. Who knows, maybe in a fu­ture pro­ject it may be use­ful for me.

If there is enough in­terest by sev­eral people, I can even put up a wiki page where those people can co­ordin­ate, but that is most prob­ably all I am will­ing to in­vest fur­ther into this (at least in my un­paid time).

Some more WP plu­gins

Ad­di­tion­ally to the WP plu­gins I already talked about, I in­stalled some more since then:

  • aLinks: auto­matic link gen­er­a­tion based upon rules/​modules/​keywords
  • All in one SEO Pack: Search En­gine Op­tim­iz­a­tion, e.g. auto­matic meta tag gen­er­a­tion and more
  • Global Trans­lator: auto­matic machine-​translation of your posts into other lan­guages (see the coun­try flags in the side­bar)
  • Google XML Sitemaps: auto­mat­ic­ally gen­er­ates a sitemap and an­nounces the up­date (after a post) to sev­eral searchen­gines
  • http:BL Word­Press plu­gin: checks vis­it­ors against the Pro­ject Honey Pot black­list (email har­vest­ing) and re­jects ac­cess
  • In­foLink: adds a but­ton to look up marked text in the editor and link the markup to the first res­ult if de­sired
  • One-​Time Pass­word: al­lows to use one-​time pass­words (RFC 2289) for the WP lo­gin
  • Smart­linker: an­other but­ton for the editor re­gard­ing auto­matic link­ing (gives other auto­matic links than the other auto­matic link­ing plu­gins)
  • Up­date No­ti­fier: sends me a mail when there is a up­date for WP
  • WP-​Print: printer friendly page of the post­ing (see the link be­low the title of this post), I had to patch it to not print the links from the WP AJAX Edit Com­ments plu­gin
  • WP AJAX Edit Com­ments: provides an AJAX edit in­ter­face for com­ments

Catch­ing up… GSoC 2007

We got a lot of good pro­pos­als. Google is will­ing to give us a very nice amount of stu­dents. We didn’t ex­pec­ted this much. Thanks!

Now we need to rate the stu­dent ap­plic­a­tions and find suit­able ment­ors… not that easy. It’s easy for the strongest pro­pos­als, but for the rest I ex­pect that there will be some shuff­ling around un­til the very end.

In­ter­est­ing Sum­mer for FreeBSD

Now that we have (or “soon will have”) an of­fi­cial FreeBSD blog, I de­cided to give this kind of elec­tronic and pub­lic di­ary a try. At least to re­port the status and pro­gress of some FreeBSD re­lated pro­jects I’m in­volved in.

A lot happened in the last days. We’re past the dead­line of the Google Sum­mer of Code rat­ing pro­cess and now the lucky stu­dents are chosen. It wasn’t easy. We had more than 120 pro­pos­als (out of ~6400). From those we where will­ing to mentor 40 – 50 (based upon our re­sources and the qual­ity of the pro­pos­als). Google gran­ted 14 to us (a big thank you to Google!). I ex­pect an of­fi­cial an­nounce­ment “soon”. Hope­fully some of those stu­dents Google wasn’t able to fund are will­ing to work with us re­gard­less of the money, there are a lot of very nice pro­pos­als (I will add some items to the ideas list based upon them later). We’re at least will­ing to provide the same amount of ment­or­ship as if they where se­lec­ted.

I will work with two stu­dents. One of them will work on syncing the OSS API from re­cent re­leases from 4Front with our sound sys­tem. The other one will work on im­prov­ing the linuxolator. I will an­nounce this on the cor­res­pond­ing mailing­lists after the of­fi­cial an­nounce­ment. ATM we need to do some ad­min­is­trat­ive stuff (hand­ing out ac­cess to the wiki and per­force, set­ting up email ali­ases, …).

I already mailed some gen­eral guides to the stu­dents (note to ment­ors: it’s in the wiki on the “hid­den” and re­stric­ted to ment­ors SoC page). For the sound stuff I don’t have a nice TODO list, but luck­ily the stu­dent already in­vest­ig­ated the 4Front stuff for the pro­posal and is eager to start the work (ba­sic­ally this gives us the user­land in­ter­face for multi-​channel mixer sup­port). Re­gard­ing the linuxolator im­prove­ments the stu­dent has a little bit less luck. I com­piled a nice TODO list which is based upon his own pro­posal and all the vari­ous things I know about the linuxolator (PR’s, mes­sages on emulation@, private con­ver­sa­tions, things we no­ticed while work­ing on the up­date of the linux base to a re­cent Fe­dora Core, …). I will be im­pressed if he man­ages to do everything on the TODO list (don’t worry, he knows that not everything has to be done to de­clare “suc­cess” to Google).

Both of them may be can­did­ates for a com­mit bit (not within the SoC, but maybe later), both are eager to do the work, in­ter­ested, mo­tiv­ated, and don’t need hand-​holding.

Fur­ther news in the area of the sound sys­tem: Ar­iff told me he is work­ing on multi-​channel and en­di­aness issues/​support, and I may be able to com­mit two more sound drivers to the tree. One driver is the emu10kx driver cur­rently avail­able in the Ports Col­lec­tion, and the other one is a driver for some envy24 chips. Cur­rently I’m in con­tact with the au­thor of the emu10kx driver and a vo­lun­teer who wants to im­prove an ex­ist­ing envy24 driver (au­thor of the driver con­tac­ted; since it’s BSD li­censed, this is a “don’t be evil” ac­tion).

And some news re­gard­ing the user­land part of the linuxolator: We’re wait­ing for a repo-​copy form linux_​base-​fc3 to linux_​base-​fc4. It seems FC4 is com­pat­ible enough so that we can dir­e­clty move from Red­Hat 8 to FC 4. FC 5 is out of the game for a while, it doesn’t want to run with the old linux ker­nel ver­sion our linuxolator an­nounces. Chan­ging the ver­sion via the sy­sctl doesn’t help (prob­ably be­cause of the changed se­mantic of the linux clone sy­scall between 2.4.x and 2.6.x), so it has to wait un­til the linuxolator SoC fin­ishes.