Dir­ect, in­dir­ect and ex­pli­cit de­pend­en­cies in progams/​ports

The dis­cus­sion about dir­ect and in­dir­ect de­pend­en­cies is com­ing up again on the FreeBSD mailing­lists. Seems I should make some blog post about it, maybe it makes this top­ic more find­able than my post­ings in the mailing­lists.

Some defin­i­tions:

  • A dir­ect de­pend­ency from A to B is when program/​port A uses sym­bols from library/​port B.
  • An in­dir­ect de­pend­ency from A to C is when program/​port A uses sym­bols from library/​port B but no sym­bols from library/​port C, and library/​port B uses sym­bols from library/​port C.
  • An ex­pli­cit de­pend­ency from A to C is when it is a dir­ect or in­dir­ect de­pend­ency A to C, and when the compiler-​time–linker ad­ded an ex­pli­cit ref­er­ence to C to the program/​lib of A.

Ideally we have no in­dir­ect de­pend­en­cies in the ex­pli­cit de­pend­en­cies, only dir­ect de­pend­en­cies. Un­for­tu­nately in real­ity we also have in­dir­ect de­pend­en­cies there. This has at least two causes:

  1. lib­tool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-​time-​linker is re­curs­ively resolv­ing de­pend­en­cies.
  2. Some pkg–con­fig setups list in­dir­ect de­pend­en­cies as ex­pli­cit de­pend­en­cies (IIRC it de­pends if Requires.private and/​or Libs.private is used in the .pc file or not; if it is used, there should be no in­dir­ect de­pend­ency ap­pear from this soft­ware, but I am not 100% sure about this).

Three years ago I wrote /​usr/​ports/​Tools/​scripts/explicit_lib_depends.sh, it looks at the files of a giv­en port (it needs to be in­stalled), and prints out ex­pli­cit de­pend­en­cies. Be­cause of the in­dir­ect de­pend­en­cies which could be lis­ted there, this list is not a list of ports which are real de­pend­en­cies from a source code point of view, but it re­flects the link-​time real­ity. If a port C shows up there, the port which is checked needs to be re­build in case the ABI of library/​port C changes.

Ment­or­ing again in the GSoC

Seems that I will act­ively ment­or again in this Google Sum­mer of Code (as op­posed to just re­view the sub­mis­sions from stu­dents and/​or act­ing as a fall-​back ment­or).

The pro­ject I will ment­or is the “Make op­tion­al ker­nel sub­sys­tems re­gister them­selves via sy­sctl”-one from the FreeBSD ideas page.

The stu­dent already got in­to con­tact with me and it looks like he is mo­tiv­ated (he is already sub­scribed to sev­er­al FreeBSD mailing­lists, which is not a re­quire­ment we have in our GSoC docs).

Some fixes for ZFS on 7-​stable (more test­ers wanted)

Due to the prob­lems with a 7–stable ma­chine, I had a look at some un­merged fixes for ZFS (58 changes not merged).

I back­por­ted some of those changes from 8-​stable to 7-​stable, I have this run­ning on one 7-​stable ma­chine. I would like to get some more feed­back for it (even an “it works for me” would be great). The main part of this change is that the FreeBSD taskqueue is used now in­stead of the opensol­ar­is one (and some oth­er changes which may im­prove the ZFS ex­per­i­ence).

It would also be nice if someone could have a look at the FIRST_​THREAD_​IN_​PROC part. Can there be more than one thread at this place (I do not think so) and I should use FOREACH_​THREAD_​IN_​PROC_​instead?

How to ap­ply:

  • cd /​usr/​src/​
  • fetch http://​www​.Leidinger​.net/FreeBSD/test/releng7_zfs_merge3.diff
  • fetch http://​www​.Leidinger​.net/FreeBSD/test/opensolaris_taskq.c
  • fetch http://​www​.Leidinger​.net/​F​r​e​e​B​S​D​/​t​e​s​t​/​t​a​s​k​q.h
  • mv taskq.h sys/cddl/contrib/opensolaris/uts/common/sys/taskq.h
  • mv opensolaris_taskq.c sys/​cddl/​com­pat/opensolaris/kern/opensolaris_taskq.c
  • patch -p 0 –quiet <releng7_zfs_merge3.diff
  • ig­nore the 2 .rej files
  • rm -f sys/cddl/compat/opensolaris/sys/taskq_impl.h*
  • rm -f sys/cddl/compat/opensolaris/sys/taskq.h*
  • rm -f sys/cddl/contrib/opensolaris/uts/common/os/taskq.c*
  • re­build ker­nel

I do not list all of those 16 of 58 out­stand­ing patches which are covered here, a de­tailed list can be found on the stable and fs mailing­lists.

Sta­bil­ity prob­lems with 7-​stable

On the ma­chine where I host this blog, I have/​had some sta­bil­ity prob­lems.

Last week I up­dated the ma­chine from FreeBSD 7.1-pX to 7.2-p5 (GENERIC ker­nel in both cases). 5 – 10 Minutes after the re­boot in­to the new ver­sion the ma­chine had a dead­lock. After some road­b­locks (or­der­ing a KVM-​switch from the hoster, the KVM-​switch not work­ing with a proxy (dur­ing lunch­time at work), a broken video-​capture of the KVM-​switch and a re­place­ment on Monday morn­ing to not pay the WE-​fees), I spend a big part of the night to get it stable. I tried dis­abling SMP, en­abling INVARIANTS and WITNESS, chan­ging the sched­uler, cut­ting the soft­ware mir­ror (to rule out a mis­match between the con­tent of the disks after all the hard re­boots) and up­dat­ing to 7-​stable.

Un­for­tu­nately noth­ing helped. 🙁

Googling a little bit around (it is a AMD Dual-​Core sys­tem with NVidia MCP61 chip­set) was lead­ing me to a post on the mailing­lists from 2008 which talks about an is­sue with the buf­fer cache. I do not know if this is still an is­sue (I have send a email to kib@ to ask about it), and my scen­ario is not the same as the one which is de­scribed in the mail, but be­cause of this I de­cided to switch one of the two UFS mir­rors to ZFS.

The first boot in­to the ZFS caused again a re­boot after some minutes (I do not know if it was be­cause of a memory ex­hausted pan­ic, or be­cause of a dead­lock), but as I did not tune the ker­nel for ZFS I am temp­ted to be­lieve that I should not count that. Now, after tun­ing the ker­nel (in­creas­ing the kmem_​size to 700M, no prefetch­ing, lim­it­ing the ARC to 40M) it is up since nearly 2h (as of this writ­ting… cross­ing fin­gers). Be­fore it was not able to sur­vive more than some minutes with just the jail for the mails up. Now I not only have the mail-​jail up, but also the jail for the blog (one jail still dis­abled, but I will take care about that after this post).

I do not know if only in­creas­ing the kmem_​size would have helped with the prob­lem, but as I was test­ing a GENERIC ker­nel + gmir­ror mod­ule in the be­gin­ning, I ex­pec­ted that the auto-​tuning of this value should have been enough for such a simple setup (2GB RAM, 2 disks with 3 par­ti­tions each, one par­ti­tion pair for root, one for swap, one for the jails).

I hope that I sta­bil­ized the sys­tem now. It may be the case that I will test some patches in case someone comes up with some­thing, so do not be sur­prised if the blog and email to me is a little bit flaky.

Back from hol­i­day: mail-​stats

After two weeks of hol­i­day I have about 5.400 emails (about 51 MB, down­loaded and filtered in 3h). Split up this is:

  • 2763 from FreeBSD mailing­lists
  • 320 SPAM (101 false pos­it­ives, mostly FreeBSD re­lated… I should have a look at this and take ap­pro­pri­ate ac­tions)
  • 115 in the nor­mal in­box = per­son­al + not filtered emails (18 false neg­at­ive SPAM)
  • 16 post­mas­ter re­lated, e.g. bounces from forged SPAM, 4 of them are not SPAM bounces but val­id mails…
  • 114 se­cur­ity re­lated (Bugtraq, …)
  • 220 status mes­sages (e.g. FreeBSD peri­od­ic mails from vari­ous sys­tems)
  • the rest is vari­ous oth­er (filtered away) mails

It will take a while un­til I had a look at them…

In­ter­est­ing Sum­mer for FreeBSD

Now that we have (or “soon will have”) an of­fi­cial FreeBSD blog, I de­cided to give this kind of elec­tron­ic and pub­lic di­ary a try. At least to re­port the status and pro­gress of some FreeBSD re­lated pro­jects I’m in­volved in.

A lot happened in the last days. We’re past the dead­line of the Google Sum­mer of Code rat­ing pro­cess and now the lucky stu­dents are chosen. It wasn’t easy. We had more than 120 pro­pos­als (out of ~6400). From those we where will­ing to ment­or 40 – 50 (based upon our re­sources and the qual­ity of the pro­pos­als). Google gran­ted 14 to us (a big thank you to Google!). I ex­pect an of­fi­cial an­nounce­ment “soon”. Hope­fully some of those stu­dents Google wasn’t able to fund are will­ing to work with us re­gard­less of the money, there are a lot of very nice pro­pos­als (I will add some items to the ideas list based upon them later). We’re at least will­ing to provide the same amount of ment­or­ship as if they where se­lec­ted.

I will work with two stu­dents. One of them will work on syncing the OSS API from re­cent re­leases from 4Front with our sound sys­tem. The oth­er one will work on im­prov­ing the linuxolat­or. I will an­nounce this on the cor­res­pond­ing mailing­lists after the of­fi­cial an­nounce­ment. ATM we need to do some ad­min­is­trat­ive stuff (hand­ing out ac­cess to the wiki and per­force, set­ting up email ali­ases, …).

I already mailed some gen­er­al guides to the stu­dents (note to ment­ors: it’s in the wiki on the “hid­den” and re­stric­ted to ment­ors SoC page). For the sound stuff I don’t have a nice TODO list, but luck­ily the stu­dent already in­vest­ig­ated the 4Front stuff for the pro­pos­al and is eager to start the work (ba­sic­ally this gives us the user­land in­ter­face for multi-​channel mix­er sup­port). Re­gard­ing the linuxolat­or im­prove­ments the stu­dent has a little bit less luck. I com­piled a nice TODO list which is based upon his own pro­pos­al and all the vari­ous things I know about the linuxolat­or (PR’s, mes­sages on emulation@, private con­ver­sa­tions, things we no­ticed while work­ing on the up­date of the linux base to a re­cent Fe­dora Core, …). I will be im­pressed if he man­ages to do everything on the TODO list (don’t worry, he knows that not everything has to be done to de­clare “suc­cess” to Google).

Both of them may be can­did­ates for a com­mit bit (not with­in the SoC, but maybe later), both are eager to do the work, in­ter­ested, mo­tiv­ated, and don’t need hand-​holding.

Fur­ther news in the area of the sound sys­tem: Ar­iff told me he is work­ing on multi-​channel and en­di­aness issues/​support, and I may be able to com­mit two more sound drivers to the tree. One driver is the emu10kx driver cur­rently avail­able in the Ports Col­lec­tion, and the oth­er one is a driver for some envy24 chips. Cur­rently I’m in con­tact with the au­thor of the emu10kx driver and a vo­lun­teer who wants to im­prove an ex­ist­ing envy24 driver (au­thor of the driver con­tac­ted; since it’s BSD li­censed, this is a “don’t be evil” ac­tion).

And some news re­gard­ing the user­land part of the linuxolat­or: We’re wait­ing for a repo-​copy form linux_​base-​fc3 to linux_​base-​fc4. It seems FC4 is com­pat­ible enough so that we can dir­e­clty move from Red­Hat 8 to FC 4. FC 5 is out of the game for a while, it doesn’t want to run with the old linux ker­nel ver­sion our linuxolat­or an­nounces. Chan­ging the ver­sion via the sy­sctl doesn’t help (prob­ably be­cause of the changed se­mant­ic of the linux clone sy­scall between 2.4.x and 2.6.x), so it has to wait un­til the linuxolat­or SoC fin­ishes.