Dir­ect, in­dir­ect and ex­pli­cit de­pend­en­cies in progams/​ports

The dis­cus­sion about dir­ect and in­dir­ect de­pend­en­cies is com­ing up again on the FreeBSD mailing­lists. Seems I should make some blog post about it, may­be it makes this top­ic more find­able than my post­ings in the mailing­lists.

Some defin­i­tions:

  • A dir­ect de­pend­ency from A to B is when program/​port A uses sym­bols from library/​port B.
  • An in­dir­ect de­pend­ency from A to C is when program/​port A uses sym­bols from library/​port B but no sym­bols from library/​port C, and library/​port B uses sym­bols from library/​port C.
  • An ex­pli­cit de­pend­ency from A to C is when it is a dir­ect or in­dir­ect de­pend­ency A to C, and when the compiler-​time–linker ad­ded an ex­pli­cit ref­er­ence to C to the program/​lib of A.

Ideally we have no in­dir­ect de­pend­en­cies in the ex­pli­cit de­pend­en­cies, only dir­ect de­pend­en­cies. Un­for­tu­nately in real­ity we also have in­dir­ect de­pend­en­cies there. This has at least two causes:

  1. lib­tool (at least 1.x) does not (or was not) come with a hint on FreeBSD, which tells that the run-​time-​linker is re­curs­ively resolv­ing de­pend­en­cies.
  2. Some pkg–con­fig setups list in­dir­ect de­pend­en­cies as ex­pli­cit de­pend­en­cies (IIRC it de­pends if Requires.private and/​or Libs.private is used in the .pc file or not; if it is used, there should be no in­dir­ect de­pend­ency ap­pear from this soft­ware, but I am not 100% sure about this).

Three years ago I wro­te /​usr/​ports/​Tools/​scripts/explicit_lib_depends.sh, it looks at the files of a given port (it needs to be in­stalled), and prints out ex­pli­cit de­pend­en­cies. Be­cause of the in­dir­ect de­pend­en­cies which could be lis­ted there, this list is not a list of ports which are real de­pend­en­cies from a source code point of view, but it re­flects the link-​time real­ity. If a port C shows up there, the port which is checked needs to be re­build in case the ABI of library/​port C changes.

Ment­or­ing again in the GSoC

Seems that I will act­ively ment­or again in this Google Sum­mer of Code (as op­posed to just re­view the sub­mis­sions from stu­dents and/​or act­ing as a fall-​back ment­or).

The pro­ject I will ment­or is the “Make op­tion­al ker­nel sub­sys­tems re­gister them­selves via sy­sctl”-one from the FreeBSD ideas page.

The stu­dent already got in­to con­tact with me and it looks like he is mo­tiv­ated (he is already sub­scribed to sev­er­al FreeBSD mailing­lists, which is not a re­quire­ment we have in our GSoC docs).

Some fixes for ZFS on 7-​stable (more test­ers wanted)

Due to the prob­lems with a 7–stable ma­chine, I had a look at some un­merged fixes for ZFS (58 changes not merged).

I back­por­ted some of those changes from 8-​stable to 7-​stable, I have this run­ning on one 7-​stable ma­chine. I would like to get some more feed­back for it (even an “it works for me” would be great). The main part of this change is that the FreeBSD taskqueue is used now in­stead of the opensol­ar­is one (and some oth­er changes which may im­prove the ZFS ex­per­i­ence).

It would also be nice if someone could have a look at the FIRST_​THREAD_​IN_​PROC part. Can there be more than one thread at this place (I do not think so) and I should use FOREACH_​THREAD_​IN_​PROC_​instead?

How to ap­ply:

  • cd /​usr/​src/​
  • fetch http://​www​.Leidinger​.net/FreeBSD/test/releng7_zfs_merge3.diff
  • fetch http://​www​.Leidinger​.net/FreeBSD/test/opensolaris_taskq.c
  • fetch http://​www​.Leidinger​.net/​F​r​e​e​B​S​D​/​t​e​s​t​/​t​a​s​k​q.h
  • mv taskq.h sys/cddl/contrib/opensolaris/uts/common/sys/taskq.h
  • mv opensolaris_taskq.c sys/​cddl/​com­pat/opensolaris/kern/opensolaris_taskq.c
  • patch –p 0 –quiet <releng7_zfs_merge3.diff
  • ig­nore the 2 .rej files
  • rm –f sys/cddl/compat/opensolaris/sys/taskq_impl.h*
  • rm –f sys/cddl/compat/opensolaris/sys/taskq.h*
  • rm –f sys/cddl/contrib/opensolaris/uts/common/os/taskq.c*
  • re­build ker­nel

I do not list all of those 16 of 58 out­stand­ing patches which are covered here, a de­tailed list can be found on the stable and fs mailing­lists.

Sta­bil­ity prob­lems with 7-​stable

On the ma­chine where I host this blog, I have/​had some sta­bil­ity prob­lems.

Last week I up­dated the ma­chine from FreeBSD 7.1-pX to 7.2-p5 (GENERIC ker­nel in both cases). 5 – 10 Minutes af­ter the re­boot in­to the new ver­sion the ma­chine had a dead­lock. Af­ter some road­b­locks (or­der­ing a KVM-​switch from the hoster, the KVM-​switch not work­ing with a proxy (dur­ing lunch­time at work), a broken video-​capture of the KVM-​switch and a re­place­ment on Monday morn­ing to not pay the WE-​fees), I spend a big part of the night to get it stable. I tried dis­abling SMP, en­abling INVARIANTS and WITNESS, chan­ging the sched­uler, cut­ting the soft­ware mir­ror (to rule out a mis­match between the con­tent of the disks af­ter all the hard re­boots) and up­dat­ing to 7-​stable.

Un­for­tu­nately noth­ing helped. 🙁

Googling a little bit around (it is a AMD Dual-​Core sys­tem with NVidia MCP61 chip­set) was lead­ing me to a post on the mailing­lists from 2008 which talks about an is­sue with the buf­fer cache. I do not know if this is still an is­sue (I have send a email to kib@ to ask about it), and my scen­ario is not the same as the one which is de­scribed in the mail, but be­cause of this I de­cided to switch one of the two UFS mir­rors to ZFS.

The first boot in­to the ZFS caused again a re­boot af­ter some minutes (I do not know if it was be­cause of a memory ex­hausted pan­ic, or be­cause of a dead­lock), but as I did not tune the ker­nel for ZFS I am temp­ted to be­lieve that I should not count that. Now, af­ter tun­ing the ker­nel (in­creas­ing the kmem_​size to 700M, no prefetch­ing, lim­it­ing the ARC to 40M) it is up since nearly 2h (as of this writ­ting… cross­ing fin­gers). Be­fore it was not able to sur­vive more than some minutes with just the jail for the mails up. Now I not only have the mail-​jail up, but also the jail for the blog (one jail still dis­abled, but I will take care about that af­ter this post).

I do not know if only in­creas­ing the kmem_​size would have helped with the prob­lem, but as I was test­ing a GENERIC ker­nel + gmir­ror mod­ule in the be­gin­ning, I ex­pec­ted that the auto-​tuning of this value should have been enough for such a sim­ple setup (2GB RAM, 2 disks with 3 par­ti­tions each, one par­ti­tion pair for root, one for swap, one for the jails).

I hope that I sta­bil­ized the sys­tem now. It may be the case that I will test some patches in case someone comes up with some­thing, so do not be sur­prised if the blog and email to me is a little bit flaky.

Back from hol­i­day: mail-​stats

Af­ter two weeks of hol­i­day I have about 5.400 emails (about 51 MB, down­loaded and filtered in 3h). Split up this is:

  • 2763 from FreeBSD mailing­lists
  • 320 SPAM (101 false pos­it­ives, mostly FreeBSD re­lated… I should have a look at this and take ap­pro­pri­ate ac­tions)
  • 115 in the nor­mal in­box = per­son­al + not filtered emails (18 false neg­at­ive SPAM)
  • 16 post­mas­ter re­lated, e.g. bounces from forged SPAM, 4 of them are not SPAM bounces but val­id mails…
  • 114 se­cur­ity re­lated (Bugtraq, …)
  • 220 status mes­sages (e.g. FreeBSD peri­od­ic mails from vari­ous sys­tems)
  • the rest is vari­ous oth­er (filtered away) mails

It will take a while un­til I had a look at them…