Linuxu­lat­or D-​Trace probes com­mit­ted to cur­rent

A while ago I com­mit­ted the linuxu­lat­or D-​Trace probes I talked about earli­er. I waited a little bit for this an­nounce­ment to make sure I have not broken any­thing. Nobody com­plained so far, so I as­sume noth­ing ob­vi­ously bad crept in.

The >500 probes I com­mit­ted do not cov­er the en­tire linuxu­lat­or, but are a good start. Adding new ones is straight for­ward, if someone is in­ter­ested in a ju­ni­or–ker­nel–hack­er task, this would be one. Just ask me (or ask on emu­la­tion@), and I can guide you through it.

DTrace in GENERIC (-cur­rent)

In case you have not no­ticed yet, KDTRACE_​HOOKS is now in the GENERIC ker­nel in FreeBSD-cur­rent. This means you just need to load the DTrace mod­ules and can use DTrace with the GENERIC ker­nel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you in­vest read­ing it.

Linuxu­lat­or pro­gress

This week­end I made some pro­gress in the linuxu­lat­or:

  • I MFCed the re­port­ing of some linux-​syscalls to 9-​stable and 8-​stable.
  • I up­dated my linuxu­lat­or-dtrace patch to a re­cent –cur­rent. I already com­piled it on i386 and arundel@ has it com­piled on amd64. I coun­ted more than 500 new DTrace probes. Now that DTrace res­cans for SDT probes when a ker­nel mod­ule is loaded, there is no ker­nel pan­ic any­more when the linux mod­ule is loaded af­ter the DTrace mod­ules and you want to use DTrace. I try to com­mit this at a morn­ing of a day where I can fix things dur­ing the day in case some prob­lems show up which I did not no­tice dur­ing my test­ing.
  • I cre­ated a PR for portmgr@ to re­po­copy a new linux_​base port.
  • I set the ex­pir­a­tion date of linux_​base-​fc4 (only used by 7.x and up­stream way past its EoL) and all de­pend­ent ports. It is set to the EoL of the last 7.x re­lease, which can not use a later linux_​base port. I also ad­ded a com­ment which ex­plains that the date is the EoL of the last 7.x re­lease.

Stat­ic DTrace probes for the linuxu­lat­or up­dated

I got a little bit of time to up­date my 3 year old work of adding stat­ic DTrace probes to the linuxu­lat­or.

The changes are not in HEAD, but in my linuxulator-​dtrace branch. The re­vi­sion to have a look at is r230910. In­cluded are some DTrace scripts:

  • script to check in­tern­al locks
  • script to trace fu­texes
  • script to gen­er­ate stats for DTra­ci­fied linuxu­lat­or parts
  • script to check for er­rors:
    • emu­la­tion er­rors (un­sup­por­ted stuff, un­known stuff, …)
    • ker­nel er­rors (re­source short­age, …)
    • pro­gram­ming er­rors (er­rors which can hap­pen if someone made a mis­take, but should not hap­pen)

The programming-​error checks give hints about user­land pro­gram­ming er­rors re­spect­ively a hint about the reas­on of er­ror re­turn val­ues due to re­source short­age or may­be a wrong com­bin­a­tion of para­met­ers. An ex­ample er­ror mes­sage for this case is “Ap­plic­a­tion %s is­sued a sy­sctl which failed the length restrictions.\nThe length passed is %d, the min length sup­por­ted is 1 and the max length sup­por­ted is %d.\n”.

The stats-​script (tailored spe­cially to the linuxu­lat­or, but this can eas­ily be ex­ten­ded to the rest of the ker­nel) can re­port about:

  • num­ber of calls to a ker­nel func­tion per ex­ecut­able bin­ary (not per PID!): al­lows to see where an op­tim­iz­a­tion would be be­ne­fi­cial for a given ap­plic­a­tion
  • graph of CPU time spend in ker­nel func­tions per ex­ecut­able bin­ary: to­geth­er with the num­ber of calls to this func­tion this al­lows to de­term­ine if a ker­nel op­tim­iz­a­tion would be be­ne­fi­cial /​ is pos­sible for a given ap­plic­a­tion
  • graph of longest run­ning (CPU-​time!) ker­nel func­tion in total
  • tim­ing stat­ist­ics for the emul_​lock
  • graph of longest held (CPU-​time!) locks

Un­for­tu­nately this can not be com­mit­ted to HEAD as-​is. The DTrace SDT pro­vider can not handle probes which are ad­ded to the ker­nel af­ter the SDT pro­vider is already loaded. This means that you either have to com­pile the linuxu­lat­or stat­ic­ally in­to the ker­nel, or you have to load the SDT ker­nel mod­ule af­ter the linuxu­lat­or mod­ule is loaded. If you do not re­spect this, you get a ker­nel pan­ic on first ac­cess of one of the pro­viders in the linuxu­lat­or (AFAIR this in­cludes list­ing the probes avail­able in the ker­nel).

New DTrace probes for the linuxu­lat­or

I for­ward por­ted my DTrace probes for the FreeBSD linuxu­lat­or from a 2008-​current to a re­cent –cur­rent. I have not the com­plete FreeBSD linuxu­lat­or covered, but a big part is already done. I can check the ma­jor locks in the linuxu­lat­or, trace fu­texes, and I have a D-​script which yells at a lot of er­rors which could hap­pen but should not.

Some of my D-​scripts need some changes, as real-​world test­ing showed that they are not really work­ing as ex­pec­ted. They can get over­whelmed by the amount of spec­u­la­tion and dy­nam­ic vari­ables (er­ror mes­sage: dy­nam­ic vari­able drops with non-​empty dirty list). For the dy­nam­ic vari­ables prob­lem I found a dis­cus­sion on the net with some sug­ges­tions. For the spec­u­la­tion part I ex­pect sim­il­ar tuning-​possibilities.

Un­for­tu­nately the D-​script which checks the in­tern­al locks fails to com­pile. Seems there is a little mis­un­der­stand­ing on my side how the D-​language is sup­posed to work.

I try to get some time later to have a look at those prob­lems.

Dur­ing my de­vel­op­ment I stumbled over some gen­er­ic DTrace prob­lems with the SDT pro­vider I use for my probes:

  • If you load the Linux mod­ule af­ter the SDT mod­ule, your sys­tem will pan­ic as soon as you want to ac­cess some probes, e.g. “dtrace –l” will pan­ic the sys­tem. Load­ing the Linux mod­ule be­fore the SDT mod­ule pre­vents the pan­ic.
  • Un­load­ing the SDT mod­ule while the Linux mod­ule with the SDT probes is still loaded pan­ics the sys­tem too. Do not un­load the Linux mod­ule if you run with my patch.

Ac­cord­ing to avg@ those are known prob­lems, but I think nobody is work­ing on this. This is bad, be­cause this means I can not com­mit my cur­rent patch­set.

If someone wants to try the new DTrace probes for the linuxu­lat­or, feel free to go to http://​www​.Leidinger​.net/​F​r​e​e​B​S​D​/​c​u​r​r​e​n​t​-​p​a​t​c​h​es/ and down­load linuxulator-dtrace.diff. I do not of­fer a work­ing hy­per­link here on pur­pose, the SDT bugs can hurt if you are not care­ful, and I want to make the use of this patch a strong opt-​in be­cause of this. If the patch hurts you, it is your fault, you have been warned.