Lin­ux­u­la­tor D‑Trace probes com­mit­ted to current

A while ago I com­mit­ted the lin­ux­u­la­tor D‑Trace probes I talked about ear­li­er. I wait­ed a lit­tle bit for this announce­ment to make sure I have not bro­ken any­thing. Nobody com­plained so far, so I assume noth­ing obvi­ous­ly bad crept in.

The >500 probes I com­mit­ted do not cov­er the entire lin­ux­u­la­tor, but are a good start. Adding new ones is straight for­ward, if some­one is inter­est­ed in a junior-kernel-hacker task, this would be one. Just ask me (or ask on emulation@), and I can guide you through it.

DTrace in GENERIC (-cur­rent)

In case you have not noticed yet, KDTRACE_HOOKS is now in the GENERIC ker­nel in FreeBSD-current. This means you just need to load the DTrace mod­ules and can use DTrace with the GENERIC kernel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you invest read­ing it.

Lin­ux­u­la­tor progress

This week­end I made some progress in the linuxulator:

  • I MFCed the report­ing of some linux-syscalls to 9‑stable and 8‑stable.
  • I updat­ed my linuxulator-dtrace patch to a recent ‑cur­rent. I already com­piled it on i386 and arundel@ has it com­piled on amd64. I count­ed more than 500 new DTrace probes. Now that DTrace res­cans for SDT probes when a ker­nel mod­ule is loaded, there is no ker­nel pan­ic any­more when the lin­ux mod­ule is loaded after the DTrace mod­ules and you want to use DTrace. I try to com­mit this at a morn­ing of a day where I can fix things dur­ing the day in case some prob­lems show up which I did not notice dur­ing my testing.
  • I cre­at­ed a PR for portmgr@ to repocopy a new linux_base port.
  • I set the expi­ra­tion date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all depen­dent ports. It is set to the EoL of the last 7.x release, which can not use a lat­er linux_base port. I also added a com­ment which explains that the date is the EoL of the last 7.x release.

Sta­t­ic DTrace probes for the lin­ux­u­la­tor updated

I got a lit­tle bit of time to update my 3 year old work of adding sta­t­ic DTrace probes to the linuxulator.

The changes are not in HEAD, but in my linuxulator-dtrace branch. The revi­sion to have a look at is r230910. Includ­ed are some DTrace scripts:

  • script to check inter­nal locks
  • script to trace futexes
  • script to gen­er­ate stats for DTracified lin­ux­u­la­tor parts
  • script to check for errors: 
    • emu­la­tion errors (unsup­port­ed stuff, unknown stuff, …)
    • ker­nel errors (resource shortage, …)
    • pro­gram­ming errors (errors which can hap­pen if some­one made a mis­take, but should not happen)

The programming-error checks give hints about user­land pro­gram­ming errors respec­tive­ly a hint about the rea­son of error return val­ues due to resource short­age or maybe a wrong com­bi­na­tion of para­me­ters. An exam­ple error mes­sage for this case is “Appli­ca­tion %s issued a sysctl which failed the length restrictions.\nThe length passed is %d, the min length sup­port­ed is 1 and the max length sup­port­ed is %d.\n”.

The stats-script (tai­lored spe­cial­ly to the lin­ux­u­la­tor, but this can eas­i­ly be extend­ed to the rest of the ker­nel) can report about:

  • num­ber of calls to a ker­nel func­tion per exe­cutable bina­ry (not per PID!): allows to see where an opti­miza­tion would be ben­e­fi­cial for a giv­en application
  • graph of CPU time spend in ker­nel func­tions per exe­cutable bina­ry: togeth­er with the num­ber of calls to this func­tion this allows to deter­mine if a ker­nel opti­miza­tion would be ben­e­fi­cial / is pos­si­ble for a giv­en application
  • graph of longest run­ning (CPU-time!) ker­nel func­tion in total
  • tim­ing sta­tis­tics for the emul_lock
  • graph of longest held (CPU-time!) locks

Unfor­tu­nate­ly this can not be com­mit­ted to HEAD as-is. The DTrace SDT provider can not han­dle probes which are added to the ker­nel after the SDT provider is already loaded. This means that you either have to com­pile the lin­ux­u­la­tor sta­t­i­cal­ly into the ker­nel, or you have to load the SDT ker­nel mod­ule after the lin­ux­u­la­tor mod­ule is loaded. If you do not respect this, you get a ker­nel pan­ic on first access of one of the providers in the lin­ux­u­la­tor (AFAIR this includes list­ing the probes avail­able in the kernel).

New DTrace probes for the linuxulator

I for­ward port­ed my DTrace probes for the FreeB­SD lin­ux­u­la­tor from a 2008-current to a recent ‑cur­rent. I have not the com­plete FreeB­SD lin­ux­u­la­tor cov­ered, but a big part is already done. I can check the major locks in the lin­ux­u­la­tor, trace futex­es, and I have a D‑script which yells at a lot of errors which could hap­pen but should not.

Some of my D‑scripts need some changes, as real-world test­ing showed that they are not real­ly work­ing as expect­ed. They can get over­whelmed by the amount of spec­u­la­tion and dynam­ic vari­ables (error mes­sage: dynam­ic vari­able drops with non-empty dirty list). For the dynam­ic vari­ables prob­lem I found a dis­cus­sion on the net with some sug­ges­tions. For the spec­u­la­tion part I expect sim­i­lar tuning-possibilities.

Unfor­tu­nate­ly the D‑script which checks the inter­nal locks fails to com­pile. Seems there is a lit­tle mis­un­der­stand­ing on my side how the D‑language is sup­posed to work.

I try to get some time lat­er to have a look at those problems.

Dur­ing my devel­op­ment I stum­bled over some gener­ic DTrace prob­lems with the SDT provider I use for my probes:

  • If you load the Lin­ux mod­ule after the SDT mod­ule, your sys­tem will pan­ic as soon as you want to access some probes, e.g. “dtrace ‑l” will pan­ic the sys­tem. Load­ing the Lin­ux mod­ule before the SDT mod­ule pre­vents the panic.
  • Unload­ing the SDT mod­ule while the Lin­ux mod­ule with the SDT probes is still loaded pan­ics the sys­tem too. Do not unload the Lin­ux mod­ule if you run with my patch.

Accord­ing to avg@ those are known prob­lems, but I think nobody is work­ing on this. This is bad, because this means I can not com­mit my cur­rent patchset.

If some­one wants to try the new DTrace probes for the lin­ux­u­la­tor, feel free to go to http://www.Leidinger.net/FreeBSD/current-patches/ and down­load linuxulator-dtrace.diff. I do not offer a work­ing hyper­link here on pur­pose, the SDT bugs can hurt if you are not care­ful, and I want to make the use of this patch a strong opt-in because of this. If the patch hurts you, it is your fault, you have been warned.