A while ago I com­mit­ted the lin­ux­u­la­tor D-Trace probes I talked about ear­lier. I waited a lit­tle bit for this announce­ment to make sure I have not bro­ken any­thing. Nobody com­plained so far, so I assume noth­ing obvi­ously bad crept in.

The >500 probes I com­mit­ted do not cover the entire lin­ux­u­la­tor, but are a good start. Adding new ones is straight for­ward, if some­one is inter­ested in a junior–ker­nel–hacker task, this would be one. Just ask me (or ask on emu­la­tion@), and I can guide you through it.

This week­end I made some progress in the lin­ux­u­la­tor:

• I MFCed the report­ing of some linux-syscalls to 9-stable and 8-stable.
• I updated my lin­ux­u­la­tor-dtrace patch to a recent -cur­rent. I already com­piled it on i386 and arundel@ has it com­piled on amd64. I counted more than 500 new DTrace probes. Now that DTrace res­cans for SDT probes when a ker­nel mod­ule is loaded, there is no ker­nel panic any­more when the linux mod­ule is loaded after the DTrace mod­ules and you want to use DTrace. I try to com­mit this at a morn­ing of a day where I can fix things dur­ing the day in case some prob­lems show up which I did not notice dur­ing my testing.
• I cre­ated a PR for portmgr@ to repocopy a new linux_base port.
• I set the expi­ra­tion date of linux_base-fc4 (only used by 7.x and upstream way past its EoL) and all depen­dent ports. It is set to the EoL of the last 7.x release, which can not use a later linux_base port. I also added a com­ment which explains that the date is the EoL of the last 7.x release.

If some­one had a look at the ear­lier post about DTrace probes for the Lin­ux­u­la­tor: I updated the patch at the same place. The dif­fer­ence between the pre­vi­ous one is that some D–scripts are fixed now to do what I meant, spe­cially the ones which pro­vide sta­tis­tics output.

I for­ward ported my DTrace probes for the FreeBSD lin­ux­u­la­tor from a 2008-current to a recent –cur­rent. I have not the com­plete FreeBSD lin­ux­u­la­tor cov­ered, but a big part is already done. I can check the major locks in the lin­ux­u­la­tor, trace futexes, and I have a D-script which yells at a lot of errors which could hap­pen but should not.

Some of my D-scripts need some changes, as real-world test­ing showed that they are not really work­ing as expected. They can get over­whelmed by the amount of spec­u­la­tion and dynamic vari­ables (error mes­sage: dynamic vari­able drops with non-empty dirty list). For the dynamic vari­ables prob­lem I found a dis­cus­sion on the net with some sug­ges­tions. For the spec­u­la­tion part I expect sim­i­lar tuning-possibilities.

Unfor­tu­nately the D-script which checks the inter­nal locks fails to com­pile. Seems there is a lit­tle mis­un­der­stand­ing on my side how the D-language is sup­posed to work.

I try to get some time later to have a look at those problems.

Dur­ing my devel­op­ment I stum­bled over some generic DTrace prob­lems with the SDT provider I use for my probes:

• If you load the Linux mod­ule after the SDT mod­ule, your sys­tem will panic as soon as you want to access some probes, e.g. “dtrace –l” will panic the sys­tem. Load­ing the Linux mod­ule before the SDT mod­ule pre­vents the panic.
• Unload­ing the SDT mod­ule while the Linux mod­ule with the SDT probes is still loaded pan­ics the sys­tem too. Do not unload the Linux mod­ule if you run with my patch.

Accord­ing to avg@ those are known prob­lems, but I think nobody is work­ing on this. This is bad, because this means I can not com­mit my cur­rent patchset.

If some­one wants to try the new DTrace probes for the lin­ux­u­la­tor, feel free to go to http://www.Leidinger.net/FreeBSD/current-patches/ and down­load linuxulator-dtrace.diff. I do not offer a work­ing hyper­link here on pur­pose, the SDT bugs can hurt if you are not care­ful, and I want to make the use of this patch a strong opt-in because of this. If the patch hurts you, it is your fault, you have been warned.

Bren­dan Gregg of Sun Ora­cle fame made a good expla­na­tion how to visu­al­ize latency to get a bet­ter under­stand­ing of what is going on (and as such about how to solve bot­tle­necks). I have seen all this already in var­i­ous posts in his blog and in the Ana­lyt­ics pack­age in an Open­Stor­age pre­sen­ta­tion, but the ACM arti­cle sum­ma­rizes it very good.

Unfor­tu­nately Ana­lyt­ics is AFAIK not avail­able in Open­So­laris, so we can not go out and adapt it for FreeBSD (which would prob­a­bly require to port/implement some addi­tional dtrace stuff/probes). I am sure some­thing like this would be very inter­est­ing to all those com­pa­nies which use FreeBSD in an appli­ance (regard­less if it is a stor­age appli­ance like NetApp, or a net­work appli­ance like a Cisco/Juniper router, or any­thing else which has to per­form good).