Sta­t­ic DTrace probes for the lin­ux­u­la­tor updated

I got a lit­tle bit of time to update my 3 year old work of adding sta­t­ic DTrace probes to the linuxulator.

The changes are not in HEAD, but in my linuxulator-dtrace branch. The revi­sion to have a look at is r230910. Includ­ed are some DTrace scripts:

  • script to check inter­nal locks
  • script to trace futexes
  • script to gen­er­ate stats for DTracified lin­ux­u­la­tor parts
  • script to check for errors: 
    • emu­la­tion errors (unsup­port­ed stuff, unknown stuff, …)
    • ker­nel errors (resource shortage, …)
    • pro­gram­ming errors (errors which can hap­pen if some­one made a mis­take, but should not happen)

The programming-error checks give hints about user­land pro­gram­ming errors respec­tive­ly a hint about the rea­son of error return val­ues due to resource short­age or maybe a wrong com­bi­na­tion of para­me­ters. An exam­ple error mes­sage for this case is “Appli­ca­tion %s issued a sysctl which failed the length restrictions.\nThe length passed is %d, the min length sup­port­ed is 1 and the max length sup­port­ed is %d.\n”.

The stats-script (tai­lored spe­cial­ly to the lin­ux­u­la­tor, but this can eas­i­ly be extend­ed to the rest of the ker­nel) can report about:

  • num­ber of calls to a ker­nel func­tion per exe­cutable bina­ry (not per PID!): allows to see where an opti­miza­tion would be ben­e­fi­cial for a giv­en application
  • graph of CPU time spend in ker­nel func­tions per exe­cutable bina­ry: togeth­er with the num­ber of calls to this func­tion this allows to deter­mine if a ker­nel opti­miza­tion would be ben­e­fi­cial / is pos­si­ble for a giv­en application
  • graph of longest run­ning (CPU-time!) ker­nel func­tion in total
  • tim­ing sta­tis­tics for the emul_lock
  • graph of longest held (CPU-time!) locks

Unfor­tu­nate­ly this can not be com­mit­ted to HEAD as-is. The DTrace SDT provider can not han­dle probes which are added to the ker­nel after the SDT provider is already loaded. This means that you either have to com­pile the lin­ux­u­la­tor sta­t­i­cal­ly into the ker­nel, or you have to load the SDT ker­nel mod­ule after the lin­ux­u­la­tor mod­ule is loaded. If you do not respect this, you get a ker­nel pan­ic on first access of one of the providers in the lin­ux­u­la­tor (AFAIR this includes list­ing the probes avail­able in the kernel).