Lin­ux­u­la­tor ex­plained: How to cre­ate Linux bin­ar­ies on FreeBSD

There may by cases where you want to gen­er­ate a Linux bin­ary on a FreeBSD ma­chine. This is not a prob­lem with the linuxu­lat­or, but not with the de­fault linux_​base port.

As you may know, the linux_​base port is de­signed to de­liv­er an in­teg­rated ex­per­i­ence with FreeBSD nat­ive pro­grams. As such some parts of the nat­ive FreeBSD in­fra­struc­ture is used. If you would try to use a Linux-com­piler to gen­er­ate Linux–bin­ar­ies, you would run in­to the prob­lem that by de­fault the FreeBSD in­cludes are used.


To have a fully fea­tured and non-​integrated Linux en­vir­on­ment on your FreeBSD sys­tem either mount an ex­ist­ing (and com­pat­ible) Linux in­stall­a­tion some­where in­to your FreeBSD sys­tem, or in­stall a linux_​dist port. This can be done ad­di­tion­ally to an already in­stalled linux_​base port.


When you have a com­plete Linux en­vir­on­ment avail­able, you need to mount the FreeBSD devfs to /​path/​to/​complete_​linux/​dev, lin­procfs to /​path/​to/​complete_​linux/​proc and lin­sysfs to /​path/​to/​complete_​linux/​sys to have a com­plete setup.

Use it

Now you just need to ch­root in­to this  /​path/​to/​complete_​linux and you configure/​make/​install or whatever you need to do to gen­er­ate your de­sired Linux bin­ary.

Linuxu­lat­or D-​Trace probes com­mit­ted to cur­rent

A while ago I com­mit­ted the linuxu­lat­or D-​Trace probes I talked about earli­er. I waited a little bit for this an­nounce­ment to make sure I have not broken any­thing. Nobody com­plained so far, so I as­sume noth­ing ob­vi­ously bad crept in.

The >500 probes I com­mit­ted do not cov­er the en­tire linuxu­lat­or, but are a good start. Adding new ones is straight for­ward, if someone is in­ter­ested in a ju­ni­or–ker­nel-hack­er task, this would be one. Just ask me (or ask on emu­la­tion@), and I can guide you through it.


Seems I for­got to an­nounce that the linux_​base-​c6 is in the Ports Col­lec­tion now. Well, it is not a re­place­ment for the cur­rent de­fault linux base, the linuxu­lat­or in­fra­struc­ture ports are miss­ing and we need to check if the ker­nel sup­ports enough of 2.6.18 that noth­ing breaks.


To my know­ledge, nobody is work­ing on any­thing of this. Any­one is wel­come to have a look and provide patches.

Linuxu­lat­or pro­gress

This week­end I made some pro­gress in the linuxu­lat­or:

  • I MFCed the re­port­ing of some linux-​syscalls to 9-​stable and 8-​stable.
  • I up­dated my linuxu­lat­or-dtrace patch to a re­cent -cur­rent. I already com­piled it on i386 and arundel@ has it com­piled on amd64. I coun­ted more than 500 new DTrace probes. Now that DTrace res­cans for SDT probes when a ker­nel mod­ule is loaded, there is no ker­nel pan­ic any­more when the linux mod­ule is loaded after the DTrace mod­ules and you want to use DTrace. I try to com­mit this at a morn­ing of a day where I can fix things dur­ing the day in case some prob­lems show up which I did not no­tice dur­ing my test­ing.
  • I cre­ated a PR for portmgr@ to re­po­copy a new linux_​base port.
  • I set the ex­pir­a­tion date of linux_​base-​fc4 (only used by 7.x and up­stream way past its EoL) and all de­pend­ent ports. It is set to the EoL of the last 7.x re­lease, which can not use a later linux_​base port. I also ad­ded a com­ment which ex­plains that the date is the EoL of the last 7.x re­lease.

New op­por­tun­it­ies in the linuxu­lat­or

Last week­end I com­mit­ted some dummy-​syscalls to the linuxu­lat­or in FreeBSD-cur­rent. I also ad­ded some com­ments to syscalls.master which should give a hint which linux ker­nel had them for the first time (if the linux man-​page I looked this up in is cor­rect). So if someone wants to ex­per­i­ment with a high­er com­pat.linux.osrelease than 2.6.16 (as it is needed for a Cen­tOS based linux_​base), he should now get some ker­nel mes­sages about un­im­ple­men­ted sy­scalls in­stead of a si­lent fail­ure.

There may be some low-​hanging fruits in there, but I did not really veri­fy this by check­ing what the dummy sy­scalls are sup­posed to do in linux and if we can eas­ily map this to ex­ist­ing FreeBSD fea­tures. In case someone has a look, please send an email to emulation@​FreeBSD.​org.

New Cen­tOS linux_​base for test­ing soon­ish

It seems my HOWTO cre­ate a new linux_​base port was not too bad. There is now a PR for a Cen­tOS 6 based linux_​base port. I had a quick look at it and it seems that it is nearly us­able to in­clude in­to the Ports Col­lec­tion (the SRPMs need to be ad­ded, but that can be done with­in some minutes).

When FreeBSD 8.3 is re­leased and the Ports Col­lec­tion open for sweep­ing com­mits again, I will ask port­m­gr to do a repo-​copy for the new port and com­mit it. This is just the linux_​base port, not the com­plete in­fra­struc­ture which is needed to com­pletely re­place the cur­rent de­fault linuxu­lat­or user­land. This is just a start. The pro­cess of switch­ing to a more re­cent linux_​base port is a long pro­cess, and in this case de­pends upon enough sup­port in the sup­por­ted FreeBSD re­leases.

At­ten­tion: Any­one in­stalling the port from the PR should be aware that us­ing it is a highly ex­per­i­ment­al task. You need to change the linuxu­lat­or to im­per­son­ate him­self as a linux 2.6.18 ker­nel (de­scribed in the pkg-​message of the port), and the code in FreeBSD is far from sup­port­ing this. Any­one who wants to try it is wel­come, but you have to run FreeBSD-​current as of at least the last week­end, and watch out for ker­nel mes­sages about un­sup­por­ted sy­scalls. Re­ports to emulation@​FreeBSD.​org please, not here on the webpage.

Stat­ic DTrace probes for the linuxu­lat­or up­dated

I got a little bit of time to up­date my 3 year old work of adding stat­ic DTrace probes to the linuxu­lat­or.

The changes are not in HEAD, but in my linuxulator-​dtrace branch. The re­vi­sion to have a look at is r230910. In­cluded are some DTrace scripts:

  • script to check in­tern­al locks
  • script to trace fu­texes
  • script to gen­er­ate stats for DTra­ci­fied linuxu­lat­or parts
  • script to check for er­rors:
    • emu­la­tion er­rors (un­sup­por­ted stuff, un­known stuff, …)
    • ker­nel er­rors (re­source short­age, …)
    • pro­gram­ming er­rors (er­rors which can hap­pen if someone made a mis­take, but should not hap­pen)

The programming-​error checks give hints about user­land pro­gram­ming er­rors re­spect­ively a hint about the reas­on of er­ror re­turn val­ues due to re­source short­age or maybe a wrong com­bin­a­tion of para­met­ers. An ex­ample er­ror mes­sage for this case is “Ap­plic­a­tion %s is­sued a sy­sctl which failed the length restrictions.\nThe length passed is %d, the min length sup­por­ted is 1 and the max length sup­por­ted is %d.\n”.

The stats-​script (tailored spe­cially to the linuxu­lat­or, but this can eas­ily be ex­ten­ded to the rest of the ker­nel) can re­port about:

  • num­ber of calls to a ker­nel func­tion per ex­ecut­able bin­ary (not per PID!): al­lows to see where an op­tim­iz­a­tion would be be­ne­fi­cial for a giv­en ap­plic­a­tion
  • graph of CPU time spend in ker­nel func­tions per ex­ecut­able bin­ary: to­geth­er with the num­ber of calls to this func­tion this al­lows to de­term­ine if a ker­nel op­tim­iz­a­tion would be be­ne­fi­cial /​ is pos­sible for a giv­en ap­plic­a­tion
  • graph of longest run­ning (CPU-​time!) ker­nel func­tion in total
  • tim­ing stat­ist­ics for the emul_​lock
  • graph of longest held (CPU-​time!) locks

Un­for­tu­nately this can not be com­mit­ted to HEAD as-​is. The DTrace SDT pro­vider can not handle probes which are ad­ded to the ker­nel after the SDT pro­vider is already loaded. This means that you either have to com­pile the linuxu­lat­or stat­ic­ally in­to the ker­nel, or you have to load the SDT ker­nel mod­ule after the linuxu­lat­or mod­ule is loaded. If you do not re­spect this, you get a ker­nel pan­ic on first ac­cess of one of the pro­viders in the linuxu­lat­or (AFAIR this in­cludes list­ing the probes avail­able in the ker­nel).