The FreeBSD-​linuxulator ex­plained (for users)

After an­oth­er mail where I ex­plained a little bit of the linuxu­lat­or be­ha­vi­or, it is time to try to make an easy text which I can ref­er­ence in fu­ture an­swers. If someone wants to add parts of this ex­plan­a­tion to the FreeBSD hand­book, go ahead.

Linux emu­la­tion? No, “nat­ive” ex­e­cu­tion (sort of)!

First, the linuxu­lat­or is not an emu­la­tion. It is “just” a bin­ary in­ter­face which is a little bit dif­fer­ent from the FreeBSD-“native”-one. This means that the bin­ary files in FreeBSD and Linux are both files which com­ply to the ELF spe­cific­a­tion.

When the FreeBSD ker­nel loads an ELF file, it looks if it is a FreeBSD ELF file or a Linux ELF file (or some oth­er fla­vor it knows about). Based upon this it looks up ap­pro­pri­ate ac­tions in a table for this bin­ary (it can also dif­fer­en­ti­ate between 64-​bit and 32-​bit, and prob­ably oth­er things too).

The FreeBSD-​table is al­ways com­piled in (for a bet­ter big pic­ture: at least on an AMD/​Intel 64-​bit plat­form there is also the pos­sib­il­ity to in­clude a 32-​bit ver­sion of this table ad­di­tion­ally, to be able to ex­ecute 32-​bit pro­grams on 64-​bit sys­tems), and oth­er ones like the Linux one can be loaded ad­di­tion­ally in­to the ker­nel (or build stat­ic­ally in the ker­nel, if de­sired).

Those tables con­tain some para­met­ers and point­ers which al­low to ex­ecute the bin­ary. If a pro­gram is mak­ing a sys­tem call, the ker­nel will look up the cor­rect func­tion in­side this table. It will do this for FreeBSD bin­ar­ies, and for Linux bin­ar­ies. This means that there is no emulation/​simulation (over­head) go­ing on… at least ideally. Some be­ha­vi­or is a little bit dif­fer­ently between Linux and FreeBSD, so that a little bit of translation/​house-​keeping has to go on for some Linux sys­tem calls for the un­der­ly­ing FreeBSD ker­nel func­tions.

This means that a lot of Linux stuff in FreeBSD is handled at the same speed as if this Linux pro­gram would be a FreeBSD pro­gram.

Linux file/​directory tricks

When the ker­nel de­tects a Linux pro­gram, it is also play­ing some tricks with files and dir­ect­or­ies (also a prop­erty of the above men­tioned table in the ker­nel, so the­or­et­ic­ally the ker­nel could play tricks for FreeBSD pro­grams too).

If you look up for a file or dir­ect­ory /​A, the ker­nel will first look for /​compat/​linux/​A, and if it does not find it, it will look for /​A. This is im­port­ant! For ex­ample if you have an empty /​compat/​linux/​home, any ap­plic­a­tion which wants to dis­play the con­tents of /​home will show /​compat/​linux/​home. As it is empty, you see noth­ing. If this ap­plic­a­tion does not al­low you to enter a dir­ect­ory manu­ally via the key­board, you have lost (ok, you can re­move /​compat/​linux/​home or fill it with what you want to have). If you can enter a dir­ect­ory via the key­board, you could enter /​home/​yourlogin, this would first let the ker­nel look for /​compat/​linux/​home/​yourlogin, and as it can not find it then have a look for /​home/​yourlogin (which we as­sume is there), and as such would dis­play the con­tents of your home dir­ect­ory.

This im­plies sev­er­al things:

  • you can hide FreeBSD dir­ect­ory con­tents from Linux pro­grams while still be­ing able to ac­cess the con­tent
  • “badly” pro­grammed Linux ap­plic­a­tions (more cor­rectly: Linux pro­grams which make as­sump­tions which do not hold in FreeBSD) can pre­vent you from ac­cess­ing FreeBSD files, or files which are the same in Linux and FreeBSD (like /​etc/​group which is not avail­able in /​compat/​linux in the linux_​base ports, so that the FreeBSD one is read)
  • you can have dif­fer­ent files for Linux than for FreeBSD

The Linux user­land

The linux_​base port in FreeBSD is com­ing from a plain in­stall­a­tion of Linux pack­ages. The dif­fer­ence is that some files are de­leted, either be­cause we can not use them in the linuxu­lat­or, or be­cause they ex­ist already in the FreeBSD tree at the same place and we want that the Linux pro­grams use the FreeBSD file (/​etc/​group and /​etc/​passwd come to mind). The in­stall­a­tion also marks bin­ary pro­grams as Linux pro­grams, so that the ker­nel knows which kernel-​table to con­sult for sys­tem calls and such (this is not really ne­ces­sary for all bin­ary pro­grams, but it is harder to script the cor­rect de­tec­tion lo­gic, than to just “brand” all bin­ary pro­grams).

Ad­di­tion­ally some con­fig­ur­a­tions are made to (hope­fully) make it do the right thing out of the box. The com­plete setup of the linux_​base ports is done to let Linux pro­grams in­teg­rate in­to FreeBSD. This means if you start acror­ead or skype, you do not want to have to have to con­fig­ure some things in /​compat/​linux/​etc/​ first to have your fonts look the same and your user IDs re­solved to names (this does not work if you use LDAP or ker­ber­os or oth­er dir­ect­ory ser­vices for the user/​group ID man­age­ment, you need to con­fig­ure this your­self). All this should just work and the ap­plic­a­tion win­dows shall just pop up on your screen so that you can do what you want to do. Some linux_​base ports also do not work on all FreeBSD re­leases. This can be be­cause some ker­nel fea­tures which this linux_​base ports de­pends upon is not avail­able (yet) in FreeBSD. Be­cause of this you should not choice a linux_​base port your­self. Just go and in­stall the pro­gram from the Ports Col­lec­tion and let it in­stall the cor­rect linux_​base port auto­mat­ic­ally (a dif­fer­ent FreeBSD re­lease may have a dif­fer­ent de­fault linux_​base port).

A note of cau­tion, there are in­struc­tions out there which tell how to in­stall more re­cent linux_​base ports in­to FreeBSD re­leases which do not have them as de­fault. You do this on your own risk, it may or may not work. It de­pends upon which pro­grams you use and at which ver­sion those pro­grams are (or more tech­nic­ally, which ker­nel fea­tures they de­pend upon). If it does not work for you, you just have two pos­sib­il­it­ies: re­vert back and for­get about it, or up­date your FreeBSD ver­sion to a more re­cent one (but it could be the case, that even the most re­cent de­vel­op­ment ver­sion of FreeBSD does not have sup­port for what you need).

Linux lib­rar­ies and “ELF file OS ABI invalid”-error mes­sages

Due to the above ex­plained fact about file/​directory tricks by the ker­nel, you have to be care­ful with (ad­di­tion­al) Linux lib­rar­ies. When a Linux pro­gram needs some lib­rar­ies, sev­er­al dir­ect­or­ies (spe­cified in /compat/linux/etc/ are searched. Let us as­sume that the /compat/linux/etc/ spe­cifies to search in /​A, /​B and /​C. This means the FreeBSD ker­nel first gets a re­quest to open /​A/​libXYZ. Be­cause of this he first tries /​compat/​linux/​A/​libXYZ, and if it does not ex­ist he tries /​A/​libXYZ. When this fails too, the Linux runtime linker tries the next dir­ect­ory in the con­fig, so that the ker­nel looks now for /​compat/​linux/​B/​libXYZ and if it does not ex­ist for /​B/​libXYZ.

Now as­sume that libXYZ is in /​compat/​linux/​C/​ as a Linux lib­rary, and in /​B as a FreeBSD lib­rary. This means that the ker­nel will first find the FreeBSD lib­rary /​B/​libXYZ. The Linux bin­ary which needs it can not do any­thing with this FreeBSD lib­rary (which de­pends upon the FreeBSD sy­scall table and FreeBSD sym­bols from e.g. libc), and the Linux runtime linker will bail out be­cause of this (ac­tu­ally he sees that the lin is not of the re­quired type by read­ing the ELF head­er of it). Un­for­tu­nately the Linux runtime linker will not con­tin­ue to search for an­oth­er lib­rary with the same name in an­oth­er dir­ect­ory (at least this was the case last time I checked and mod­i­fied the or­der in which the Linux runtime linker searches for lib­rar­ies… this has been a while, so he may be smarter now) and you will see the above er­ror mes­sage (if you star­ted the linux pro­gram in a ter­min­al).

The bot­tom line of all this is: the er­ror mes­sage about ELF file OS ABI in­val­id just means that the Linux pro­gram was not able to find the cor­rect Linux lib­rary and got a FreeBSD lib­rary in­stead. Go, in­stall the cor­res­pond­ing Linux lib­rary, and make sure the Linux pro­gram can find it in­stead of the FreeBSD lib­rary (do not for­get to run “/​compat/​linux/​sbin/​ldconfig -r /​compat/​linux” if you make changes by hand in­stead of us­ing a port, else your changes may not be taken in­to ac­count).

Con­straints re­gard­ing ch­root in­to /​compat/​linux

The linux_​base ports are de­signed to have a nice install-​and-​start ex­per­i­ence. The draw­back of this is, that there is not a full Linux sys­tem in /​compat/​linux, so do­ing a ch­root in­to /​compat/​linux will cause trouble (de­pend­ing on what you want to do). If you want to ch­root in­to the linux sys­tem on your FreeBSD ma­chine, you bet­ter in­stall a linux_​dist port. A linux_​dist port can be in­stalled in par­al­lel to a linux_​base port. Both of them are in­de­pend­ent and as such you need to redo/​copy con­fig­ur­a­tion changes you want to have in both en­vir­on­ments.

4 thoughts on “The FreeBSD-​linuxulator ex­plained (for users)”

  1. Pingback: Linuxulator expliqué aux utilisateurs at FreeBSD-fr: Les nouvelles du géant en français
  2. Pingback: Tweets that mention The FreeBSD-linuxulator explained (for users) | Alexander Leidinger --

Leave a Reply

Your email address will not be published. Required fields are marked *