Alexander Leidinger

Just another weblog


The FreeBSD-linuxulator explained (for users)

After another mail where I explained a lit­tle bit of the lin­ux­u­la­tor behav­ior, it is time to try to make an easy text which I can ref­er­ence in future answers. If some­one wants to add parts of this expla­na­tion to the FreeBSD hand­book, go ahead.

Linux emu­la­tion? No, “native” exe­cu­tion (sort of)!

First, the lin­ux­u­la­tor is not an emu­la­tion. It is “just” a binary inter­face which is a lit­tle bit dif­fer­ent from the FreeBSD-“native”-one. This means that the binary files in FreeBSD and Linux are both files which com­ply to the ELF spec­i­fi­ca­tion.

When the FreeBSD ker­nel loads an ELF file, it looks if it is a FreeBSD ELF file or a Linux ELF file (or some other fla­vor it knows about). Based upon this it looks up appro­pri­ate actions in a table for this binary (it can also dif­fer­en­ti­ate between 64-bit and 32-bit, and prob­a­bly other things too).

The FreeBSD-table is always com­piled in (for a bet­ter big pic­ture: at least on an AMD/Intel 64-bit plat­form there is also the pos­si­bil­ity to include a 32-bit ver­sion of this table addi­tion­ally, to be able to exe­cute 32-bit pro­grams on 64-bit sys­tems), and other ones like the Linux one can be loaded addi­tion­ally into the ker­nel (or build sta­t­i­cally in the ker­nel, if desired).

Those tables con­tain some para­me­ters and point­ers which allow to exe­cute the binary. If a pro­gram is mak­ing a sys­tem call, the ker­nel will look up the cor­rect func­tion inside this table. It will do this for FreeBSD bina­ries, and for Linux bina­ries. This means that there is no emulation/simulation (over­head) going on… at least ide­ally. Some behav­ior is a lit­tle bit dif­fer­ently between Linux and FreeBSD, so that a lit­tle bit of translation/house-keeping has to go on for some Linux sys­tem calls for the under­ly­ing FreeBSD ker­nel functions.

This means that a lot of Linux stuff in FreeBSD is han­dled at the same speed as if this Linux pro­gram would be a FreeBSD program.

Linux file/directory tricks

When the ker­nel detects a Linux pro­gram, it is also play­ing some tricks with files and direc­to­ries (also a prop­erty of the above men­tioned table in the ker­nel, so the­o­ret­i­cally the ker­nel could play tricks for FreeBSD pro­grams too).

If you look up for a file or direc­tory /A, the ker­nel will first look for /compat/linux/A, and if it does not find it, it will look for /A. This is impor­tant! For exam­ple if you have an empty /compat/linux/home, any appli­ca­tion which wants to dis­play the con­tents of /home will show /compat/linux/home. As it is empty, you see noth­ing. If this appli­ca­tion does not allow you to enter a direc­tory man­u­ally via the key­board, you have lost (ok, you can remove /compat/linux/home or fill it with what you want to have). If you can enter a direc­tory via the key­board, you could enter /home/yourlogin, this would first let the ker­nel look for /compat/linux/home/yourlogin, and as it can not find it then have a look for /home/yourlogin (which we assume is there), and as such would dis­play the con­tents of your home directory.

This implies sev­eral things:

  • you can hide FreeBSD direc­tory con­tents from Linux pro­grams while still being able to access the content
  • badly” pro­grammed Linux appli­ca­tions (more cor­rectly: Linux pro­grams which make assump­tions which do not hold in FreeBSD) can pre­vent you from access­ing FreeBSD files, or files which are the same in Linux and FreeBSD (like /etc/group which is not avail­able in /compat/linux in the linux_base ports, so that the FreeBSD one is read)
  • you can have dif­fer­ent files for Linux than for FreeBSD

The Linux userland

The linux_base port in FreeBSD is com­ing from a plain instal­la­tion of Linux pack­ages. The dif­fer­ence is that some files are deleted, either because we can not use them in the lin­ux­u­la­tor, or because they exist already in the FreeBSD tree at the same place and we want that the Linux pro­grams use the FreeBSD file (/etc/group and /etc/passwd come to mind). The instal­la­tion also marks binary pro­grams as Linux pro­grams, so that the ker­nel knows which kernel-table to con­sult for sys­tem calls and such (this is not really nec­es­sary for all binary pro­grams, but it is harder to script the cor­rect detec­tion logic, than to just “brand” all binary programs).

Addi­tion­ally some con­fig­u­ra­tions are made to (hope­fully) make it do the right thing out of the box. The com­plete setup of the linux_base ports is done to let Linux pro­grams inte­grate into FreeBSD. This means if you start acroread or skype, you do not want to have to have to con­fig­ure some things in /compat/linux/etc/ first to have your fonts look the same and your user IDs resolved to names (this does not work if you use LDAP or ker­beros or other direc­tory ser­vices for the user/group ID man­age­ment, you need to con­fig­ure this your­self). All this should just work and the appli­ca­tion win­dows shall just pop up on your screen so that you can do what you want to do. Some linux_base ports also do not work on all FreeBSD releases. This can be because some ker­nel fea­tures which this linux_base ports depends upon is not avail­able (yet) in FreeBSD. Because of this you should not choice a linux_base port your­self. Just go and install the pro­gram from the Ports Col­lec­tion and let it install the cor­rect linux_base port auto­mat­i­cally (a dif­fer­ent FreeBSD release may have a dif­fer­ent default linux_base port).

A note of cau­tion, there are instruc­tions out there which tell how to install more recent linux_base ports into FreeBSD releases which do not have them as default. You do this on your own risk, it may or may not work. It depends upon which pro­grams you use and at which ver­sion those pro­grams are (or more tech­ni­cally, which ker­nel fea­tures they depend upon). If it does not work for you, you just have two pos­si­bil­i­ties: revert back and for­get about it, or update your FreeBSD ver­sion to a more recent one (but it could be the case, that even the most recent devel­op­ment ver­sion of FreeBSD does not have sup­port for what you need).

Linux libraries and “ELF file OS ABI invalid”-error messages

Due to the above explained fact about file/directory tricks by the ker­nel, you have to be care­ful with (addi­tional) Linux libraries. When a Linux pro­gram needs some libraries, sev­eral direc­to­ries (spec­i­fied in /compat/linux/etc/ are searched. Let us assume that the /compat/linux/etc/ spec­i­fies to search in /A, /B and /C. This means the FreeBSD ker­nel first gets a request to open /A/libXYZ. Because of this he first tries /compat/linux/A/libXYZ, and if it does not exist he tries /A/libXYZ. When this fails too, the Linux run­time linker tries the next direc­tory in the con­fig, so that the ker­nel looks now for /compat/linux/B/libXYZ and if it does not exist for /B/libXYZ.

Now assume that libXYZ is in /compat/linux/C/ as a Linux library, and in /B as a FreeBSD library. This means that the ker­nel will first find the FreeBSD library /B/libXYZ. The Linux binary which needs it can not do any­thing with this FreeBSD library (which depends upon the FreeBSD syscall table and FreeBSD sym­bols from e.g. libc), and the Linux run­time linker will bail out because of this (actu­ally he sees that the lin is not of the required type by read­ing the ELF header of it). Unfor­tu­nately the Linux run­time linker will not con­tinue to search for another library with the same name in another direc­tory (at least this was the case last time I checked and mod­i­fied the order in which the Linux run­time linker searches for libraries… this has been a while, so he may be smarter now) and you will see the above error mes­sage (if you started the linux pro­gram in a terminal).

The bot­tom line of all this is: the error mes­sage about ELF file OS ABI invalid just means that the Linux pro­gram was not able to find the cor­rect Linux library and got a FreeBSD library instead. Go, install the cor­re­spond­ing Linux library, and make sure the Linux pro­gram can find it instead of the FreeBSD library (do not for­get to run “/compat/linux/sbin/ldconfig –r /compat/linux” if you make changes by hand instead of using a port, else your changes may not be taken into account).

Con­straints regard­ing chroot into /compat/linux

The linux_base ports are designed to have a nice install-and-start expe­ri­ence. The draw­back of this is, that there is not a full Linux sys­tem in /compat/linux, so doing a chroot into /compat/linux will cause trou­ble (depend­ing on what you want to do). If you want to chroot into the linux sys­tem on your FreeBSD machine, you bet­ter install a linux_dist port. A linux_dist port can be installed in par­al­lel to a linux_base port. Both of them are inde­pen­dent and as such you need to redo/copy con­fig­u­ra­tion changes you want to have in both environments.


Tags: , , , , , , , , ,

4 Responses to “The FreeBSD-linuxulator explained (for users)”

  1. Erik Ulven Says:

    Thanks for the expla­na­tion and all the work you have done! I really appre­ci­ate it!

  2. Kaip veikia Linuxulator | Says:

    […] Plačiau […]

  3. Linuxulator expliqué aux utilisateurs at FreeBSD-fr: Les nouvelles du géant en français Says:

    […] Lei­dinger viens de pub­lier un court arti­cle sur son blog expli­quant le fonc­tion­nement de Lin­ux­u­la­tor. Un bon moyen de démys­ti­fier et de bal­ayer les […]

  4. Tweets that mention The FreeBSD-linuxulator explained (for users) | Alexander Leidinger -- Says:

    […] This post was men­tioned on Twit­ter by JJ Merelo and Manuel Tru­jillo, Oliver Herold. Oliver Herold said: THE FREEBSD-LINUXULATOR EXPLAINED #freebsd […]

Leave a Reply