iocage: HOWTO cre­ate a base­jail from src (instead of from an offi­cial release)

Back­ground

So far I have used ezjail to man­age FreeB­SD jails. I use jails since years to have dif­fer­ent parts of a soft­ware stack in some kind of a con­tain­er (in a ZFS dataset for the filesys­tem side of the con­tain­er). On one hand to not let depen­den­cies of one part of the soft­ware stack have influ­ence of oth­er parts of the soft­ware stack. On the oth­er hand to have the pos­si­bil­i­ty to move parts of the soft­ware stack to a dif­fer­ent sys­tem if nec­es­sary. Nor­mal­ly I run -sta­ble or -cur­rent or more gen­er­al­ly speak­ing, a self‐compiled FreeB­SD on those sys­tems. In ezjail I like the fact that all jails on a sys­tem have one com­mon base­jail under­ly­ing, so that I update one place for the user­land and all jails get the updat­ed code.

Since a while I heard good things about iocage and how it inte­grates ZFS, so I decid­ed to give it a try myself. As iocage does not come with an offi­cial way of cre­at­ing a base­jail (respec­tive­ly a release) from a self‐compiled FreeB­SD (at least doc­u­ment­ed in those places I looked, and yes, I am aware that I can cre­ate a FreeB­SD release myself and use it, but I do not like to have to cre­ate a release addi­tion­al­ly to the build­world I use to update the host sys­tem) here now the short HOWTO achieve this.

Invari­ants

In the fol­low­ing I assume the iocage ZFS parts are already cre­at­ed in dataset ${POOLNAME}/iocage which is mount­ed on ${IOCAGE_BASE}/iocage. Addi­tion­al­ly the build­world in /usr/src (or wher­ev­er you have the FreeB­SD source) should be fin­ished.

Pre‐requisites

To have the nec­es­sary dataset‐infrastructure cre­at­ed for own basejails/releases, at least one offi­cial release needs to be fetched before. So run the com­mand below (if there is no ${IOCAGE_BASE}/iocage/releases direc­to­ry) and fol­low the on‐screen instruc­tions.

iocage fetch

HOWTO

Some vari­ables:

POOLNAME=mpool
SRC_REV=r$(cd /usr/src; svnliteversion)
IOCAGE_BASE=""

Cre­at­ing the iocage basejail‐datasets for this ${SRC_REV}:

zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/bin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/boot
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/lib
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/libexec
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/rescue
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/sbin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/include
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib32
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libdata
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libexec
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/share
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/src

Install from /usr/src (the exe­cutable “chown” is hardlinked across an iocage base­jail dataset bound­ary, this fails in the nor­mal install­world, so we have to ignore this error and install a copy of the chown bina­ry to the place where the hardlink nor­mal­ly is):

cd /usr/src
make -i installworld DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >&! iocage_installworld_base.log
cp -pv ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin/chown ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin/chgrp
make distribution DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >>& iocage_installworld_base.log

While we are here, also cre­ate a release and not only a base­jail:

zfs create -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE
zfs create -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE/root
make installworld DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >&! iocage_installworld_release.log
make distribution DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >>& iocage_installworld_release.log

And final­ly make this the default release which iocage uses when cre­at­ing new jails (this is option­al):

iocage set release=${SRC_REV}-RELEASE default

Now the self‐build FreeB­SD is avail­able in iocage for new jails.

Send to Kin­dle

Sock­ets and nullfs: works now in -cur­rent

I just updat­ed to a recent -cur­rent and tried the new nullfs. Sock­ets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the sock­et to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Send to Kin­dle

X11 in a jail with NVidia hard­ware

Just before christ­mas I decid­ed I will spend the “immense” amount of 40 EUR for a graph­ic card for a sys­tem which was with­out one. The sys­tem is sup­posed to replace my dying home‐server. I already moved every­thing, except my Desktop‐in‐a‐Jail (actu­al­ly it is my home‐cinema‐jail).

The old sys­tem had a Radeon 9200SE, and it was enough for what I used it for. Now… for a few bucks you can get a lot more horse­pow­er today. After look­ing around a lit­tle bit I decid­ed to buy a NVidia card. I made this deci­sion because it looks like I can get bet­ter dri­ver sup­port for it. So I got a GeForce GT 520 with 1 GB of RAM (I doubt I will be able to use that much RAM) and with­out a fan.

With the Radeon 9200SE I was not able to get the 3D stuff acti­vat­ed (at least in the jail, I did not try with­out), Xorg com­plains about a miss­ing agp­gart mod­ule but I have AGP in the ker­nel (no /dev/agpgart out­side the jail). I did not spend time to inves­ti­gate this, as the main pur­pose – play­ing movies – worked. Now with the NVidia card I decid­ed to give the 3D part a try again.

After adding the NVidia device entries to the jail, and a lit­tle bit of fight­ing with the Xorg‐HAL inter­ac­tion, I got a work­ing desk­top. The biggest prob­lem to ver­i­fy that 3D is work­ing was, that I did not had xdri­in­fo installed. After installing it, I noticed that it does not work with the NVidia dri­ver.  🙁  Next stop nvidia‐settings: runs great, dis­plays a nice FreeB­SD+NVidia logo, and … tells me that OpenGL is con­fig­ured. Hmmm… OK, but I want to see it!

As I decid­ed to switch from Gnome to KDE 4 at  the same time (I was using KDE when it was at V 0.x, switched to Gnome as it looked nicer to me, and now I switch back after read­ing all the stuff in the net that KDE 4 is “bet­ter” than Gnome 3), I was a lit­tle bit out of knowl­edge how to see the 3D stuff in action. So I quick­ly went to the set­tings and searched for some­thing which looks like it may use 3D. To my sur­prise, it was already using 3D stuff. Nice. I ful­ly real­ized how nice, when play­ing a video and using Alt‐Tab to switch win­dows: the video was play­ing full speed scaled down in the window‐switcher‐thumbnail‐view.

That was too easy. I am hap­py about it.

Now that I have a work­ing set­up of X11‐in‐a‐jail for Radeon and GeForce cards, I want to cleanup my changes to the ker­nel and the con­fig files (devfs.rules) and have a look to get this com­mit­ted. A big part of this work is prob­a­bly writ­ing doc­u­men­ta­tion (most prob­a­bly in the wiki).

I still want to see some fan­cy 3D stuff now. I tried to install x11‐clocks/glclock, but the build fails with an unde­fined ref­er­ence to ‘glPoly­gonOff­se­tEXT’. 🙁 Any rec­om­men­da­tion for a fan­cy 3D dis­play? My pri­or­i­ty is on “fancy/nice” with as less vio­lence as pos­si­ble. Most prob­a­bly I will look at it once and then dein­stall it again, so it should be avail­able in the Ports Col­lec­tion (or includ­ed in KDE 4).

Send to Kin­dle

How I set­up a Jail‐Host

Every­one has his own way of set­ting up a machine to serve as a host of mul­ti­ple jails. Here is my way, YMMV.

Ini­tial FreeB­SD install

I use sev­er­al hard­disks in a Soft­ware-RAID set­up. It does not mat­ter much if you set them up with one big par­ti­tion or with sev­er­al par­ti­tions, feel free to fol­low your pref­er­ences here. My way of par­ti­tion­ing the hard­disks is described in a pre­vi­ous post. That post only shows the com­mands to split the hard­disks into two par­ti­tions and use ZFS for the rootfs. The com­mands to ini­tial­ize the ZFS data par­ti­tion are not described, but you should be able to fig­ure it out your­self (and you can decide on your own what kind of RAID lev­el you want to use). For this FS I set atime, exec and setu­id to off in the ZFS options.

On the ZFS data par­ti­tion I cre­ate a new dataset for the sys­tem. For this dataset I set atime, exec and setu­id to off in the ZFS options. Inside this dataset I cre­ate datasets for /home, /usr/compat, /usr/local, /usr/obj, /usr/ports/, /usr/src, /usr/sup and /var/ports. There are two ways of doing this. One way is to set the ZFS mount­point. The way I pre­fer is to set rel­a­tive sym­links to it, e.g. “cd /usr; ln -s ../data/system/usr_obj obj”. I do this because this way I can tem­po­rary import the pool on anoth­er machine (e.g. my desk­top, if the need aris­es) with­out fear to inter­fere with the sys­tem. The ZFS options are set as fol­lows:

ZFS options for data/system/*

Dataset

Option

Val­ue
data/system/home exec on
data/system/usr_compat exec on
data/system/usr_compat setu­id on
data/system/usr_local exec on
data/system/usr_local setu­id on
data/system/usr_obj exec on
data/system/usr_ports exec on
data/system/usr_ports setu­id on
data/system/usr_src exec on
data/system/usr_sup sec­ondarycache none
data/system/var_ports exec on

The exec option for home is not nec­es­sary if you keep sep­a­rate datasets for each user. Nor­mal­ly I keep sep­a­rate datasets for home direc­to­ries, but Jail‐Hosts should not have users (except the admins, but they should not keep data in their homes), so I just cre­ate a sin­gle home dataset. The setu­id option for the usr_ports should not be nec­es­sary if you redi­rect the build direc­to­ry of the ports to a dif­fer­ent place (WRKDIRPREFIX in /etc/make.conf).

Installing ports

The ports I install by default are net/rsync, ports‐mgmt/portaudit, ports‐mgmt/portmaster, shells/zsh, sysutils/bsdstats, sysutils/ezjail, sysutils/smartmontools and sysutils/tmux.

Basic set­up

In the crontab of root I set­up a job to do a port­snap update once a day (I pick a ran­dom num­ber between 0 and 59 for the minute, but keep a fixed hour). I also have http_proxy spec­i­fied in /etc/profile, so that all machines in this net­work do not down­load every­thing from far away again and again, but can get the data from the local caching proxy. As a lit­tle watch­dog I have a lit­tle @reboot rule in the crontab, which noti­fies me when a machine reboots:

@reboot grep "kernel boot file is" /var/log/messages | mail -s "`hostname` rebooted" root >/dev/null 2>&1

This does not replace a real mon­i­tor­ing solu­tion, but in cas­es where real mon­i­tor­ing is overkill it pro­vides a nice HEADSUP (and shows you direct­ly which ker­nel is loaded in case a non‐default one is used).

Some default alias­es I use every­where are:

alias portmlist="portmaster -L | egrep -B1 '(ew|ort) version|Aborting|installed|dependencies|IGNORE|marked|Reason:|MOVED|deleted|exist|update' | grep -v '^--'"
alias portmclean="portmaster -t --clean-distfiles --clean-packages"
alias portmcheck="portmaster -y --check-depends"

Addi­tion­al devfs rules for Jails

I have the need to give access to some spe­cif­ic devices in some jails. For this I need to set­up a cus­tom /etc/devfs.rules file. The files con­tains some ID num­bers which need to be unique in the sys­tem. On a 9‐current sys­tem the num­bers one to four are already used (see /etc/defaults/devfs.rules). The next avail­able num­ber is obvi­ous­ly five then. First I present my devfs.rules entries, then I explain them:

[devfsrules_unhide_audio=5]
add path 'audio*' unhide
add path 'dsp*' unhide
add path midistat unhide
add path 'mixer*' unhide
add path 'music*' unhide
add path 'sequencer*' unhide
add path sndstat unhide
add path speaker unhide

[devfsrules_unhide_printers=6]
add path 'lpt*' unhide
add path 'ulpt*' unhide user 193 group 193
add path 'unlpt*' unhide user 193 group 193

[devfsrules_unhide_zfs=7]
add path zfs unhide

[devfsrules_jail_printserver=8]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_printers
add include $devfsrules_unhide_zfs

[devfsrules_jail_withzfs=9]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_zfs

The devfs_rules_unhide_XXX ones give access to spe­cif­ic devices, e.g. all the sound relat­ed devices or to local print­ers. The devfsrules_jail_XXX ones com­bine all the unhide rules for spe­cif­ic jail setups. Unfor­tu­nate­ly the include direc­tive is not recur­sive, so that we can not include the default devfsrules_jail pro­file and need to repli­cate its con­tents. The first three includes of each devfsrules_jail_XXX accom­plish this. The unhide_zfs rule gives access to /dev/zfs, which is need­ed if you attach one or more ZFS datasets to a jail. I will explain how to use those pro­files with ezjail in a follow‐up post.

Jails set­up

I use ezjail to man­age jails, it is more com­fort­able than doing it by hand while at the same time allows me to do some­thing by hand. My jails nor­mal­ly reside inside ZFS datasets, for this rea­son I have set­up a spe­cial area (ZFS dataset data/jails) which is han­dled by ezjail.The cor­re­spond­ing ezjail.conf set­tings are:

ezjail_jaildir=/data/jails
ezjail_use_zfs="YES"
ezjail_jailzfs="data/jails"

I also dis­abled procfs and fde­scfs in jails (but they can be enabled lat­er for spe­cif­ic jails if nec­es­sary).

Unfor­tu­nate­ly ezjail (as of v3.1) sets the mount­point of a new­ly cre­at­ed dataset even if it is not nec­es­sary. For this rea­son I always issue a “zfs inher­it mount­point ” after cre­at­ing a jail. This sim­pli­fies the case where you want to move/rename a dataset and want to have the mount­point autom­cat­i­cal­ly fol­low the change.

The access flags of  /data/jails direc­to­ry are 700, this pre­vents local users (there should be none, but bet­ter safe than sor­ry) to get access to files from users in jails with the same UID.

After the first create/update of the ezjail base­jail the ZFS options of base­jail (data/jails/basejail) and new­jail (data/jails/newjail) need to be changed. For both exec and setu­id should be changed to “on” The same needs to be done after cre­at­ing a new jail for the new jail (before start­ing it).

The default ezjail flavour

In my default ezjail flavour I cre­ate some default user(s) with a basesystem‐shell (via /data/jails/flavours/mydef/ezjail.flavour) before the pack­age install, and change the shell to my pre­ferred zsh after­wards (this is only valid if the jails are used only by in‐house peo­ple, if you want to offer light­weight vir­tu­al machines to (unknown) cus­tomers, the default user(s) and shell(s) are obvi­ous­ly up to dis­cus­sion). At the end I also run a “/usr/local/sbin/portmaster -y –check‐depends” to make sure every­thing is in a sane state.

For the pack­ages (/data/jails/flavours/mydef/pkg/) I add sym­links to the unver­sioned pack­ages I want to install. I have the pack­ages in a com­mon (think about set­ting PACKAGES in make.conf and using PACKAGES/Latest/XYZ.tbz) direc­to­ry (if they can be shared over var­i­ous flavours), and they are unver­sioned so that I do not have to update the ver­sion num­ber each time there is an update. The pack­ages I install by default are bsd­stats, por­tau­dit, port­mas­ter, zsh, tmux and all their depen­den­cies.

In case you use jails to vir­tu­al­ize ser­vices and con­sol­i­date servers (e.g. DNS, HTTP, MySQL each in a sep­a­rate jail) instead of pro­vid­ing light­weight vir­tu­al machines to (unknown) cus­tomers, there is also a ben­e­fit of shar­ing the dis­t­files and pack­ages between jails on the same machine. To do this I cre­ate /data/jails/flavours/mydef/shared/ports/{distfiles,packages} which are then mount­ed via nullfs or NFS into all the jails from a com­mon direc­to­ry. This requires the fol­low­ing vari­ables in /data/jails/flavours/mydef/etc/make.conf (I also keep the pack­ages for dif­fer­ent CPU types and com­pil­ers in the same sub­tree, if you do not care, just remove the “/${CC}/${CPUTYPE}” from the PACAKGES line):

DISTDIR=  /shared/ports/distfiles
PACKAGES= /shared/ports/packages/${CC}/${CPUTYPE}

New jails

A future post will cov­er how I set­up new jails in such a set­up and how I cus­tomize the start order of jails or use some non‐default set­tings for the jail‐startup.

Send to Kin­dle

All inter­nal ser­vices migrat­ed to IPv6

In the last days I migrat­ed all my inter­nal ser­vices to IPv6.

All my jails have an IPv4 and an IPv6 address now. All Apach­es (I have one for my pic­ture gallery, one for web­mail, and one for inter­nal man­age­ment) now lis­ten on the inter­nal IPv6 address too. Squid is updat­ed from 2.x to 3.1 (the most recent ver­sion in the Ports Col­lec­tion) and I added some IPv6 ACLs. The inter­nal Post­fix is con­fig­ured to han­dle IPv6 too (it is deliv­er­ing every­thing via an authen­ti­cat­ed and encrypt­ed chan­nel to a machine with a sta­t­ic IPv4 address for final deliv­ery). My MySQL does not need an IPv6 address, as it is only lis­ten­ing to requests via IPC (the sock­et is hardlinked between jails). All ssh dae­mons are con­fig­ured to lis­ten to IPv6 too. The IMAP and CUPS serv­er was pick­ing the new IPv6 address­es auto­mat­i­cal­ly. I also updat­ed Sam­ba to han­dle IPv6, but due to lack of a Win­dows machine which prefers IPv6 over IPv4 for CIFS access (at least I think my Win­dows XP net­book only tries IPv4 con­nec­tions) I can not real­ly test this.

Only my Wii is a lit­tle bit behind, and I have not checked if my Sony‐TV will DTRT (but for this I first have to get some time to have a look if I have to update my DDWRT firmware on the lit­tle WLAN‐router which is “extend­ing the cable” from the TV to the inter­nal net­work, and I have to look how to con­fig­ure IPv6 with DDWRT).

Send to Kin­dle