How I setup a Jail-​Host

Every­one has his own way of set­ting up a ma­chine to serve as a host of mul­tiple jails. Here is my way, YMMV.

Ini­tial FreeBSD in­stall

I use sev­eral hard­disks in a Soft­ware–RAID setup. It does not mat­ter much if you set them up with one big par­ti­tion or with sev­eral par­ti­tions, feel free to fol­low your pref­er­ences here. My way of par­ti­tion­ing the hard­disks is de­scribed in a pre­vi­ous post. That post only shows the com­mands to split the hard­disks into two par­ti­tions and use ZFS for the rootfs. The com­mands to ini­tial­ize the ZFS data par­ti­tion are not de­scribed, but you should be able to fig­ure it out your­self (and you can de­cide on your own what kind of RAID level you want to use). For this FS I set atime, exec and setuid to off in the ZFS op­tions.

On the ZFS data par­ti­tion I cre­ate a new data­set for the sys­tem. For this data­set I set atime, exec and setuid to off in the ZFS op­tions. In­side this data­set I cre­ate data­sets for /​home, /​usr/​compat, /​usr/​local, /​usr/​obj, /​usr/​ports/​, /​usr/​src, /​usr/​sup and /​var/​ports. There are two ways of do­ing this. One way is to set the ZFS moun­t­point. The way I prefer is to set re­l­at­ive sym­links to it, e.g. “cd /​usr; ln –s ../​data/​system/​usr_​obj obj”. I do this be­cause this way I can tem­por­ary im­port the pool on an­other ma­chine (e.g. my desktop, if the need arises) without fear to in­ter­fere with the sys­tem. The ZFS op­tions are set as fol­lows:

ZFS op­tions for data/​system/​*

Data­set

Op­tion

Value
data/​system/​home exec on
data/​system/​usr_​compat exec on
data/​system/​usr_​compat setuid on
data/​system/​usr_​local exec on
data/​system/​usr_​local setuid on
data/​system/​usr_​obj exec on
data/​system/​usr_​ports exec on
data/​system/​usr_​ports setuid on
data/​system/​usr_​src exec on
data/​system/​usr_​sup sec­ond­arycache none
data/​system/​var_​ports exec on

The exec op­tion for home is not ne­ces­sary if you keep sep­ar­ate data­sets for each user. Nor­mally I keep sep­ar­ate data­sets for home dir­ect­or­ies, but Jail-​Hosts should not have users (ex­cept the ad­mins, but they should not keep data in their homes), so I just cre­ate a single home data­set. The setuid op­tion for the usr_​ports should not be ne­ces­sary if you re­dir­ect the build dir­ect­ory of the ports to a dif­fer­ent place (WRKDIRPREFIX in /etc/make.conf).

In­stalling ports

The ports I in­stall by de­fault are net/​rsync, ports-​mgmt/​portaudit, ports-​mgmt/​portmaster, shells/​zsh, sysutils/​bsdstats, sysutils/​ezjail, sysutils/​smartmontools and sysutils/​tmux.

Ba­sic setup

In the crontab of root I setup a job to do a portsnap up­date once a day (I pick a ran­dom num­ber between 0 and 59 for the minute, but keep a fixed hour). I also have http_​proxy spe­cified in /​etc/​profile, so that all ma­chines in this net­work do not down­load everything from far away again and again, but can get the data from the local cach­ing proxy. As a little watch­dog I have a little @reboot rule in the crontab, which no­ti­fies me when a ma­chine re­boots:

@reboot grep “ker­nel boot file is” /​var/​log/​messages | mail –s “„host­name„ re­booted” root >/​dev/​null 2>&1

This does not re­place a real mon­it­or­ing solu­tion, but in cases where real mon­it­or­ing is overkill it provides a nice HEADS-​UP (and shows you dir­ectly which ker­nel is loaded in case a non-​default one is used).

Some de­fault ali­ases I use every­where are:

alias portmlist=“portmaster –L | egrep –B1 „(ew|ort) version|Aborting|installed|dependencies|IGNORE|marked|Reason:|MOVED|deleted|exist|update“ | grep –v „^ – “”
alias portmclean=“portmaster –t –clean-​distfiles –clean-​packages“
alias portmcheck=“portmaster –y –check-​depends”

Ad­di­tional devfs rules for Jails

I have the need to give ac­cess to some spe­cific devices in some jails. For this I need to setup a cus­tom /etc/devfs.rules file. The files con­tains some ID num­bers which need to be unique in the sys­tem. On a 9-​current sys­tem the num­bers one to four are already used (see /etc/defaults/devfs.rules). The next avail­able num­ber is ob­vi­ously five then. First I present my devfs.rules entries, then I ex­plain them:

[devfsrules_unhide_audio=5]
add path „au­dio*“ un­hide
add path „dsp*“ un­hide
add path midistat un­hide
add path „mixer*“ un­hide
add path „mu­sic*“ un­hide
add path „se­quen­cer*“ un­hide
add path snd­stat un­hide
add path speaker un­hide

[devfsrules_unhide_printers=6] add path „lpt*“ un­hide add path „ulpt*“ un­hide user 193 group 193 add path „un­lpt*“ un­hide user 193 group 193 

[devfsrules_unhide_zfs=7] add path zfs un­hide

[devfsrules_jail_printserver=8] add in­clude $devfsrules_​hide_​all add in­clude $devfsrules_​unhide_​basic add in­clude $devfsrules_​unhide_​login add in­clude $devfsrules_​unhide_​printers add in­clude $devfsrules_​unhide_​zfs

[devfsrules_jail_withzfs=9] add in­clude $devfsrules_​hide_​all add in­clude $devfsrules_​unhide_​basic add in­clude $devfsrules_​unhide_​login add in­clude $devfsrules_​unhide_​zfs

The devfs_​rules_​unhide_​XXX ones give ac­cess to spe­cific devices, e.g. all the sound re­lated devices or to local print­ers. The devfsrules_​jail_​XXX ones com­bine all the un­hide rules for spe­cific jail setups. Un­for­tu­nately the in­clude dir­ect­ive is not re­curs­ive, so that we can not in­clude the de­fault devfsrules_​jail pro­file and need to rep­lic­ate its con­tents. The first three in­cludes of each devfsrules_​jail_​XXX ac­com­plish this. The unhide_​zfs rule gives ac­cess to /​dev/​zfs, which is needed if you at­tach one or more ZFS data­sets to a jail. I will ex­plain how to use those pro­files with ez­jail in a follow-​up post.

Jails setup

I use ez­jail to man­age jails, it is more com­fort­able than do­ing it by hand while at the same time al­lows me to do some­thing by hand. My jails nor­mally reside in­side ZFS data­sets, for this reason I have setup a spe­cial area (ZFS data­set data/​jails) which is handled by ezjail.The cor­res­pond­ing ezjail.conf set­tings are:

ezjail_jaildir=/data/jails
ezjail_use_zfs=“YES“
ezjail_jailzfs=“data/jails”

I also dis­abled procfs and fdescfs in jails (but they can be en­abled later for spe­cific jails if ne­ces­sary).

Un­for­tu­nately ez­jail (as of v3.1) sets the moun­t­point of a newly cre­ated data­set even if it is not ne­ces­sary. For this reason I al­ways is­sue a “zfs in­herit moun­t­point ” after cre­at­ing a jail. This sim­pli­fies the case where you want to move/​rename a data­set and want to have the moun­t­point autom­c­at­ic­ally fol­low the change.

The ac­cess flags of  /​data/​jails dir­ect­ory are 700, this pre­vents local users (there should be none, but bet­ter safe than sorry) to get ac­cess to files from users in jails with the same UID.

After the first create/​update of the ez­jail base­jail the ZFS op­tions of base­jail (data/​jails/​basejail) and new­jail (data/​jails/​newjail) need to be changed. For both exec and setuid should be changed to “on” The same needs to be done after cre­at­ing a new jail for the new jail (be­fore start­ing it).

The de­fault ez­jail fla­vour

In my de­fault ez­jail fla­vour I cre­ate some de­fault user(s) with a basesys­tem–shell (via /data/jails/flavours/mydef/ezjail.flavour) be­fore the pack­age in­stall, and change the shell to my pre­ferred zsh af­ter­wards (this is only valid if the jails are used only by in-​house people, if you want to of­fer light­weight vir­tual ma­chines to (un­known) cus­tom­ers, the de­fault user(s) and shell(s) are ob­vi­ously up to dis­cus­sion). At the end I also run a “/​usr/​local/​sbin/​portmaster –y –check-​depends” to make sure everything is in a sane state.

For the pack­ages (/​data/​jails/​flavours/​mydef/​pkg/​) I add sym­links to the un­ver­sioned pack­ages I want to in­stall. I have the pack­ages in a com­mon (think about set­ting PACKAGES in make.conf and us­ing PACKAGES/Latest/XYZ.tbz) dir­ect­ory (if they can be shared over vari­ous fla­vours), and they are un­ver­sioned so that I do not have to up­date the ver­sion num­ber each time there is an up­date. The pack­ages I in­stall by de­fault are bsdstats, portaudit, port­mas­ter, zsh, tmux and all their de­pend­en­cies.

In case you use jails to vir­tu­al­ize ser­vices and con­sol­id­ate serv­ers (e.g. DNS, HTTP, MySQL each in a sep­ar­ate jail) in­stead of provid­ing light­weight vir­tual ma­chines to (un­known) cus­tom­ers, there is also a be­ne­fit of shar­ing the dist­files and pack­ages between jails on the same ma­chine. To do this I cre­ate /data/jails/flavours/mydef/shared/ports/{distfiles,packages} which are then moun­ted via nullfs or NFS into all the jails from a com­mon dir­ect­ory. This re­quires the fol­low­ing vari­ables in /data/jails/flavours/mydef/etc/make.conf (I also keep the pack­ages for dif­fer­ent CPU types and com­pilers in the same sub­tree, if you do not care, just re­move the “/${CC}/${CPUTYPE}” from the PACAKGES line):

DISTDIR=  /​shared/​ports/​distfiles
PACKAGES= /shared/ports/packages/${CC}/${CPUTYPE}

New jails

A fu­ture post will cover how I setup new jails in such a setup and how I cus­tom­ize the start or­der of jails or use some non-​default set­tings for the jail-​startup.

StumbleUponXINGBalatarinBox.netDiggGoogle GmailNetvouzPlurkSiteJotTypePad PostYahoo BookmarksVKSlashdotPocketHacker NewsDiigoBuddyMarksRedditLinkedInBibSonomyBufferEmailHatenaLiveJournalNewsVinePrintViadeoYahoo MailAIMBitty BrowserCare2 NewsEvernoteMail.RuPrintFriendlyWaneloYahoo MessengerYoolinkWebnewsStumpediaProtopage BookmarksOdnoklassnikiMendeleyInstapaperFarkCiteULikeBlinklistAOL MailTwitterGoogle+PinterestTumblrAmazon Wish ListBlogMarksDZoneDeliciousFlipboardFolkdJamespotMeneameMixiOknotiziePushaSvejoSymbaloo FeedsWhatsAppYouMobdiHITTWordPressRediff MyPageOutlook.comMySpaceDesign FloatBlogger PostApp.netDiary.RuKindle ItNUjijSegnaloTuentiWykopTwiddlaSina WeiboPinboardNetlogLineGoogle BookmarksDiasporaBookmarks.frBaiduFacebookGoogle ClassroomKakaoQzoneSMSTelegramRenrenKnownYummlyShare/​Save

Leave a Reply

Your email address will not be published. Required fields are marked *