I just updated to a recent -cur­rent and tried the new nullfs. Sock­ets (e.g. the MySQL one) work now with nullfs. No need to have e.g. jails on the same FS and hardlink the socket to not need to use TCP in MySQL (or an IP at all for the jail).

Great work!

Tags: current, ip, jails, mysql, sockets, tcp —

Just before christ­mas I decided I will spend the “immense” amount of 40 EUR for a graphic card for a sys­tem which was with­out one. The sys­tem is sup­posed to replace my dying home-server. I already moved every­thing, except my Desktop-in-a-Jail (actu­ally it is my home-cinema-jail).

The old sys­tem had a Radeon 9200SE, and it was enough for what I used it for. Now… for a few bucks you can get a lot more horse­power today. After look­ing around a lit­tle bit I decided to buy a NVidia card. I made this deci­sion because it looks like I can get bet­ter dri­ver sup­port for it. So I got a GeForce GT 520 with 1 GB of RAM (I doubt I will be able to use that much RAM) and with­out a fan.

With the Radeon 9200SE I was not able to get the 3D stuff acti­vated (at least in the jail, I did not try with­out), Xorg com­plains about a miss­ing agp­gart mod­ule but I have AGP in the ker­nel (no /dev/agpgart out­side the jail). I did not spend time to inves­ti­gate this, as the main pur­pose — play­ing movies — worked. Now with the NVidia card I decided to give the 3D part a try again.

After adding the NVidia device entries to the jail, and a lit­tle bit of fight­ing with the Xorg-HAL inter­ac­tion, I got a work­ing desk­top. The biggest prob­lem to ver­ify that 3D is work­ing was, that I did not had xdri­info installed. After installing it, I noticed that it does not work with the NVidia dri­ver.    Next stop nvidia-settings: runs great, dis­plays a nice FreeBSD+NVidia logo, and … tells me that OpenGL is con­fig­ured. Hmmm… OK, but I want to see it!

As I decided to switch from Gnome to KDE 4 at  the same time (I was using KDE when it was at V 0.x, switched to Gnome as it looked nicer to me, and now I switch back after read­ing all the stuff in the net that KDE 4 is “bet­ter” than Gnome 3), I was a lit­tle bit out of knowl­edge how to see the 3D stuff in action. So I quickly went to the set­tings and searched for some­thing which looks like it may use 3D. To my sur­prise, it was already using 3D stuff. Nice. I fully real­ized how nice, when play­ing a video and using Alt-Tab to switch win­dows: the video was play­ing full speed scaled down in the window-switcher-thumbnail-view.

That was too easy. I am happy about it.

Now that I have a work­ing setup of X11-in-a-jail for Radeon and GeForce cards, I want to cleanup my changes to the ker­nel and the con­fig files (devfs.rules) and have a look to get this com­mit­ted. A big part of this work is prob­a­bly writ­ing doc­u­men­ta­tion (most prob­a­bly in the wiki).

I still want to see some fancy 3D stuff now. I tried to install x11-clocks/glclock, but the build fails with an unde­fined ref­er­ence to ‘glPoly­gonOff­se­tEXT’. Any rec­om­men­da­tion for a fancy 3D dis­play? My pri­or­ity is on “fancy/nice” with as less vio­lence as pos­si­ble. Most prob­a­bly I will look at it once and then dein­stall it again, so it should be avail­able in the Ports Col­lec­tion (or included in KDE 4).

Tags: agpgart, driver nvidia, gnome, graphic card, home cinema, home server, kde 4, nvidia card, nvidia driver, radeon 9200se —

Every­one has his own way of set­ting up a machine to serve as a host of mul­ti­ple jails. Here is my way, YMMV.

Ini­tial FreeBSD install

I use sev­eral hard­disks in a Soft­ware–RAID setup. It does not mat­ter much if you set them up with one big par­ti­tion or with sev­eral par­ti­tions, feel free to fol­low your pref­er­ences here. My way of par­ti­tion­ing the hard­disks is described in a pre­vi­ous post. That post only shows the com­mands to split the hard­disks into two par­ti­tions and use ZFS for the rootfs. The com­mands to ini­tial­ize the ZFS data par­ti­tion are not described, but you should be able to fig­ure it out your­self (and you can decide on your own what kind of RAID level you want to use). For this FS I set atime, exec and setuid to off in the ZFS options.

On the ZFS data par­ti­tion I cre­ate a new dataset for the sys­tem. For this dataset I set atime, exec and setuid to off in the ZFS options. Inside this dataset I cre­ate datasets for /home, /usr/compat, /usr/local, /usr/obj, /usr/ports/, /usr/src, /usr/sup and /var/ports. There are two ways of doing this. One way is to set the ZFS mount­point. The way I pre­fer is to set rel­a­tive sym­links to it, e.g. “cd /usr; ln –s ../data/system/usr_obj obj”. I do this because this way I can tem­po­rary import the pool on another machine (e.g. my desk­top, if the need arises) with­out fear to inter­fere with the sys­tem. The ZFS options are set as follows:

ZFS options for data/system/*

Dataset	Option	Value
data/system/home	exec	on
data/system/usr_compat	exec	on
data/system/usr_compat	setuid	on
data/system/usr_local	exec	on
data/system/usr_local	setuid	on
data/system/usr_obj	exec	on
data/system/usr_ports	exec	on
data/system/usr_ports	setuid	on
data/system/usr_src	exec	on
data/system/usr_sup	sec­ondarycache	none
data/system/var_ports	exec	on

The exec option for home is not nec­es­sary if you keep sep­a­rate datasets for each user. Nor­mally I keep sep­a­rate datasets for home direc­to­ries, but Jail-Hosts should not have users (except the admins, but they should not keep data in their homes), so I just cre­ate a sin­gle home dataset. The setuid option for the usr_ports should not be nec­es­sary if you redi­rect the build direc­tory of the ports to a dif­fer­ent place (WRKDIRPREFIX in /etc/make.conf).

Installing ports

The ports I install by default are net/rsync, ports-mgmt/portaudit, ports-mgmt/portmaster, shells/zsh, sysutils/bsdstats, sysutils/ezjail, sysutils/smartmontools and sysutils/tmux.

Basic setup

In the crontab of root I setup a job to do a port­snap update once a day (I pick a ran­dom num­ber between 0 and 59 for the minute, but keep a fixed hour). I also have http_proxy spec­i­fied in /etc/profile, so that all machines in this net­work do not down­load every­thing from far away again and again, but can get the data from the local caching proxy. As a lit­tle watch­dog I have a lit­tle @reboot rule in the crontab, which noti­fies me when a machine reboots:

@reboot grep "kernel boot file is" /var/log/messages | mail -s "`hostname` rebooted" root >/dev/null 2>&1

This does not replace a real mon­i­tor­ing solu­tion, but in cases where real mon­i­tor­ing is overkill it pro­vides a nice HEADS-UP (and shows you directly which ker­nel is loaded in case a non-default one is used).

Some default aliases I use every­where are:

alias portmlist="portmaster -L | egrep -B1 '(ew|ort) version|Aborting|installed|dependencies|IGNORE|marked|Reason:|MOVED|deleted|exist|update' | grep -v '^--'"
alias portmclean="portmaster -t --clean-distfiles --clean-packages"
alias portmcheck="portmaster -y --check-depends"

Addi­tional devfs rules for Jails

I have the need to give access to some spe­cific devices in some jails. For this I need to setup a cus­tom /etc/devfs.rules file. The files con­tains some ID num­bers which need to be unique in the sys­tem. On a 9–cur­rent sys­tem the num­bers one to four are already used (see /etc/defaults/devfs.rules). The next avail­able num­ber is obvi­ously five then. First I present my devfs.rules entries, then I explain them:

[devfsrules_unhide_audio=5]
add path 'audio*' unhide
add path 'dsp*' unhide
add path midistat unhide
add path 'mixer*' unhide
add path 'music*' unhide
add path 'sequencer*' unhide
add path sndstat unhide
add path speaker unhide

[devfsrules_unhide_printers=6]
add path 'lpt*' unhide
add path 'ulpt*' unhide user 193 group 193
add path 'unlpt*' unhide user 193 group 193

[devfsrules_unhide_zfs=7]
add path zfs unhide

[devfsrules_jail_printserver=8]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_printers
add include $devfsrules_unhide_zfs

[devfsrules_jail_withzfs=9]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_zfs

The devfs_rules_unhide_XXX ones give access to spe­cific devices, e.g. all the sound related devices or to local print­ers. The devfsrules_jail_XXX ones com­bine all the unhide rules for spe­cific jail setups. Unfor­tu­nately the include direc­tive is not recur­sive, so that we can not include the default devfsrules_jail pro­file and need to repli­cate its con­tents. The first three includes of each devfsrules_jail_XXX accom­plish this. The unhide_zfs rule gives access to /dev/zfs, which is needed if you attach one or more ZFS datasets to a jail. I will explain how to use those pro­files with ezjail in a follow-up post.

Jails setup

I use ezjail to man­age jails, it is more com­fort­able than doing it by hand while at the same time allows me to do some­thing by hand. My jails nor­mally reside inside ZFS datasets, for this rea­son I have setup a spe­cial area (ZFS dataset data/jails) which is han­dled by ezjail.The cor­re­spond­ing ezjail.conf set­tings are:

ezjail_jaildir=/data/jails
ezjail_use_zfs="YES"
ezjail_jailzfs="data/jails"

I also dis­abled procfs and fde­scfs in jails (but they can be enabled later for spe­cific jails if necessary).

Unfor­tu­nately ezjail (as of v3.1) sets the mount­point of a newly cre­ated dataset even if it is not nec­es­sary. For this rea­son I always issue a “zfs inherit mount­point ” after cre­at­ing a jail. This sim­pli­fies the case where you want to move/rename a dataset and want to have the mount­point autom­cat­i­cally fol­low the change.

The access flags of  /data/jails direc­tory are 700, this pre­vents local users (there should be none, but bet­ter safe than sorry) to get access to files from users in jails with the same UID.

After the first create/update of the ezjail base­jail the ZFS options of base­jail (data/jails/basejail) and new­jail (data/jails/newjail) need to be changed. For both exec and setuid should be changed to “on” The same needs to be done after cre­at­ing a new jail for the new jail (before start­ing it).

The default ezjail flavour

In my default ezjail flavour I cre­ate some default user(s) with a basesystem-shell (via /data/jails/flavours/mydef/ezjail.flavour) before the pack­age install, and change the shell to my pre­ferred zsh after­wards (this is only valid if the jails are used only by in-house peo­ple, if you want to offer light­weight vir­tual machines to (unknown) cus­tomers, the default user(s) and shell(s) are obvi­ously up to dis­cus­sion). At the end I also run a “/usr/local/sbin/portmaster –y –check-depends” to make sure every­thing is in a sane state.

For the pack­ages (/data/jails/flavours/mydef/pkg/) I add sym­links to the unver­sioned pack­ages I want to install. I have the pack­ages in a com­mon (think about set­ting PACKAGES in make.conf and using PACKAGES/Latest/XYZ.tbz) direc­tory (if they can be shared over var­i­ous flavours), and they are unver­sioned so that I do not have to update the ver­sion num­ber each time there is an update. The pack­ages I install by default are bsd­stats, por­tau­dit, port­mas­ter, zsh, tmux and all their dependencies.

In case you use jails to vir­tu­al­ize ser­vices and con­sol­i­date servers (e.g. DNS, HTTP, MySQL each in a sep­a­rate jail) instead of pro­vid­ing light­weight vir­tual machines to (unknown) cus­tomers, there is also a ben­e­fit of shar­ing the dis­t­files and pack­ages between jails on the same machine. To do this I cre­ate /data/jails/flavours/mydef/shared/ports/{distfiles,packages} which are then mounted via nullfs or NFS into all the jails from a com­mon direc­tory. This requires the fol­low­ing vari­ables in /data/jails/flavours/mydef/etc/make.conf (I also keep the pack­ages for dif­fer­ent CPU types and com­pil­ers in the same sub­tree, if you do not care, just remove the “/${CC}/${CPUTYPE}” from the PACAKGES line):

DISTDIR=  /shared/ports/distfiles
PACKAGES= /shared/ports/packages/${CC}/${CPUTYPE}

New jails

A future post will cover how I setup new jails in such a setup and how I cus­tomize the start order of jails or use some non–default set­tings for the jail-startup.

Tags: dataset, harddisks, jails, option value, raid level, setuid, software raid, symlinks, temporary import, zfs —

In the last days I migrated all my inter­nal ser­vices to IPv6.

All my jails have an IPv4 and an IPv6 address now. All Apaches (I have one for my pic­ture gallery, one for web­mail, and one for inter­nal man­age­ment) now lis­ten on the inter­nal IPv6 address too. Squid is updated from 2.x to 3.1 (the most recent ver­sion in the Ports Col­lec­tion) and I added some IPv6 ACLs. The inter­nal Post­fix is con­fig­ured to han­dle IPv6 too (it is deliv­er­ing every­thing via an authen­ti­cated and encrypted chan­nel to a machine with a sta­tic IPv4 address for final deliv­ery). My MySQL does not need an IPv6 address, as it is only lis­ten­ing to requests via IPC (the socket is hardlinked between jails). All ssh dae­mons are con­fig­ured to lis­ten to IPv6 too. The IMAP and CUPS server was pick­ing the new IPv6 addresses auto­mat­i­cally. I also updated Samba to han­dle IPv6, but due to lack of a Win­dows machine which prefers IPv6 over IPv4 for CIFS access (at least I think my Win­dows XP net­book only tries IPv4 con­nec­tions) I can not really test this.

Only my Wii is a lit­tle bit behind, and I have not checked if my Sony-TV will DTRT (but for this I first have to get some time to have a look if I have to update my DD-WRT firmware on the lit­tle WLAN-router which is “extend­ing the cable” from the TV to the inter­nal net­work, and I have to look how to con­fig­ure IPv6 with DD-WRT).

Tags: dd wrt firmware, internal management, ipv4 address, ipv6 address, ipv6 addresses, ipv6 ipv4, netbook, sony tv, wii, wlan router —

After enabling IPv6 in my WLAN router, I also enabled IPv6 in my FreeBSD sys­tems. I have to tell that the IPv6 chap­ter in the FreeBSD hand­book does not con­tain as much infor­ma­tion as I would like to have about this.

Con­fig­ur­ing the inter­faces of my two 9–cur­rent sys­tems to also carry a spe­cific IPv6 address (an easy one from the ULA I use) was easy after read­ing the man-page for rc.conf. After a lit­tle bit of exper­i­ment­ing it came down to:

ifconfig_rl0_ipv6=“inet6 ::2:1 pre­fixlen 64 accept_rtadv“
ipv6_defaultrouter=”<router address>”

Apart from this address (I chose it because the IPv4 address ends in “.2″, this way I can add some easy to remem­ber addresses for this machine if needed), I also have two auto­mat­i­cally con­fig­ured addresses. One is with the same ULA and some not so easy to remem­ber end (con­structed from the MAC address), and one is from the offi­cial pre­fix the router con­structed out of the offi­cial IPv4 address from the ISP (+ the same end than the other end).

Addi­tion­ally I also have all my jails on this machine with an IPv6 address now (yes, they are like “…:2:100″ with the :100 because the IPv4 address ends in “.100″). Still TODO is the con­ver­sion of all the ser­vices in the jails to also lis­ten on the IPv6 address.

I already changed the con­fig of my inter­nal DNS to have the IPv6 addresses for all sys­tems, lis­ten on the IPv6 address (when I add an IPv6 net­work to allow-query/allow-query-cache/allow-recursion bind does not want to start). And as I was there, I also enabled the DNSSEC ver­i­fi­ca­tion (but I get a lot of error mes­sages in the logs: “unable to con­vert errno to isc_result: 42: Pro­to­col not avail­able”, one search result which talks exactly about this error tells it is a “cos­metic error”…).

I noticed that an IPv6 ping between two phys­i­cal machines takes a lit­tle bit more time than an IPv4 ping (no IPsec enabled). It sur­prised me that this is such a notice­able dif­fer­ence (not within the std-dev at all):

— m87.Leidinger.net ping sta­tis­tics —
10 pack­ets trans­mit­ted, 10 pack­ets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.168÷0.193÷0.220÷0.017 ms

— m87.Leidinger.net ping6 sta­tis­tics —
10 pack­ets trans­mit­ted, 10 pack­ets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.207÷0.325÷0.370÷0.047 ms

The infor­ma­tion I miss in the FreeBSD hand­book in the IPv6 chap­ter is what those other IPv6 related ser­vices are and when/how to con­fig­ure them. I have an idea now what this radvd is, but I am not sure what the inter­ac­tion is with the accept_rtadv set­ting for ifcon­fig (and I do not think I need it, as my WLAN router seems to do it already). I know that I get the IPv6-friendly net­work neigh­bor­hood dis­played with ndp(8). I did not have a look at enabling IPv6 mul­ti­cast sup­port in FreeBSD, and I do not know what those other IPv6 options for rc.conf do.

Tags: address ends, cosmetic error, current systems, freebsd handbook, freebsd systems, ipv4 address, ipv6 address, ipv6 addresses, ipv6 network, wlan router —