Sol­ar­is 10/11(.3) boot panic/​crash after mov­ing rpool to a new stor­age sys­tem

Situ­ation

The boot disks of some Sol­ar­is LDOMs were mi­grated from one stor­age sys­tem to an­oth­er one via ZFS mir­ror­ing the rpool to the new sys­tem and de­tach­ing the old LUN.

Is­sue

After re­boot with on the new stor­age sys­tem Sol­ar­is 10 and 11(.3) pan­ic at boot.

Cause

  • rpool not on slice 0 but on slice 2
  • bug in Sol­ar­is when do­ing such a mir­ror and “just” do­ing a re­boot <- this is the real is­sue, it seems Sol­ar­is can not handle a change of the name of the un­der­ly­ing device for a rpool, as just mov­ing the par­ti­tion­ing to slice 0 is not fix­ing the pan­ic.

Fix

# boot from net­work (or an al­tern­ate pool which was not yet moved), import/​export the pools, boot from the pools
boot net -
# go to shell
# if needed: change the par­ti­tion­ing so that slice 0 has the same val­ues as slice 2 (re­spect­ively make sure the rpool is in slice 0)
zpool im­port -R /​tmp/​yyy rpool
zpool ex­port rpool
re­boot

 

ioc­age: HOWTO cre­ate a base­jail from src (in­stead of from an of­fi­cial re­lease)

Back­ground

So far I have used ez­jail to man­age FreeBSD jails. I use jails since years to have dif­fer­ent parts of a soft­ware stack in some kind of a con­tain­er (in a ZFS data­set for the filesys­tem side of the con­tain­er). On one hand to not let de­pend­en­cies of one part of the soft­ware stack have in­flu­ence of oth­er parts of the soft­ware stack. On the oth­er hand to have the pos­sib­il­ity to move parts of the soft­ware stack to a dif­fer­ent sys­tem if ne­ces­sary. Nor­mally I run -stable or -cur­rent or more gen­er­ally speak­ing, a self-​compiled FreeBSD on those sys­tems. In ez­jail I like the fact that all jails on a sys­tem have one com­mon base­jail un­der­ly­ing, so that I up­date one place for the user­land and all jails get the up­dated code.

Since a while I heard good things about ioc­age and how it in­teg­rates ZFS, so I de­cided to give it a try my­self. As ioc­age does not come with an of­fi­cial way of cre­at­ing a base­jail (re­spect­ively a re­lease) from a self-​compiled FreeBSD (at least doc­u­mented in those places I looked, and yes, I am aware that I can cre­ate a FreeBSD re­lease my­self and use it, but I do not like to have to cre­ate a re­lease ad­di­tion­ally to the build­world I use to up­date the host sys­tem) here now the short HOWTO achieve this.

In­vari­ants

In the fol­low­ing I as­sume the ioc­age ZFS parts are already cre­ated in data­set ${POOLNAME}/iocage which is moun­ted on ${IOCAGE_BASE}/iocage. Ad­di­tion­ally the build­world in /​usr/​src (or wherever you have the FreeBSD source) should be fin­ished.

Pre-​requisites

To have the ne­ces­sary dataset-​infrastructure cre­ated for own basejails/​releases, at least one of­fi­cial re­lease needs to be fetched be­fore. So run the com­mand be­low (if there is no ${IOCAGE_BASE}/iocage/releases dir­ect­ory) and fol­low the on-​screen in­struc­tions.

ioc­age fetch

HOWTO

Some vari­ables:

POOLNAME=mpool
SRC_REV=r$(cd /​usr/​src; svn­litever­sion)
IOCAGE_​BASE=””

Cre­at­ing the ioc­age basejail-​datasets for this ${SRC_​REV}:

zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/bin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/boot
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/lib
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/libexec
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/rescue
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/sbin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/include
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib32
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libdata
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libexec
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/share
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/src

In­stall from /​usr/​src (the ex­ecut­able “chown” is hard­linked across an ioc­age base­jail data­set bound­ary, this fails in the nor­mal in­stall­world, so we have to ig­nore this er­ror and in­stall a copy of the chown bin­ary to the place where the hard­link nor­mally is):

cd /​usr/​src
make -i in­stall­world DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >&! iocage_installworld_base.log
cp -pv ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin/chown ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin/chgrp
make dis­tri­bu­tion DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root »& iocage_installworld_base.log

While we are here, also cre­ate a re­lease and not only a base­jail:

zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE/root
make in­stall­world DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >&! iocage_installworld_release.log
make dis­tri­bu­tion DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root »& iocage_installworld_release.log

And fi­nally make this the de­fault re­lease which ioc­age uses when cre­at­ing new jails (this is op­tion­al):

ioc­age set release=${SRC_REV}-RELEASE de­fault

Now the self-​build FreeBSD is avail­able in ioc­age for new jails.

HOWTO: “Blind” re­mote in­stall of FreeBSD via tiny disk im­age (ZFS edi­tion)

In a past post I de­scribed how to in­stall a FreeBSD re­motely via a tiny UFS based disk im­age over a linux sys­tem. In this post I de­scribe how to do it with a ZFS based disk im­age.

In­vari­ants

Giv­en a unix based re­mote sys­tem (in this case a linux sys­tem) from which you know what kind of hard­ware it runs on (e.g. PCI IDs) and what the cor­res­pond­ing FreeBSD drivers are.

HOWTO

In the title of this post I wrote “via a tiny disk im­age”. This is true for a suit­able defin­i­tion of tiny.

What we have in the root­server are two ~900 GB hard­disks. They shall be used in a soft­ware mir­ror. The ma­chine has 8 GB of RAM. I do not ex­pect much ker­nel pan­ics (= crash dumps) there, so we do not really need >8 GB of swap (for­get the rule of hav­ing twice as much swap than RAM, with the cur­rent amount of RAM in a ma­chine you are in “trouble” when you need even the same amount of swap than RAM). I de­cided to go with 2 GB of swap.

Pushing/​pulling a 900 GB im­age over the net­work to in­stall a sys­tem is not really some­thing I want to do. I am OK to trans­fer 5 GB (that is 0.5% of the en­tire disk) to get this job done, and this is feas­ible.

First let us define some vari­ables in the shell, this way you just need to change the val­ues in one place and the rest is copy&paste (I use the SVN re­vi­sion of the source which I use to in­stall the sys­tem as the name of the sysutils/​beadm com­pat­ible boot-​dataset in the root­fs, as such I also have the re­vi­sion num­ber avail­able in a vari­able):

ROOTFS_SIZE=5G
ROOTFS_NAME=root
FILENAME=rootfs
POOLNAME=mpool
VERSION=r$(cd /​usr/​src; svn­litever­sion)
SWAPSIZE=2G

Then change your cur­rent dir­ect­ory to a place where you have enough space for the im­age. There we will cre­ate a con­tainer for the im­age, and make it ready for par­ti­tion­ing:

trun­cate -s ${ROOTFS_​SIZE} ${FILENAME}
md­con­fig -a -t vnode -f ${FILENAME}
# if you want to fully al­loc­ate
# dd if=/dev/zero of=/dev/md0 bs=1m

Cre­ate the par­ti­tion table and the root­fs (in a sysutils/​beadm com­pat­ible way – as I in­stall FreeBSD-​current there – and mount it tem­por­ary to /​temppool):

gpart cre­ate -s GPT /​dev/​md0
gpart add -s 512K -t freebsd-​boot -l bootcode0 /​dev/​md0
gpart add -a 4k -t freebsd-​swap -s ${SWAPSIZE} -l swap0 /​dev/​md0
gpart add -a 1m -t freebsd-​zfs -l ${POOLNAME}0 /​dev/​md0
gpart boot­code -b /​boot/​pmbr -p /​boot/​gptzfsboot -i 1 /​dev/​md0
# if not already the case and you want to have 4k phys­ic­al sec­tor size of the pool
# sy­scl vfs.zfs.min_auto_ashift=12
zpool cre­ate -o cachefile=/boot/zfs/zpool.cache_temp -o altroot=/temppool -O compress=lz4 -O atime=off -O utf8only=on ${POOLNAME} /dev/gpt/${POOLNAME}0
zfs cre­ate -o mountpoint=none ${POOLNAME}/ROOT
zfs cre­ate -o mountpoint=/ ${POOLNAME}/ROOT/${VERSION}
zfs cre­ate -o mountpoint=/tmp -o exec=on -o setuid=off ${POOLNAME}/tmp
zfs cre­ate -o mountpoint=/usr -o canmount=off ${POOLNAME}/usr
zfs cre­ate -o mountpoint=/home ${POOLNAME}/home
zfs cre­ate -o setuid=off ${POOLNAME}/usr/ports
zfs cre­ate ${POOLNAME}/usr/src
zfs cre­ate -o mountpoint=/var -o canmount=off ${POOLNAME}/var
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/audit
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/crash
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/log
zfs cre­ate -o atime=on ${POOLNAME}/var/mail
zfs cre­ate -o setuid=off ${POOLNAME}/var/tmp
zfs cre­ate ${POOLNAME}/var/ports
zfs cre­ate -o exec=off -o setuid=off -o mountpoint=/shared ${POOLNAME}/shared
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/shared/distfiles
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/shared/packages
zfs cre­ate -o exec=off -o setuid=off -o compression=lz4 ${POOLNAME}/shared/ccache
zfs cre­ate ${POOLNAME}/usr/obj
zpool set bootfs=${POOLNAME}/ROOT/${VERSION} ${POOLNAME}

In­stall FreeBSD (from source):

cd /​usr/​src
#make build­world >&! buildworld.log
#make buildker­nel -j 8 KERNCONF=GENERIC >&! buildkernel_generic.log
make in­stall­world DESTDIR=/temppool/ >& installworld.log
make dis­tri­bu­tion DESTDIR=/temppool/ >& distrib.log
make in­stallker­nel KERNCONF=GENERIC DESTDIR=/temppool/ >& installkernel.log

Copy the tem­por­ary zpool cache cre­ated above in the pool-​creation part to the im­age (I have the im­pres­sion it is not really needed and will work without, but I have not tried this):

cp /boot/zfs/zpool.cache_temp /​temppool/​boot/​
cp /boot/zfs/zpool.cache_temp /temppool/boot/zpool.cache

Add the zfs mod­ule to loader.conf:

zfs_load=“yes”
opensolaris_load=“yes”

Now you need to cre­ate /temppool/etc/rc.conf (set the de­faultrouter, the IP ad­dress via ifconfig_​IF (and do not for­get to use the right IF for it), the host­name, set sshd_​enable to yes, zfs_enable=“YES”)  /temppool/boot/loader.conf (zfs_load=“yes”, opensolaris_load=“yes”, vfs.root.mountfrom=“zfs:${POOLNAME}/ROOT/r${VERSION}”)
/​temppool/​etc/​hosts, /temppool/etc/resolv.conf and maybe /temppool/etc/sysctl.conf and /temppool/etc/periodic.conf.

Do not al­low password-​less root lo­gins in single-​user mode on the phys­ic­al con­sole, cre­ate a resolv.conf and an user:

cd /​temppool/​etc
sed -ie „s:console.*off.:&in:“ ttys
cat >resolv.conf «EOT
search YOURDOMAIN
nameserv­er 8.8.8.8
EOT
pw -V /​temppool/​etc groupadd YOURGROUP -g 1001
pw -V /​temppool/​etc useradd YOURUSER -u 1001 -d /​home/​YOURUSER -g YOURUSER -G wheel -s /​bin/​tcsh
pw -V /​temppool/​etc user­mod YOURUSER -h 0 pw -V /​temppool/​etc user­mod root -h 0
zfs cre­ate mpool/​home/​YOURUSER
chown YOURUSER:YOURGROUP /​temppool/​home/​YOURUSER

Now you can make some more modi­fic­a­tions to the sys­tem if wanted, and then ex­port the pool and de­tach the im­age:

zpool ex­port ${POOLNAME}

md­con­fig -d -u 0

De­pend­ing on the up­load speed you can achieve, it is be­ne­fi­cial to com­press the im­age now, e.g. with bzip2. Then trans­fer the im­age to the disk of the re­mote sys­tem. In my case I did this via:

ssh –C –o CompressionLevel=9 root@remote_host dd of=/dev/hda bs=1m < /path/to/${FILENAME}

Then reboot/​power-​cycle the re­mote sys­tem.

Post-​install tasks

Now we have a new FreeBSD sys­tem which uses only a frac­tion of the the hard­disk and is not re­si­li­ent against harddisk-​failures.

FreeBSD will de­tect that the disk is big­ger than the im­age we used when cre­at­ing the GPT la­bel and warn about it (cor­rupt GPT table). To fix this and to res­ize the par­ti­tion for the zpool to use the en­tire disk we first mir­ror the zpool to the second disk and res­ize the par­ti­tion of the first disk, and when the zpool is in-​sync and then we res­ize the boot disk (at­ten­tion, you need to change the “-s” part in the fol­low­ing to match your disk size).

First backup the la­bel of the first disk, this makes it more easy to cre­ate the la­bel of the second disk:

/​sbin/​gpart backup ada0 > ada0.gpart

Edit ada0.gpart (give dif­fer­ent names for the la­bels, mainly change the num­ber 0 on the label-​name to 1) and then use it to cre­ate the par­ti­tion of the second disk:

gpart re­store -Fl ada1 < ada0.gpart
gpart res­ize -i 3 -a 4k -s 929g ada1
gpart boot­code -b /​boot/​pmbr -p /​boot/​gptzfsboot -i 1 ada1
zpool set autoexpand=on mpool

Fix the warn­ing about the GPT la­bel and res­ize the par­ti­tion:

gpart re­cov­er ada0
gpart res­ize -i 3 -a 4k -s 929g ada0

Af­ter­wards it should look sim­il­ar to this:

gpart show -l
=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  bootcode0  (512K)
        1064     4194304     2  swap0  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool0  (929G)
  1952450560     1074568        – free –  (525M)

=>        40  1953525088  ada1  GPT  (932G)
          40        1024     1  bootcode1  (512K)
        1064     4194304     2  swap1  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool1  (929G)
  1952450560     1074568        – free –  (525M)

Add the second disk to the zpool:

zpool at­tach mpool gpt/​mpool0 gpt/​mpool1

When the mir­ror is in sync (zpool status mpool), we can ex­tend the size of the pool it­self:

zpool off­line mpool /​dev/​gpt/​mpool0
zpool on­line mpool /​dev/​gpt/​mpool0

As a last step we can add now an en­cryp­ted swap (de­pend­ing on the im­port­ance of the sys­tem maybe a gmirror-​ed one – not ex­plained here), and spe­cify where to dump (text-​dumps) on.

/boot/loader.conf:

dumpdev=“/dev/ada0p2”

/etc/rc.conf:

dumpdev=“/dev/gpt/swap0”
crashinfo_enable=“YES”
ddb_enable=“yes”
encswap_enable=“YES”
geli_swap_flags=”-a hmac/​sha256 -l 256 -s 4096 -d”

/​etc/​fstab:

# Device        Moun­t­point      FStype  Op­tions                 Dump    Pass#
/dev/ada1p2.eli none    swap    sw      0 0

Now the sys­tem is ready for some ap­plic­a­tions.

Trans­ition to nginx: part 4 – CGI scripts

I still have some CGI scripts on this web­site. They still work, and they are good enough for my needs. When I switched this web­site to nginx (the Word­Press setup was a little bit more com­plex than what I wrote in part 1, part 2 and part 3… the con­fig will be one of my next blog posts) I was a little bit puzzled how to do that with nginx. It took me some minutes to get an idea how to do it and to find the right FreeBSD port for this.

  • In­stall www/​fcgiwrap
  • Add the fol­low­ing to rc.conf:

fcgiwrap_enable=“YES”
fcgiwrap_user=“www”

  • Run “ser­vice fc­gi­wrap start”
  • Add the fol­low­ing to your nginx con­fig:
loc­a­tion ^~ /​cgi-​bin/​ {
    gzip off; #gzip makes scripts feel slower since they have to com­plete be­fore get­ting gzipped
    fastcgi_​pass  unix:/var/run/fcgiwrap/fcgiwrap.sock;
    fastcgi_​index index.cgi;
    fastcgi_​param SCRIPT_​FILENAME /path/to/location$fastcgi_script_name;
    fastcgi_​param GATEWAY_​INTERFACE  CGI/1.1;
}

Tran­si­tion to nginx: part 3 — short and easy con­fig snip­pets

After some medium-​difficoulty trans­itions in part 1 and part 2, here some easy ones:

phpMy­Ad­min: take the ba­sics from one of the two oth­er blog posts (see above) without loc­a­tion dir­ect­ives. For “loc­a­tion /​” set the doc­u­ment root and copy the “loc­a­tion ~ \.php” from the con­fig of one of the parts above. Done.

TT-​RSS: take the con­fig like for phpMy­Ad­min and add (as­sum­ing it is in the root of the serv­er, else you have to add the path in the front of the loc­a­tion)

loc­a­tion ^~ /(utils|templates|schema|cache|lock|locale|classes) {
     deny all;
}

Al­low client-​side cach­ing for stat­ic con­tent:

loc­a­tion ~* \.(?:jpe?g|gif|png|ico|cur|gz|bz2|xz|tbz|tgz|txz|svg|svgz|mp4|ogg|ogv|webm|htc|css|js|
pdf|zip|rar|tar|txt|conf)$ {
    try_​files $uri =404;

    ex­pires 1w;     # If you are not a big site,

                    # and don’t change stat­ic con­tent of­ten,

                    # 1 week is not bad.
    access_​log off; # If you don’t need the logs
    add_​header Cache-​Control “pub­lic”;
}

Se­cur­ity: Des­pite the fact that the docs I’ve read tell that no-​SSLv3 is the de­fault, the first set­ting makes a dif­fer­ence (tested via SSLlabs“ SSLtest).

ssl_​protocols TLSv1 TLSv1.1 TLSv1.2; # No SSLv 23
ssl_​dhparam /path/to/dhparams.pem;   # gen­er­ate via “openssl dh­param -out /path/to/dhparams.pem 2048”