HOWTO: “Blind” re­mote in­stall of FreeB­SD via tiny disk im­age (ZFS edition)

In a past post I described how to install a FreeB­SD remote­ly via a tiny UFS based disk image over a lin­ux sys­tem. In this post I describe how to do it with a ZFS based disk image.

Invari­ants

Giv­en a unix based remote sys­tem (in this case a lin­ux sys­tem) from which you know what kind of hard­ware it runs on (e.g. PCI IDs) and what the cor­re­spond­ing FreeB­SD dri­vers are.

HOWTO

In the title of this post I wrote “via a tiny disk im­age”. This is true for a suit­able defin­i­tion of tiny.

What we have in the root­server are two ~900 GB hard­disks. They shall be used in a soft­ware mir­ror. The ma­chine has 8 GB of RAM. I do not ex­pect much ker­nel pan­ics (= crash dumps) there, so we do not real­ly need >8 GB of swap (for­get the rule of hav­ing twice as much swap than RAM, with the cur­rent amount of RAM in a ma­chine you are in “trou­ble” when you need even the same amount of swap than RAM). I de­cided to go with 2 GB of swap.

Pushing/pulling a 900 GB im­age over the net­work to in­stall a sys­tem is not real­ly some­thing I want to do. I am OK to trans­fer 5 GB (that is 0.5% of the en­tire disk) to get this job done, and this is feasible.

First let us define some vari­ables in the shell, this way you just need to change the val­ues in one place and the rest is copy&paste (I use the SVN revi­sion of the source which I use to install the sys­tem as the name of the sysutils/beadm com­pat­i­ble boot-dataset in the rootfs, as such I also have the revi­sion num­ber avail­able in a variable):

ROOTFS_SIZE=5G
ROOTFS_NAME=root
FILENAME=rootfs
POOLNAME=mpool
VERSION=r$(cd /usr/src; svnliteversion)
SWAPSIZE=2G

Then change your cur­rent dir­ect­ory to a place where you have enough space for the im­age. There we will cre­ate a con­tainer for the im­age, and make it ready for partitioning:

truncate -s ${ROOTFS_SIZE} ${FILENAME}
mdconfig -a -t vnode -f ${FILENAME}
# if you want to fully allocate
# dd if=/dev/zero of=/dev/md0 bs=1m

Cre­ate the par­ti­tion table and the rootfs (in a sysutils/beadm com­pat­i­ble way – as I install FreeBSD-current there – and mount it tem­po­rary to /temppool):

gpart create -s GPT /dev/md0
gpart add -s 512K -t freebsd-boot -l bootcode0 /dev/md0
gpart add -a 4k -t freebsd-swap -s ${SWAPSIZE} -l swap0 /dev/md0
gpart add -a 1m -t freebsd-zfs -l ${POOLNAME}0 /dev/md0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 /dev/md0
# if not already the case and you want to have 4k physical sector size of the pool
# syscl vfs.zfs.min_auto_ashift=12
zpool create -o cachefile=/boot/zfs/zpool.cache_temp -o altroot=/temppool -O compress=lz4 -O atime=off -O utf8only=on ${POOLNAME} /dev/gpt/${POOLNAME}0
zfs create -o mountpoint=none ${POOLNAME}/ROOT
zfs create -o mountpoint=/ ${POOLNAME}/ROOT/${VERSION}
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off ${POOLNAME}/tmp
zfs create -o mountpoint=/usr -o canmount=off ${POOLNAME}/usr
zfs create -o mountpoint=/home ${POOLNAME}/home
zfs create -o setuid=off ${POOLNAME}/usr/ports
zfs create ${POOLNAME}/usr/src
zfs create -o mountpoint=/var -o canmount=off ${POOLNAME}/var
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/audit
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/crash
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/log
zfs create -o atime=on ${POOLNAME}/var/mail
zfs create -o setuid=off ${POOLNAME}/var/tmp
zfs create ${POOLNAME}/var/ports
zfs create -o exec=off -o setuid=off -o mountpoint=/shared ${POOLNAME}/shared
zfs create -o exec=off -o setuid=off ${POOLNAME}/shared/distfiles
zfs create -o exec=off -o setuid=off ${POOLNAME}/shared/packages
zfs create -o exec=off -o setuid=off -o compression=lz4 ${POOLNAME}/shared/ccache
zfs create ${POOLNAME}/usr/obj
zpool set bootfs=${POOLNAME}/ROOT/${VERSION} ${POOLNAME}

In­stall FreeB­SD (from source):

cd /usr/src
#make buildworld >&! buildworld.log
#make buildkernel -j 8 KERNCONF=GENERIC >&! buildkernel_generic.log
make installworld DESTDIR=/temppool/ >& installworld.log
make distribution DESTDIR=/temppool/ >& distrib.log
make installkernel KERNCONF=GENERIC DESTDIR=/temppool/ >& installkernel.log

Copy the tem­po­rary zpool cache cre­at­ed above in the pool-creation part to the image (I have the impres­sion it is not real­ly need­ed and will work with­out, but I have not tried this):

cp /boot/zfs/zpool.cache_temp /temppool/boot/
cp /boot/zfs/zpool.cache_temp /temppool/boot/zpool.cache

Add the zfs mod­ule to loader.conf:

zfs_load="yes"
opensolaris_load="yes"

Now you need to cre­ate /temppool/etc/rc.conf (set the de­faultrouter, the IP ad­dress via ifconfig_IF (and do not for­get to use the right IF for it), the host­name, set sshd_enable to yes, zfs_enable=“YES”)  /temppool/boot/loader.conf (zfs_load=“yes”, opensolaris_load=“yes”, vfs.root.mountfrom=“zfs:${POOLNAME}/ROOT/r${VERSION}”)
/temppool/etc/hosts, /temppool/etc/resolv.conf and maybe /temppool/etc/sysctl.conf and /temppool/etc/periodic.conf.

Do not allow password-less root logins in single-user mode on the phys­i­cal con­sole, cre­ate a resolv.conf and an user:

cd /temppool/etc
sed -ie 's:console.*off.:&in:' ttys
cat >resolv.conf <<EOT
search YOURDOMAIN
nameserver 8.8.8.8
EOT
pw -V /temppool/etc groupadd YOURGROUP -g 1001
pw -V /temppool/etc useradd YOURUSER -u 1001 -d /home/YOURUSER -g YOURUSER -G wheel -s /bin/tcsh
pw -V /temppool/etc usermod YOURUSER -h 0
pw -V /temppool/etc usermod root -h 0
zfs create mpool/home/YOURUSER
chown YOURUSER:YOURGROUP /temppool/home/YOURUSER

Now you can make some more mod­i­fi­ca­tions to the sys­tem if want­ed, and then export the pool and detach the image:

zpool export ${POOLNAME}

mdconfig -d -u 0

Depend­ing on the upload speed you can achieve, it is ben­e­fi­cial to com­press the image now, e.g. with bzip2. Then trans­fer the image to the disk of the remote sys­tem. In my case I did this via:

ssh –C –o CompressionLevel=9 root@remote_host dd of=/dev/hda bs=1m < /path/to/${FILENAME}

Then reboot/power-cycle the remote system.

Post-install tasks

Now we have a new FreeB­SD sys­tem which uses only a frac­tion of the the hard­disk and is not resilient against harddisk-failures.

FreeB­SD will detect that the disk is big­ger than the image we used when cre­at­ing the GPT label and warn about it (cor­rupt GPT table). To fix this and to resize the par­ti­tion for the zpool to use the entire disk we first mir­ror the zpool to the sec­ond disk and resize the par­ti­tion of the first disk, and when the zpool is in-sync and then we resize the boot disk (atten­tion, you need to change the “-s” part in the fol­low­ing to match your disk size).

First back­up the label of the first disk, this makes it more easy to cre­ate the label of the sec­ond disk:

/sbin/gpart backup ada0 > ada0.gpart

Edit ada0.gpart (give dif­fer­ent names for the labels, main­ly change the num­ber 0 on the label-name to 1) and then use it to cre­ate the par­ti­tion of the sec­ond disk:

gpart restore -Fl ada1 < ada0.gpart
gpart resize -i 3 -a 4k -s 929g ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
zpool set autoexpand=on mpool

Fix the warn­ing about the GPT label and resize the partition:

gpart recover ada0
gpart resize -i 3 -a 4k -s 929g ada0

After­wards it should look sim­i­lar to this:

gpart show -l
=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  bootcode0  (512K)
        1064     4194304     2  swap0  (2.0G)
     4195368         984        - free -  (492K)
     4196352  1948254208     3  mpool0  (929G)
  1952450560     1074568        - free -  (525M)

=>        40  1953525088  ada1  GPT  (932G)
          40        1024     1  bootcode1  (512K)
        1064     4194304     2  swap1  (2.0G)
     4195368         984        - free -  (492K)
     4196352  1948254208     3  mpool1  (929G)
  1952450560     1074568        - free -  (525M)

Add the sec­ond disk to the zpool:

zpool attach mpool gpt/mpool0 gpt/mpool1

When the mir­ror is in sync (zpool sta­tus mpool), we can extend the size of the pool itself:

zpool offline mpool /dev/gpt/mpool0
zpool online mpool /dev/gpt/mpool0

As a last step we can add now an encrypt­ed swap (depend­ing on the impor­tance of the sys­tem maybe a gmirror-ed one – not explained here), and spec­i­fy where to dump (text-dumps) on.

/boot/loader.conf:

dumpdev="/dev/ada0p2"

/etc/rc.conf:

dumpdev="/dev/gpt/swap0"
crashinfo_enable="YES"
ddb_enable="yes"
encswap_enable="YES"
geli_swap_flags="-a hmac/sha256 -l 256 -s 4096 -d"

/etc/fstab:

# Device        Mountpoint      FStype  Options                 Dump    Pass#
/dev/ada1p2.eli none    swap    sw      0       0

Now the sys­tem is ready for some applications.

Gain­ing space on Android after ART->Dalvik switch (root access required)

I (still) use a Nexus S phone. I am using Cyanogen­mod on it. After an arti­cle in a com­put­er mag­a­zine I decid­ed to give the ART-runtime a try instead of the default Dalvic-runtime. Unfor­tu­nate­ly I do not have enough free space free (and all what I can is moved to the USB stor­age already) to real­ly use the ART-runtime.

After switch­ing back to the Dalvic-runtime, I had only 23 of the pre­vi­ous­ly avail­able space free. After a lit­tle bit of look­ing around I found /data/dalvik-cache. I delet­ed with a file man­ag­er the con­tent of the direc­to­ry (you will get some “app crashed/died” mes­sages) and reboot­ed the phone (this is not the same as for­mat­ting the cache par­ti­tion in the recov­ery system).

Dur­ing boot it pop­u­lat­ed the direc­to­ry again and now I have more than 43 of free space on the inter­nal storage.

Updat­ing FreeB­SD 8.2 (or 9.x) to 10 (beta4)

This is a lit­tle descrip­tion how I remote­ly (no con­sole, boot­ed into multi-user dur­ing update, no exter­nal ser­vices like jails/httpd/… run­ning) updat­ed a FreeB­SD 8.2 to 10 (beta4) from source. This should also work when updat­ing from FreeB­SD 9.x. Note, I had already switched to ATA_CAM on 8.2, so not instruc­tions for the name change of the ata devices. No IPv6, WLAN or CARP is in use here, so changes which are need­ed in this area are not cov­ered. Read UPDATING care­ful­ly, there are a lot of changes between major releases.

What I did:

  • update /usr/src
  • make build­world
  • replace “make ” in /usr/src/Makefile.inc1 with ${MAKE} (two times, one for “VERSION”, one for “BRANCH”)
  • ver­i­fy ker­nel con­fig for changes need­ed (run­ning “con­fig MyK­er­nel” in /usr/src/sys/YourArch/conf/ helps to iden­ti­fy syn­tax prob­lems), sor­ry I did­n’t take notes, but I diffed the old and the new GENERIC con­fig and added/removed accord­ing to my interests
  • /usr/obj/…/src/usr.bin/bmake/make build­ker­nel KERNCONF=MyKernel
  • /usr/obj/…/src/usr.bin/bmake/make instal­lk­er­nel KERNCONF=MyKernel KODIR=/boot/kernel.10
  • merge­mas­ter ‑p
  • /usr/obj/…/src/usr.bin/bmake/make install­world DESTDIR=/somewhere/test
  • mkdir /root/net10; cp /somewhere/test/rescue/ifconfig /somewhere/test/rescue/route /root/net10
  • cre­ate the file /etc/rc.10update with:
    case $(uname ‑r) in
    8.2*)
            MYIFCONFIG=/sbin/ifconfig
            MYROUTE=/sbin/route
            ;;
    10*)
            MYIFCONFIG=/root/net10/ifconfig
            MYROUTE=/root/net10/route
            ;;
    esac
    export MYIFCONFIG
    export MYROUTE
  • change the files (stu­pid approach: grep for “ifcon­fig” and “route” in /etc/rc.d to iden­ti­fy files which need to change, I skipped files which I iden­ti­fied as not need­ed in my case, if you use pf/IPv6/bridge/…, you may have to change some more files) /etc/rc.d/auto_linklocal /etc/rc.d/defaultroute /etc/rc.d/netif /etc/rc.d/netwait /etc/rc.d/routing: add “. /etc/rc.10update” at the end of the block with “. /etc/rc.subr”, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • change /etc/network.subr: add “. /etc/rc.10update” before the first func­tion, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • make sure that the changes you made are 100% cor­rect, rather triple-check than to not check at all (you will be locked out if they are not 100% OK)
  • stop any jails and make sure they do not restart at boot
  • deac­ti­vate the gmir­ror of the root-fs, if there is one (it is maybe eas­i­er to ask a remote hand to swap the boot order in case of problems)
  • here you could just a reboot of the serv­er to come back to your cur­rent OS ver­sion, so make sure that the mod­i­fi­ca­tions in /etc did not cause any prob­lems with the old ver­sion (in case you see prob­lems with the v10 ker­nel), but if you do not have a remote con­sole to single-user mode you have no chance to direct­ly fix the prob­lem (risk mit­i­ga­tion described above), no mat­ter which ver­sion of the ker­nel you boot
  • next­boot ‑k kernel.10
  • shut­down ‑r now
  • login
  • check dmesg
  • option­al: mv /boot/kernel /boot/kernel.8
  • make instal­lk­er­nel KERNCONF=MyKernel
    to have a v10 /boot/kernel
  • make install­world
  • merge­mas­ter
  • make delete-old
  • rm ‑r /etc/rc.10update /root/net10
  • change rc.conf: add “inet” in ifconfig-aliases
  • review sysctl.conf for out­dat­ed entries
  • shut­down ‑r now
  • option­al: rm ‑r /boot/kernel.10
  • enable jails again (or lat­er… updat­ing jails is not described here)
  • activate/resync mirror(s)
  • rebuild all ports (atten­tion: new pkg system)
  • make delete-old-libs
  • reboot again to make sure every­thing is OK after the port-rebuild and removal of old libs (a console.log (syslog.conf) helps here

Cal­cu­lat­ing the tar­get size of H264 videos

From time to time I con­vert videos to H264. When I do this I want to get the best qual­i­ty out of a give file­size. This means I cre­ate VBR videos. The ques­tion here is, how big the tar­get video file shall be.

After search­ing around a lit­tle bit in the net I found a for­mu­la which is sup­posed to give a hint about the tar­get file­size. Nat­u­ral­ly this depends on the encoder (or even the encoder-version), the encoder set­tings and even the video.

The for­mu­la is at least a good hint for my use, so I wrote a script which cal­cu­lates sev­er­al file­size val­ues for a giv­en video (based upon the out­put of medi­ain­fo, which the scripts expects in a file with the file­name as an argu­ment to the script). It cal­cu­lates a CBR and a VBR val­ue for a giv­en video based upon the width, height and dura­tion. It should work on all sys­tem with a POSIX com­pat­i­ble shell.

Exam­ple out­put for a video from my HD-ready cam, orig­i­nal file­size 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per sec­ond: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per sec­ond: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per sec­ond: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sec­tions, the dif­fer­ence is the “motion” val­ue. It is a kind of mul­ti­pli­ca­tor depend­ing on the amount of motion in the video. For the videos I made myself (fam­i­ly videos, and even some videos of vol­ley ball games), the first sec­tion seems to be just fine. So I reduced the orig­i­nal MP4 file to about 50% (not vis­i­ble here is the audio size, nor­mal­ly I copy the orig­i­nal audio unmodified).

For the curi­ous ones, the for­mu­la is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps val­ue. The CBR val­ue is

bps * playtime_in_seconds / 8

and the VBR val­ue is

34 * CBR_value.

Algo­rithm to detect repo-copies in CVS

FreeB­SD is on its way to move from CVS to SVN  for the ver­sion con­trol sys­tem for the Ports Col­lec­tion. The deci­sion was made to keep the com­plete his­to­ry, so the com­plete CVS repos­i­to­ry has to be con­vert­ed to SVN.

As CVS has no way to record a copy or move of files inside the repos­i­to­ry, we copied the CVS files inside the repos­i­to­ry in case we want­ed to copy or move a file (the so called “repocopy”). While this allows to see the full his­to­ry of a file, the draw­back is that you do not real­ly know when a file was copied/moved if you are not strict at record­ing this info after doing a copy. Guess what, we where not.

Now with the move to SVN which has a build-in way for copies/moves, it would be nice if we could record this info. In an inter­nal dis­cus­sion some­one told its not pos­si­ble to detect a repocopy reliably.

Well, I thought oth­er­wise and an hour lat­er my mail went out how to detect one. The longest time was need­ed to write how to do it, not to come up with a solu­tion. I do not know if some­one picked up this algo­rithm and imple­ment­ed some­thing for the cvs2svn con­vert­er, but I decid­ed to pub­lish the algo­rithm here if some­one needs a sim­i­lar func­tion­al­i­ty some­where else. Note, the fol­low­ing is tai­lored to the struc­ture of the Ports Col­lec­tion. This allows to speed up some things (no need to do all steps on all files). If you want to use this in a gener­ic repos­i­to­ry where the struc­ture is not as reg­u­lar as in our Ports Col­lec­tion, you have to run this algo­rithm on all files.

It also detects com­mits where mul­ti­ple files where com­mit­ted at once in one com­mit (sweep­ing commits).

Prepa­ra­tion

  • check only category/name/Makefile
  • gen­er­ate a hash of each commitlog+committer
  • if you are memory-limited use ha/sh/ed/dirs/cvs-rev and store path­name in the list cvs-rev (path­name = “category-name”) as storage
  • store the hash also in pathname/cvs-rev

If you have only one item in ha/sh/ed/dirs/cvs-rev in the end, there was no repocopy and no sweep­ing com­mit, you can delete this ha/sh/ed/dirs/cvs-rev.

If you have more than … let’s say … 10 (sub­ject to tun­ing) path­names in ha/sh/ed/dirs/cvs-rev you found a sweep­ing com­mit and you can delete the ha/sh/ed/dirs/cvs-rev.

The meat

The remain­ing ha/sh/ed/dirs/cvs-rev are prob­a­bly repocopies. Take one ha/sh/ed/dirs/cvs-rev and for each path­name (there may be more than 2 path­names) in there have a look at pathname/. Take the first cvs-rev of each and check if they have the same hash. Con­tin­ue with the next rev-number for each until you found a cvs-rev which does not con­tain the same hash. If the num­ber of cvs-revs since the begin­ning is >= … let’s say … 3 (sub­ject to tun­ing), you have a can­di­date for a repocopy. If it is >=  … 10 (sub­ject to tun­ing), you have a very good indi­ca­tor for a repocopy. You have to pro­ceed until you have only one path­name left.

You may detect mul­ti­ple repocopies like A->B->C->D or A->B + A->D + A->C here.

Write out the repocopy can­di­date to a list and delete the ha/sh/ed/dirs/cvs-rev for each cvs-rev in a detect­ed sequence.

This finds repocopy can­di­dates for category/name/Makefile. To detect the cor­rect repocopy-date (there are maybe cas­es where anoth­er file was changed after the Make­file but before the repocopy), you now have to look at all the files for a giv­en repocopy-pair and check if there is a match­ing com­mit after the Makefile-commit-date. If you want to be 100% sure, you com­pare the com­plete commit-history of all files for a giv­en repocopy-pair.