Sol­ar­is 10/11(.3) boot panic/​crash after mov­ing rpool to a new stor­age sys­tem

Situ­ation

The boot disks of some Sol­ar­is LDOMs were mi­grated from one stor­age sys­tem to an­oth­er one via ZFS mir­ror­ing the rpool to the new sys­tem and de­tach­ing the old LUN.

Is­sue

After re­boot with on the new stor­age sys­tem Sol­ar­is 10 and 11(.3) pan­ic at boot.

Cause

  • rpool not on slice 0 but on slice 2
  • bug in Sol­ar­is when do­ing such a mir­ror and “just” do­ing a re­boot <- this is the real is­sue, it seems Sol­ar­is can not handle a change of the name of the un­der­ly­ing device for a rpool, as just mov­ing the par­ti­tion­ing to slice 0 is not fix­ing the pan­ic.

Fix

# boot from net­work (or an al­tern­ate pool which was not yet moved), import/​export the pools, boot from the pools
boot net -
# go to shell
# if needed: change the par­ti­tion­ing so that slice 0 has the same val­ues as slice 2 (re­spect­ively make sure the rpool is in slice 0)
zpool im­port -R /​tmp/​yyy rpool
zpool ex­port rpool
re­boot

 

ioc­age: HOWTO cre­ate a base­jail from src (in­stead of from an of­fi­cial re­lease)

Back­ground

So far I have used ez­jail to man­age FreeBSD jails. I use jails since years to have dif­fer­ent parts of a soft­ware stack in some kind of a con­tain­er (in a ZFS data­set for the filesys­tem side of the con­tain­er). On one hand to not let de­pend­en­cies of one part of the soft­ware stack have in­flu­ence of oth­er parts of the soft­ware stack. On the oth­er hand to have the pos­sib­il­ity to move parts of the soft­ware stack to a dif­fer­ent sys­tem if ne­ces­sary. Nor­mally I run -stable or -cur­rent or more gen­er­ally speak­ing, a self-​compiled FreeBSD on those sys­tems. In ez­jail I like the fact that all jails on a sys­tem have one com­mon base­jail un­der­ly­ing, so that I up­date one place for the user­land and all jails get the up­dated code.

Since a while I heard good things about ioc­age and how it in­teg­rates ZFS, so I de­cided to give it a try my­self. As ioc­age does not come with an of­fi­cial way of cre­at­ing a base­jail (re­spect­ively a re­lease) from a self-​compiled FreeBSD (at least doc­u­mented in those places I looked, and yes, I am aware that I can cre­ate a FreeBSD re­lease my­self and use it, but I do not like to have to cre­ate a re­lease ad­di­tion­ally to the build­world I use to up­date the host sys­tem) here now the short HOWTO achieve this.

In­vari­ants

In the fol­low­ing I as­sume the ioc­age ZFS parts are already cre­ated in data­set ${POOLNAME}/iocage which is moun­ted on ${IOCAGE_BASE}/iocage. Ad­di­tion­ally the build­world in /​usr/​src (or wherever you have the FreeBSD source) should be fin­ished.

Pre-​requisites

To have the ne­ces­sary dataset-​infrastructure cre­ated for own basejails/​releases, at least one of­fi­cial re­lease needs to be fetched be­fore. So run the com­mand be­low (if there is no ${IOCAGE_BASE}/iocage/releases dir­ect­ory) and fol­low the on-​screen in­struc­tions.

ioc­age fetch

HOWTO

Some vari­ables:

POOLNAME=mpool
SRC_REV=r$(cd /​usr/​src; svn­litever­sion)
IOCAGE_​BASE=””

Cre­at­ing the ioc­age basejail-​datasets for this ${SRC_​REV}:

zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/bin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/boot
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/lib
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/libexec
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/rescue
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/sbin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/include
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib32
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libdata
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libexec
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/share
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/src

In­stall from /​usr/​src (the ex­ecut­able “chown” is hard­linked across an ioc­age base­jail data­set bound­ary, this fails in the nor­mal in­stall­world, so we have to ig­nore this er­ror and in­stall a copy of the chown bin­ary to the place where the hard­link nor­mally is):

cd /​usr/​src
make -i in­stall­world DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >&! iocage_installworld_base.log
cp -pv ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin/chown ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin/chgrp
make dis­tri­bu­tion DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root »& iocage_installworld_base.log

While we are here, also cre­ate a re­lease and not only a base­jail:

zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE
zfs cre­ate -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE/root
make in­stall­world DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >&! iocage_installworld_release.log
make dis­tri­bu­tion DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root »& iocage_installworld_release.log

And fi­nally make this the de­fault re­lease which ioc­age uses when cre­at­ing new jails (this is op­tion­al):

ioc­age set release=${SRC_REV}-RELEASE de­fault

Now the self-​build FreeBSD is avail­able in ioc­age for new jails.

HOWTO: “Blind” re­mote in­stall of FreeBSD via tiny disk im­age (ZFS edi­tion)

In a past post I de­scribed how to in­stall a FreeBSD re­motely via a tiny UFS based disk im­age over a linux sys­tem. In this post I de­scribe how to do it with a ZFS based disk im­age.

In­vari­ants

Giv­en a unix based re­mote sys­tem (in this case a linux sys­tem) from which you know what kind of hard­ware it runs on (e.g. PCI IDs) and what the cor­res­pond­ing FreeBSD drivers are.

HOWTO

In the title of this post I wrote “via a tiny disk im­age”. This is true for a suit­able defin­i­tion of tiny.

What we have in the root­server are two ~900 GB hard­disks. They shall be used in a soft­ware mir­ror. The ma­chine has 8 GB of RAM. I do not ex­pect much ker­nel pan­ics (= crash dumps) there, so we do not really need >8 GB of swap (for­get the rule of hav­ing twice as much swap than RAM, with the cur­rent amount of RAM in a ma­chine you are in “trouble” when you need even the same amount of swap than RAM). I de­cided to go with 2 GB of swap.

Pushing/​pulling a 900 GB im­age over the net­work to in­stall a sys­tem is not really some­thing I want to do. I am OK to trans­fer 5 GB (that is 0.5% of the en­tire disk) to get this job done, and this is feas­ible.

First let us define some vari­ables in the shell, this way you just need to change the val­ues in one place and the rest is copy&paste (I use the SVN re­vi­sion of the source which I use to in­stall the sys­tem as the name of the sysutils/​beadm com­pat­ible boot-​dataset in the root­fs, as such I also have the re­vi­sion num­ber avail­able in a vari­able):

ROOTFS_SIZE=5G
ROOTFS_NAME=root
FILENAME=rootfs
POOLNAME=mpool
VERSION=r$(cd /​usr/​src; svn­litever­sion)
SWAPSIZE=2G

Then change your cur­rent dir­ect­ory to a place where you have enough space for the im­age. There we will cre­ate a con­tainer for the im­age, and make it ready for par­ti­tion­ing:

trun­cate -s ${ROOTFS_​SIZE} ${FILENAME}
md­con­fig -a -t vnode -f ${FILENAME}
# if you want to fully al­loc­ate
# dd if=/dev/zero of=/dev/md0 bs=1m

Cre­ate the par­ti­tion table and the root­fs (in a sysutils/​beadm com­pat­ible way – as I in­stall FreeBSD-​current there – and mount it tem­por­ary to /​temppool):

gpart cre­ate -s GPT /​dev/​md0
gpart add -s 512K -t freebsd-​boot -l bootcode0 /​dev/​md0
gpart add -a 4k -t freebsd-​swap -s ${SWAPSIZE} -l swap0 /​dev/​md0
gpart add -a 1m -t freebsd-​zfs -l ${POOLNAME}0 /​dev/​md0
gpart boot­code -b /​boot/​pmbr -p /​boot/​gptzfsboot -i 1 /​dev/​md0
# if not already the case and you want to have 4k phys­ic­al sec­tor size of the pool
# sy­scl vfs.zfs.min_auto_ashift=12
zpool cre­ate -o cachefile=/boot/zfs/zpool.cache_temp -o altroot=/temppool -O compress=lz4 -O atime=off -O utf8only=on ${POOLNAME} /dev/gpt/${POOLNAME}0
zfs cre­ate -o mountpoint=none ${POOLNAME}/ROOT
zfs cre­ate -o mountpoint=/ ${POOLNAME}/ROOT/${VERSION}
zfs cre­ate -o mountpoint=/tmp -o exec=on -o setuid=off ${POOLNAME}/tmp
zfs cre­ate -o mountpoint=/usr -o canmount=off ${POOLNAME}/usr
zfs cre­ate -o mountpoint=/home ${POOLNAME}/home
zfs cre­ate -o setuid=off ${POOLNAME}/usr/ports
zfs cre­ate ${POOLNAME}/usr/src
zfs cre­ate -o mountpoint=/var -o canmount=off ${POOLNAME}/var
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/audit
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/crash
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/var/log
zfs cre­ate -o atime=on ${POOLNAME}/var/mail
zfs cre­ate -o setuid=off ${POOLNAME}/var/tmp
zfs cre­ate ${POOLNAME}/var/ports
zfs cre­ate -o exec=off -o setuid=off -o mountpoint=/shared ${POOLNAME}/shared
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/shared/distfiles
zfs cre­ate -o exec=off -o setuid=off ${POOLNAME}/shared/packages
zfs cre­ate -o exec=off -o setuid=off -o compression=lz4 ${POOLNAME}/shared/ccache
zfs cre­ate ${POOLNAME}/usr/obj
zpool set bootfs=${POOLNAME}/ROOT/${VERSION} ${POOLNAME}

In­stall FreeBSD (from source):

cd /​usr/​src
#make build­world >&! buildworld.log
#make buildker­nel -j 8 KERNCONF=GENERIC >&! buildkernel_generic.log
make in­stall­world DESTDIR=/temppool/ >& installworld.log
make dis­tri­bu­tion DESTDIR=/temppool/ >& distrib.log
make in­stallker­nel KERNCONF=GENERIC DESTDIR=/temppool/ >& installkernel.log

Copy the tem­por­ary zpool cache cre­ated above in the pool-​creation part to the im­age (I have the im­pres­sion it is not really needed and will work without, but I have not tried this):

cp /boot/zfs/zpool.cache_temp /​temppool/​boot/​
cp /boot/zfs/zpool.cache_temp /temppool/boot/zpool.cache

Add the zfs mod­ule to loader.conf:

zfs_load=“yes”
opensolaris_load=“yes”

Now you need to cre­ate /temppool/etc/rc.conf (set the de­faultrouter, the IP ad­dress via ifconfig_​IF (and do not for­get to use the right IF for it), the host­name, set sshd_​enable to yes, zfs_enable=“YES”)  /temppool/boot/loader.conf (zfs_load=“yes”, opensolaris_load=“yes”, vfs.root.mountfrom=“zfs:${POOLNAME}/ROOT/r${VERSION}”)
/​temppool/​etc/​hosts, /temppool/etc/resolv.conf and maybe /temppool/etc/sysctl.conf and /temppool/etc/periodic.conf.

Do not al­low password-​less root lo­gins in single-​user mode on the phys­ic­al con­sole, cre­ate a resolv.conf and an user:

cd /​temppool/​etc
sed -ie „s:console.*off.:&in:“ ttys
cat >resolv.conf «EOT
search YOURDOMAIN
nameserv­er 8.8.8.8
EOT
pw -V /​temppool/​etc groupadd YOURGROUP -g 1001
pw -V /​temppool/​etc useradd YOURUSER -u 1001 -d /​home/​YOURUSER -g YOURUSER -G wheel -s /​bin/​tcsh
pw -V /​temppool/​etc user­mod YOURUSER -h 0 pw -V /​temppool/​etc user­mod root -h 0
zfs cre­ate mpool/​home/​YOURUSER
chown YOURUSER:YOURGROUP /​temppool/​home/​YOURUSER

Now you can make some more modi­fic­a­tions to the sys­tem if wanted, and then ex­port the pool and de­tach the im­age:

zpool ex­port ${POOLNAME}

md­con­fig -d -u 0

De­pend­ing on the up­load speed you can achieve, it is be­ne­fi­cial to com­press the im­age now, e.g. with bzip2. Then trans­fer the im­age to the disk of the re­mote sys­tem. In my case I did this via:

ssh –C –o CompressionLevel=9 root@remote_host dd of=/dev/hda bs=1m < /path/to/${FILENAME}

Then reboot/​power-​cycle the re­mote sys­tem.

Post-​install tasks

Now we have a new FreeBSD sys­tem which uses only a frac­tion of the the hard­disk and is not re­si­li­ent against harddisk-​failures.

FreeBSD will de­tect that the disk is big­ger than the im­age we used when cre­at­ing the GPT la­bel and warn about it (cor­rupt GPT table). To fix this and to res­ize the par­ti­tion for the zpool to use the en­tire disk we first mir­ror the zpool to the second disk and res­ize the par­ti­tion of the first disk, and when the zpool is in-​sync and then we res­ize the boot disk (at­ten­tion, you need to change the “-s” part in the fol­low­ing to match your disk size).

First backup the la­bel of the first disk, this makes it more easy to cre­ate the la­bel of the second disk:

/​sbin/​gpart backup ada0 > ada0.gpart

Edit ada0.gpart (give dif­fer­ent names for the la­bels, mainly change the num­ber 0 on the label-​name to 1) and then use it to cre­ate the par­ti­tion of the second disk:

gpart re­store -Fl ada1 < ada0.gpart
gpart res­ize -i 3 -a 4k -s 929g ada1
gpart boot­code -b /​boot/​pmbr -p /​boot/​gptzfsboot -i 1 ada1
zpool set autoexpand=on mpool

Fix the warn­ing about the GPT la­bel and res­ize the par­ti­tion:

gpart re­cov­er ada0
gpart res­ize -i 3 -a 4k -s 929g ada0

Af­ter­wards it should look sim­il­ar to this:

gpart show -l
=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  bootcode0  (512K)
        1064     4194304     2  swap0  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool0  (929G)
  1952450560     1074568        – free –  (525M)

=>        40  1953525088  ada1  GPT  (932G)
          40        1024     1  bootcode1  (512K)
        1064     4194304     2  swap1  (2.0G)
     4195368         984        – free –  (492K)
     4196352  1948254208     3  mpool1  (929G)
  1952450560     1074568        – free –  (525M)

Add the second disk to the zpool:

zpool at­tach mpool gpt/​mpool0 gpt/​mpool1

When the mir­ror is in sync (zpool status mpool), we can ex­tend the size of the pool it­self:

zpool off­line mpool /​dev/​gpt/​mpool0
zpool on­line mpool /​dev/​gpt/​mpool0

As a last step we can add now an en­cryp­ted swap (de­pend­ing on the im­port­ance of the sys­tem maybe a gmirror-​ed one – not ex­plained here), and spe­cify where to dump (text-​dumps) on.

/boot/loader.conf:

dumpdev=“/dev/ada0p2”

/etc/rc.conf:

dumpdev=“/dev/gpt/swap0”
crashinfo_enable=“YES”
ddb_enable=“yes”
encswap_enable=“YES”
geli_swap_flags=”-a hmac/​sha256 -l 256 -s 4096 -d”

/​etc/​fstab:

# Device        Moun­t­point      FStype  Op­tions                 Dump    Pass#
/dev/ada1p2.eli none    swap    sw      0 0

Now the sys­tem is ready for some ap­plic­a­tions.

Gain­ing space on An­droid after ART->Dalvik switch (root ac­cess re­quired)

I (still) use a Nex­us S phone. I am us­ing Cy­ano­gen­mod on it. After an art­icle in a com­puter magazine I de­cided to give the ART-​runtime a try in­stead of the de­fault Dalvic-​runtime. Un­for­tu­nately I do not have enough free space free (and all what I can is moved to the USB stor­age already) to really use the ART-​runtime.

After switch­ing back to the Dalvic-​runtime, I had only 23 of the pre­vi­ously avail­able space free. After a little bit of look­ing around I found /​data/​dalvik-​cache. I de­leted with a file man­ager the con­tent of the dir­ect­ory (you will get some “app crashed/​died” mes­sages) and re­booted the phone (this is not the same as format­ting the cache par­ti­tion in the re­cov­ery sys­tem).

Dur­ing boot it pop­u­lated the dir­ect­ory again and now I have more than 43 of free space on the in­tern­al stor­age.

Up­dat­ing FreeBSD 8.2 (or 9.x) to 10 (beta4)

This is a little de­scrip­tion how I re­motely (no con­sole, booted in­to multi-​user dur­ing up­date, no ex­tern­al ser­vices like jails/​httpd/​… run­ning) up­dated a FreeBSD 8.2 to 10 (beta4) from source. This should also work when up­dat­ing from FreeBSD 9.x. Note, I had already switched to ATA_​CAM on 8.2, so not in­struc­tions for the name change of the ata devices. No IPv6, WLAN or CARP is in use here, so changes which are needed in this area are not covered. Read UPDATING care­fully, there are a lot of changes between ma­jor re­leases.

What I did:

  • up­date /​usr/​src
  • make build­world
  • re­place “make ” in /usr/src/Makefile.inc1 with ${MAKE} (two times, one for “VERSION”, one for “BRANCH”)
  • veri­fy ker­nel con­fig for changes needed (run­ning “con­fig MyKer­nel” in /​usr/​src/​sys/​YourArch/​conf/​ helps to identi­fy syn­tax prob­lems), sorry I didn’t take notes, but I diffed the old and the new GENERIC con­fig and added/​removed ac­cord­ing to my in­terests
  • /usr/obj/…/src/usr.bin/bmake/make buildker­nel KERNCONF=MyKernel
  • /usr/obj/…/src/usr.bin/bmake/make in­stallker­nel KERNCONF=MyKernel KODIR=/boot/kernel.10
  • merge­mas­ter -p
  • /usr/obj/…/src/usr.bin/bmake/make in­stall­world DESTDIR=/somewhere/test
  • mk­dir /​root/​net10; cp /​somewhere/​test/​rescue/​ifconfig /​somewhere/​test/​rescue/​route /​root/​net10
  • cre­ate the file /etc/rc.10update with:
    case $(un­ame -r) in
    8.2*)
            MYIFCONFIG=/sbin/ifconfig
            MYROUTE=/sbin/route
            ;;
    10*)
            MYIFCONFIG=/root/net10/ifconfig
            MYROUTE=/root/net10/route
            ;;
    es­ac
    ex­port MYIFCONFIG
    ex­port MYROUTE
  • change the files (stu­pid ap­proach: grep for “if­con­fig” and “route” in /etc/rc.d to identi­fy files which need to change, I skipped files which I iden­ti­fied as not needed in my case, if you use pf/​IPv6/​bridge/​…, you may have to change some more files) /etc/rc.d/auto_linklocal /etc/rc.d/defaultroute /etc/rc.d/netif /etc/rc.d/netwait /etc/rc.d/routing: add “. /etc/rc.10update” at the end of the block with “. /etc/rc.subr”, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • change /​etc/​net­work.subr: add “. /etc/rc.10update” be­fore the first func­tion, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • make sure that the changes you made are 100% cor­rect, rather triple-​check than to not check at all (you will be locked out if they are not 100% OK)
  • stop any jails and make sure they do not re­start at boot
  • de­ac­tiv­ate the gmir­ror of the root-​fs, if there is one (it is maybe easi­er to ask a re­mote hand to swap the boot or­der in case of prob­lems)
  • here you could just a re­boot of the serv­er to come back to your cur­rent OS ver­sion, so make sure that the modi­fic­a­tions in /​etc did not cause any prob­lems with the old ver­sion (in case you see prob­lems with the v10 ker­nel), but if you do not have a re­mote con­sole to single-​user mode you have no chance to dir­ectly fix the prob­lem (risk mit­ig­a­tion de­scribed above), no mat­ter which ver­sion of the ker­nel you boot
  • next­boot -k kernel.10
  • shut­down -r now
  • lo­gin
  • check dmesg
  • op­tion­al: mv /​boot/​kernel /boot/kernel.8
  • make in­stallker­nel KERNCONF=MyKernel
    to have a v10 /​boot/​kernel
  • make in­stall­world
  • merge­mas­ter
  • make delete-​old
  • rm -r /etc/rc.10update /​root/​net10
  • change rc.conf: add “in­et” in ifconfig-​aliases
  • re­view sysctl.conf for out­dated entries
  • shut­down -r now
  • op­tion­al: rm -r /boot/kernel.10
  • en­able jails again (or later… up­dat­ing jails is not de­scribed here)
  • activate/​resync mirror(s)
  • re­build all ports (at­ten­tion: new pkg sys­tem)
  • make delete-​old-​libs
  • re­boot again to make sure everything is OK after the port-​rebuild and re­mov­al of old libs (a console.log (syslog.conf) helps here

Cal­cu­lat­ing the tar­get size of H264 videos

From time to time I con­vert videos to H264. When I do this I want to get the best qual­ity out of a give files­ize. This means I cre­ate VBR videos. The ques­tion here is, how big the tar­get video file shall be.

After search­ing around a little bit in the net I found a for­mula which is sup­posed to give a hint about the tar­get files­ize. Nat­ur­ally this de­pends on the en­coder (or even the encoder-​version), the en­coder set­tings and even the video.

The for­mula is at least a good hint for my use, so I wrote a script which cal­cu­lates sev­er­al files­ize val­ues for a giv­en video (based upon the out­put of me­di­ainfo, which the scripts ex­pects in a file with the fi­le­name as an ar­gu­ment to the script). It cal­cu­lates a CBR and a VBR value for a giv­en video based upon the width, height and dur­a­tion. It should work on all sys­tem with a POSIX com­pat­ible shell.

Ex­ample out­put for a video from my HD-​ready cam, ori­gin­al files­ize 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 2
Per second: 6451200.000 bps /​ 6300 Kibps
Total CBR: 1148313600 bytes /​ 1121400 KiB /​ 1095 MiB
Total VBR: 861235200 bytes /​ 841050 KiB /​ 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 3
Per second: 9676800.000 bps /​ 9450 Kibps
Total CBR: 1722470400 bytes /​ 1682100 KiB /​ 1642 MiB
Total VBR: 1291852800 bytes /​ 1261575 KiB /​ 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 4
Per second: 12902400.000 bps /​ 12600 Kibps
Total CBR: 2296627200 bytes /​ 2242800 KiB /​ 2190 MiB
Total VBR: 1722470400 bytes /​ 1682100 KiB /​ 1642 MiB

There are 3 sec­tions, the dif­fer­ence is the “mo­tion” value. It is a kind of mul­ti­plic­at­or de­pend­ing on the amount of mo­tion in the video. For the videos I made my­self (fam­ily videos, and even some videos of vol­ley ball games), the first sec­tion seems to be just fine. So I re­duced the ori­gin­al MP4 file to about 50% (not vis­ible here is the au­dio size, nor­mally I copy the ori­gin­al au­dio un­mod­i­fied).

For the curi­ous ones, the for­mula is

width_​in_​pixels * height_​in_​pixels * fps * motion_​value * 0.07

for the bps value. The CBR value is

bps * playtime_​in_​seconds /​ 8

and the VBR value is

3 /​ 4 * CBR_​value.

Al­gorithm to de­tect repo-​copies in CVS

FreeBSD is on its way to move from CVS to SVN  for the ver­sion con­trol sys­tem for the Ports Col­lec­tion. The de­cision was made to keep the com­plete his­tory, so the com­plete CVS re­pos­it­ory has to be con­ver­ted to SVN.

As CVS has no way to re­cord a copy or move of files in­side the re­pos­it­ory, we copied the CVS files in­side the re­pos­it­ory in case we wanted to copy or move a file (the so called “re­po­copy”). While this al­lows to see the full his­tory of a file, the draw­back is that you do not really know when a file was copied/​moved if you are not strict at re­cord­ing this info after do­ing a copy. Guess what, we where not.

Now with the move to SVN which has a build-​in way for copies/​moves, it would be nice if we could re­cord this info. In an in­tern­al dis­cus­sion someone told its not pos­sible to de­tect a re­po­copy re­li­ably.

Well, I thought oth­er­wise and an hour later my mail went out how to de­tect one. The longest time was needed to write how to do it, not to come up with a solu­tion. I do not know if someone picked up this al­gorithm and im­ple­men­ted some­thing for the cvs2svn con­vert­er, but I de­cided to pub­lish the al­gorithm here if someone needs a sim­il­ar func­tion­al­ity some­where else. Note, the fol­low­ing is tailored to the struc­ture of the Ports Col­lec­tion. This al­lows to speed up some things (no need to do all steps on all files). If you want to use this in a gen­er­ic re­pos­it­ory where the struc­ture is not as reg­u­lar as in our Ports Col­lec­tion, you have to run this al­gorithm on all files.

It also de­tects com­mits where mul­tiple files where com­mit­ted at once in one com­mit (sweep­ing com­mits).

Pre­par­a­tion

  • check only category/​name/​Make­file
  • gen­er­ate a hash of each commitlog+committer
  • if you are memory-​limited use ha/​sh/​ed/​dirs/​cvs-​rev and store path­name in the list cvs-​rev (path­name = “category-​name”) as stor­age
  • store the hash also in pathname/​cvs-​rev

If you have only one item in ha/​sh/​ed/​dirs/​cvs-​rev in the end, there was no re­po­copy and no sweep­ing com­mit, you can de­lete this ha/​sh/​ed/​dirs/​cvs-​rev.

If you have more than … let’s say … 10 (sub­ject to tun­ing) path­names in ha/​sh/​ed/​dirs/​cvs-​rev you found a sweep­ing com­mit and you can de­lete the ha/​sh/​ed/​dirs/​cvs-​rev.

The meat

The re­main­ing ha/​sh/​ed/​dirs/​cvs-​rev are prob­ably re­po­cop­ies. Take one ha/​sh/​ed/​dirs/​cvs-​rev and for each path­name (there may be more than 2 path­names) in there have a look at pathname/​. Take the first cvs-​rev of each and check if they have the same hash. Con­tin­ue with the next rev-​number for each un­til you found a cvs-​rev which does not con­tain the same hash. If the num­ber of cvs-​revs since the be­gin­ning is >= … let’s say … 3 (sub­ject to tun­ing), you have a can­did­ate for a re­po­copy. If it is >=  … 10 (sub­ject to tun­ing), you have a very good in­dic­at­or for a re­po­copy. You have to pro­ceed un­til you have only one path­name left.

You may de­tect mul­tiple re­po­cop­ies like A->B->C->D or A->B + A->D + A->C here.

Write out the re­po­copy can­did­ate to a list and de­lete the ha/​sh/​ed/​dirs/​cvs-​rev for each cvs-​rev in a de­tec­ted se­quence.

This finds re­po­copy can­did­ates for category/​name/​Makefile. To de­tect the cor­rect repocopy-​date (there are maybe cases where an­oth­er file was changed after the Make­file but be­fore the re­po­copy), you now have to look at all the files for a giv­en repocopy-​pair and check if there is a match­ing com­mit after the Makefile-​commit-​date. If you want to be 100% sure, you com­pare the com­plete commit-​history of all files for a giv­en repocopy-​pair.