Alexander Leidinger

Just another weblog


Updat­ing FreeBSD 8.2 (or 9.x) to 10 (beta4)

This is a lit­tle descrip­tion how I remotely (no con­sole, booted into multi-user dur­ing update, no exter­nal ser­vices like jails/httpd/… run­ning) updated a FreeBSD 8.2 to 10 (beta4) from source. This should also work when updat­ing from FreeBSD 9.x. Note, I had already switched to ATA_CAM on 8.2, so not instruc­tions for the name change of the ata devices. No IPv6, WLAN or CARP is in use here, so changes which are needed in this area are not cov­ered. Read UPDATING care­fully, there are a lot of changes between major releases.

What I did:

  • update /usr/src
  • make build­world
  • replace “make ” in /usr/src/Makefile.inc1 with ${MAKE} (two times, one for “VERSION”, one for “BRANCH”)
  • ver­ify ker­nel con­fig for changes needed (run­ning “con­fig MyK­er­nel” in /usr/src/sys/YourArch/conf/ helps to iden­tify syn­tax prob­lems), sorry I didn’t take notes, but I diffed the old and the new GENERIC con­fig and added/removed accord­ing to my interests
  • /usr/obj/…/src/usr.bin/bmake/make build­ker­nel KERNCONF=MyKernel
  • /usr/obj/…/src/usr.bin/bmake/make instal­lk­er­nel KERNCONF=MyKernel KODIR=/boot/kernel.10
  • merge­mas­ter –p
  • /usr/obj/…/src/usr.bin/bmake/make install­world DESTDIR=/somewhere/test
  • mkdir /root/net10; cp /somewhere/test/rescue/ifconfig /somewhere/test/rescue/route /root/net10
  • cre­ate the file /etc/rc.10update with:
    case $(uname –r) in
    export MYIFCONFIG
    export MYROUTE
  • change the files (stu­pid approach: grep for “ifcon­fig” and “route” in /etc/rc.d to iden­tify files which need to change, I skipped files which I iden­ti­fied as not needed in my case, if you use pf/IPv6/bridge/…, you may have to change some more files) /etc/rc.d/auto_linklocal /etc/rc.d/defaultroute /etc/rc.d/netif /etc/rc.d/netwait /etc/rc.d/routing: add “. /etc/rc.10update” at the end of the block with “. /etc/rc.subr”, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • change /etc/net­work.subr: add “. /etc/rc.10update” before the first func­tion, change the “ifconfig”-command to ${MYIFCONFIG}, change the “route”-command to ${MYROUTE}
  • make sure that the changes you made are 100% cor­rect, rather triple-check than to not check at all (you will be locked out if they are not 100% OK)
  • stop any jails and make sure they do not restart at boot
  • deac­ti­vate the gmir­ror of the root-fs, if there is one (it is maybe eas­ier to ask a remote hand to swap the boot order in case of problems)
  • here you could just a reboot of the server to come back to your cur­rent OS ver­sion, so make sure that the mod­i­fi­ca­tions in /etc did not cause any prob­lems with the old ver­sion (in case you see prob­lems with the v10 ker­nel), but if you do not have a remote con­sole to single-user mode you have no chance to directly fix the prob­lem (risk mit­i­ga­tion described above), no mat­ter which ver­sion of the ker­nel you boot
  • next­boot –k kernel.10
  • shut­down –r now
  • login
  • check dmesg
  • optional: mv /boot/kernel /boot/kernel.8
  • make instal­lk­er­nel KERNCONF=MyKernel
    to have a v10 /boot/kernel
  • make install­world
  • merge­mas­ter
  • make delete-old
  • rm –r /etc/rc.10update /root/net10
  • change rc.conf: add “inet” in ifconfig-aliases
  • review sysctl.conf for out­dated entries
  • shut­down –r now
  • optional: rm –r /boot/kernel.10
  • enable jails again (or later… updat­ing jails is not described here)
  • activate/resync mirror(s)
  • rebuild all ports (atten­tion: new pkg system)
  • make delete-old-libs
  • reboot again to make sure every­thing is OK after the port-rebuild and removal of old libs (a console.log (syslog.conf) helps here
GD Star Rat­ing
GD Star Rat­ing


Lin­ux­u­la­tor explained: How to cre­ate Linux bina­ries on FreeBSD

There may by cases where you want to gen­er­ate a Linux binary on a FreeBSD machine. This is not a prob­lem with the lin­ux­u­la­tor, but not with the default linux_base port.

As you may know, the linux_base port is designed to deliver an inte­grated expe­ri­ence with FreeBSD native pro­grams. As such some parts of the native FreeBSD infra­struc­ture is used. If you would try to use a Linux–com­piler to gen­er­ate Linux–bina­ries, you would run into the prob­lem that by default the FreeBSD includes are used.


To have a fully fea­tured and non-integrated Linux envi­ron­ment on your FreeBSD sys­tem either mount an exist­ing (and com­pat­i­ble) Linux instal­la­tion some­where into your FreeBSD sys­tem, or install a linux_dist port. This can be done addi­tion­ally to an already installed linux_base port.


When you have a com­plete Linux envi­ron­ment avail­able, you need to mount the FreeBSD devfs to /path/to/complete_linux/dev, lin­procfs to /path/to/complete_linux/proc and lin­sysfs to /path/to/complete_linux/sys to have a com­plete setup.

Use it

Now you just need to chroot into this  /path/to/complete_linux and you configure/make/install or what­ever you need to do to gen­er­ate your desired Linux binary.

GD Star Rat­ing
GD Star Rat­ing


Sta­tus crypto cards HOWTO: prob­lems with the card reader (sup­port could be better)

After hours (spread over weeks) I come to the con­clu­sion that there is a lot of poten­tial to improve the doc­u­men­ta­tion of card read­ers (but I doubt the card reader ven­dors will do it) and of the pcsc doc­u­men­ta­tion. It is not easy to arrive at a point where you under­stand every­thing. The com­pat­i­bil­ity list does not help much, as the card read­ers are partly past their end of life and the mod­els which replace them are not listed. Respec­tively the one I bought does not sup­port all the fea­tures I need. I even ported the dri­ver to FreeBSD (not com­mit­ted, I wanted to test every­thing first) and a lot of stuff works, but one crit­i­cal part is that I can not store a cer­tifi­cate on the crypto card as the card reader or the dri­ver  does not sup­port extended APDUs (needed to trans­fer more than 255 bytes to the card reader).

Well, the sta­tus so far:

  • I have a HOWTO what to install to use crypto cards in FreeBSD
  • I have a HOWOT what to install / con­fig­ure in Windows
  • I have a HOWTO regard­ing cre­at­ing keys on a openpgp v2 card and how to use this key with ssh on FreeBSD (or any other unix-like OS which can run pcsc)
  • I have a card reader which does not sup­port extended APDUs
  • I want to make sure what I write in the HOW­TOs is also suit­able for the use with Win­dows / PuTTY
  • it seems Win­dows needs a cer­tifi­cate and not only a key when using the Win­dows CAPI (using the ven­dor sup­plied card reader dri­ver) in PuTTY-CSC (works at work with a USB token)
  • the pcsc pkcs11 Win­dows DLL is not suit­able yet for use on Win­dows 8 64bit
  • I con­tacted the card reader ven­dor if the card reader or the dri­ver is the prob­lem regard­ing the extended APDUs
  • I found prob­lems in gpg4win / pcsc on Win­dows 8
  • I have send some money to the devel­op­ers of gpg4win to sup­port their work (if you use gnupg on Win­dows, try to send a few units of money to them, the work stag­nated as they need to spend their time for paid work)

So either I need a new card reader, or have to wait for an update of the linux dri­ver of the ven­dor… which prob­a­bly means it may be a lot faster to buy a new card reader. When look­ing for one with at least a PIN pad, I either do not find any­thing which is listed as sup­ported by pcsc on the ven­dor pages (it is incred­i­ble how hard it is to nav­i­gate the web­sites of some com­pa­nies… a lot of buzz­words but no way to get to the real prod­ucts), or they only list updated mod­els where I do not know if they will work.

When I have some­thing which works with FreeBSD and Win­dows, I will pub­lish all the HOW­TOs here at once.

GD Star Rat­ing
GD Star Rat­ing


OpenPGP crypto cards ordered

I wrote in a pre­vi­ous blog post that I want to switch to crypto cards for use with ssh and GnuPG. After some research I set­tled on the OpenPGP cryto cards. I ordered them from ker­nel­con­cepts. As soon as they arrive (and I have some free time), I will start to use them and write down how to work with them with FreeBSD.

GD Star Rat­ing
GD Star Rat­ing


Com­plete net­work loss on Solaris 10u10 CPU 2012-10 on vir­tu­al­ized T4-2

The prob­lem I see at work: A T4-2 with 3 guest LDOMs, vir­tu­al­ized disks and net­works lost the com­plete net­work con­nec­tiv­ity “out of the blue” once, and maybe “spo­radic” directly after a cold boot. After a lot of dis­cus­sion with Ora­cle, I have the impres­sion that we have two prob­lems here.

1st prob­lem:
Total net­work loss of the machine (no zone or guest LDOM or the pri­mary LDOM was able to have receive or send IP pack­ets). This hap­pened once. No idea how to repro­duce it. In the logs we see the mes­sage “[ID 920994 kern.warning] WARNING: vnetX: exceeded num­ber of per­mit­ted hand­shake attempts (5) on chan­nel xxx”. Accord­ing to Ora­cle this is sup­posed to be fixed in 148677 – 01 which will come with Solaris 10u11. They sug­gested to use a vsw inter­face instead of a vnet inter­face on the pri­mary domain to at least lower the prob­a­bil­ity of this prob­lem hit­ting us. They were not able to tell us how to repro­duce the prob­lem (seems to be a race con­di­tion, at least I get this impres­sion based upon the descrip­tion of the Ora­cle engi­neer han­dling the SR). Only a reboot helped to get the prob­lem solved. I was told we are the only client which reported this kind of prob­lem, the patch for this prob­lem is based upon an inter­nal bugre­port from inter­nal tests.

2nd prob­lem:
After cold boots some­times some machines (not all) are not able to con­nect to an IP on the T4. A reboot helps, as does remov­ing an inter­face from an aggre­gate and directly adding it again (see below for the sys­tem con­fig). To try to repro­duce the prob­lem, we did a lot of warm reboots of the pri­mary domain, and the prob­lem never showed up. We did some cold reboots, and the prob­lem showed up once.

In case some­one else sees one of those prob­lems on his machines too, please get in con­tact with me to see what we have in com­mon to try to track this down fur­ther and to share info which may help in maybe repro­duc­ing the problems.

Sys­tem setup:

  • T4-2 with 4 HBAs and 8 NICs (4 * igb on-board, 4 * nxge on addi­tional net­work card)
  • 3 guest LDOMs and one io+control domain (both in the pri­mary domain)
  • the guest LDOMs use SAN disks over the 4 HBAs
  • the pri­mary domain uses a mir­rored zpool on SSDs
  • 5 vswitch in the hypervisor
  • 4 aggre­gates (aggr1 — aggr4 with L2-policy), each one with one igb and one nxge NIC
  • each aggre­gate is con­nected to a sep­a­rate vswitch (the 5th vswitch is for machine-internal communication)
  • each guest LDOM has three vnets, each vnets con­nected to a vswitch (1 guest LDOM has aggr1+2 only for zones (via vnets), 2 guest LDOMs have aggr 3+4 only for zones (via vnets), and all LDOMs have aggr2+3 (via vnets) for global-zone com­mu­ni­ca­tion, all LDOMs are addi­tion­ally con­nected to the machine-internal-only vswitch via the 3rd vnet)
  • pri­mary domain uses 2 vnets con­nected to the vswitch which is con­nected to aggr2 and aggr3 (con­sis­tency with the other LDOMs on this machine) and has no zones
  • this means each entity (pri­mary domain, guest LDOMs and each zone) has two vnets in and those two vnets are con­fig­ured in a link-based IPMP setup (vnet-linkprop=phys-state)
  • each vnet has VLAN tag­ging con­fig­ured in the hyper­vi­sor (with the zones being in dif­fer­ent VLANs than the LDOMs)

The pro­posed change by Ora­cle is to replace the 2 vnet inter­faces in the pri­mary domain with 2 vsw inter­faces (which means to do VLAN tag­ging in the pri­mary domain directly instead of in the vnet con­fig). To have IPMP work­ing this means to have vsw-linkprop=phys-state. We have two sys­tems with the same setup, on one sys­tem we already changed this and it is work­ing as before. As we don’t know how to repro­duce the 1st prob­lem, we don’t know if the prob­lem is fixed or not, respec­tively what the prob­a­bil­ity is to get hit again by this problem.

Ideas / sug­ges­tions / info welcome.

GD Star Rat­ing
GD Star Rat­ing