Does every geek take a photo of Google in Zürich?

Maybe not all… my col­leagues wouldn’t have gone there if I wouldn’t have said some­thing…

To be hon­est, I would have ex­pec­ted some­thing a little bit more shiny/​geeky/​… to take a pic­ture of (any­way, if you don’t see it: I have a real smile in my face on the pic­ture, so the geek in me en­joyed tak­ing the photo).


Alexander Leidinger in front of Google CH
Me in front of Google CH


Google CH
Google CH


Com­plete net­work loss on Sol­ar­is 10u10 CPU 2012-​10 on vir­tu­al­ized T4-​2

The prob­lem I see at work: A T4-​2 with 3 guest LDOMs, vir­tu­al­ized disks and net­works lost the com­plete net­work con­nectiv­ity “out of the blue” once, and maybe “sporad­ic” dir­ectly after a cold boot. After a lot of dis­cus­sion with Or­acle, I have the im­pres­sion that we have two prob­lems here.

1st prob­lem:
Total net­work loss of the ma­chine (no zone or guest LDOM or the primary LDOM was able to have re­ceive or send IP pack­ets). This happened once. No idea how to re­pro­duce it. In the logs we see the mes­sage “[ID 920994 kern.warning] WARNING: vnetX: ex­ceeded num­ber of per­mit­ted hand­shake at­tempts (5) on chan­nel xxx”. Ac­cord­ing to Or­acle this is sup­posed to be fixed in 148677 – 01 which will come with Sol­ar­is 10u11. They sug­ges­ted to use a vsw in­ter­face in­stead of a vnet in­ter­face on the primary do­main to at least lower the prob­ab­il­ity of this prob­lem hit­ting us. They were not able to tell us how to re­pro­duce the prob­lem (seems to be a race con­di­tion, at least I get this im­pres­sion based upon the de­scrip­tion of the Or­acle en­gin­eer hand­ling the SR). Only a re­boot helped to get the prob­lem solved. I was told we are the only cli­ent which re­por­ted this kind of prob­lem, the patch for this prob­lem is based upon an in­tern­al bu­gre­port from in­tern­al tests.

2nd prob­lem:
After cold boots some­times some ma­chines (not all) are not able to con­nect to an IP on the T4. A re­boot helps, as does re­mov­ing an in­ter­face from an ag­greg­ate and dir­ectly adding it again (see be­low for the sys­tem con­fig). To try to re­pro­duce the prob­lem, we did a lot of warm re­boots of the primary do­main, and the prob­lem nev­er showed up. We did some cold re­boots, and the prob­lem showed up once.

In case someone else sees one of those prob­lems on his ma­chines too, please get in con­tact with me to see what we have in com­mon to try to track this down fur­ther and to share info which may help in maybe re­pro­du­cing the prob­lems.

Sys­tem setup:

  • T4-​2 with 4 HBAs and 8 NICs (4 * igb on-​board, 4 * nxge on ad­di­tion­al net­work card)
  • 3 guest LDOMs and one io+control do­main (both in the primary do­main)
  • the guest LDOMs use SAN disks over the 4 HBAs
  • the primary do­main uses a mirrored zpool on SSDs
  • 5 vswitch in the hy­per­visor
  • 4 ag­greg­ates (aggr1 – aggr4 with L2-​policy), each one with one igb and one nxge NIC
  • each ag­greg­ate is con­nec­ted to a sep­ar­ate vswitch (the 5th vswitch is for machine-​internal com­mu­nic­a­tion)
  • each guest LDOM has three vnets, each vnets con­nec­ted to a vswitch (1 guest LDOM has aggr1+2 only for zones (via vnets), 2 guest LDOMs have ag­gr 3+4 only for zones (via vnets), and all LDOMs have aggr2+3 (via vnets) for global-​zone com­mu­nic­a­tion, all LDOMs are ad­di­tion­ally con­nec­ted to the machine-​internal-​only vswitch via the 3rd vnet)
  • primary do­main uses 2 vnets con­nec­ted to the vswitch which is con­nec­ted to aggr2 and aggr3 (con­sist­ency with the oth­er LDOMs on this ma­chine) and has no zones
  • this means each en­tity (primary do­main, guest LDOMs and each zone) has two vnets in and those two vnets are con­figured in a link-​based IPMP setup (vnet-linkprop=phys-state)
  • each vnet has VLAN tag­ging con­figured in the hy­per­visor (with the zones be­ing in dif­fer­ent VLANs than the LDOMs)

The pro­posed change by Or­acle is to re­place the 2 vnet in­ter­faces in the primary do­main with 2 vsw in­ter­faces (which means to do VLAN tag­ging in the primary do­main dir­ectly in­stead of in the vnet con­fig). To have IPMP work­ing this means to have vsw-linkprop=phys-state. We have two sys­tems with the same setup, on one sys­tem we already changed this and it is work­ing as be­fore. As we don’t know how to re­pro­duce the 1st prob­lem, we don’t know if the prob­lem is fixed or not, re­spect­ively what the prob­ab­il­ity is to get hit again by this prob­lem.

Ideas /​ sug­ges­tions /​ info wel­come.

Which crypto card to use with FreeBSD (ssh/​gpg)

The re­cent se­cur­ity in­cid­ent triggered a dis­cus­sion how to se­cure ssh/​gpg keys.

One way I want to fo­cus on here (be­cause it is the way I want to use at home), is to store the keys on a crypto card. I did some re­search for suit­able crypto cards and found one which is called Fei­tian PKI Smart­card, and one which is called Open­P­GP card. The Open­P­GP card also ex­ists in a USB ver­sion (ba­sic­ally a small ver­sion of the card is already in­teg­rated in­to a small USB card read­er).

The Fei­tian card is re­por­ted to be able to handle RSA keys upto 2048 bits. They do not seem to handle DSA (or ECDSA) keys. The smart­card quick starter guide they have  (the Tun­ing smart­card file sys­tem part) tells how to change the para­met­ers of the card to store upto 9 keys on it.

The spec of the Open­P­GP card tells that it sup­ports RSA keys upto 3072 bits, but there are re­ports that it is able to handle RSA keys upto 4096 bits (you need to have at least GPG 2.0.18 to handle that big keys on the crypto card). It looks to me like the card is not handle DSA (or ECDSA) cards. There are only slots for upto 3 keys on it.

If I go this way, I would also need a card read­er. It seems a class 3 one (hard­ware PIN pad and dis­play) would be the most “future-​proof” way to go ahead. I found a Rein­er SCT cy­ber­Jack secoder card read­er, which is be­lieved to be sup­por­ted by OpenSC and seems to be a good bal­ance between cost and fea­tures of the Rein­er SCT card read­ers.

If any­one read­ing this can sug­gest a bet­ter crypto card (keys upto 4096 bits, more than 3 slots, and/​or DSA/​ECDSA  sup­port), or a bet­ter card read­er, or has any prac­tic­al ex­per­i­ence with any of those com­pon­ents on FreeBSD, please add a com­ment.

For­cing a route in Sol­ar­is?

I have a little prob­lem find­ing a clean solu­tion to the fol­low­ing prob­lem.

A ma­chine with two net­work in­ter­faces and no de­fault route. The first in­ter­face gets an IP at boot time and the cor­res­pond­ing stat­ic route is in­ser­ted dur­ing boot in­to the rout­ing table without prob­lems. The second in­ter­face only gets an IP ad­dress when the shared-​IP zones on the ma­chine are star­ted, dur­ing boot the in­ter­face is plumbed but without any ad­dress. The net­works on those in­ter­faces are not con­nec­ted and the ma­chine is not a gate­way (this means we have a machine-​administration net­work and a production-​network). The stat­ic routes we want to have for the ad­dresses of the zones are not ad­ded to the rout­ing table, be­cause the next hop is not reach­able at the time the routing-​setup is done. As soon as the zones are up (and the in­ter­face gets an IP), a re-​run of the routing-​setup adds the miss­ing stat­ic routes.

Un­for­tu­nately I can not tell Sol­ar­is to keep the stat­ic route even if the next hop is not reach­able ATM (at least I have not found an op­tion to the route com­mand which does this).

One solu­tion to this prob­lem would be to add an ad­dress at boot to the in­ter­face which does not have an ad­dress at boot-​time ATM (prob­ably with the de­prec­ated flag set). The prob­lem is, that this sub­net (/​28) has not enough free ad­dresses any­more, so this is not an op­tion.

An­oth­er solu­tion is to use a script which re-​runs the routing-​setup after the zones are star­ted. This is a prag­mat­ic solu­tion, but not a clean solu­tion.

As I un­der­stand the in.routed man-​page in.routed is not an op­tion with the de­fault con­fig, be­cause the ma­chine shall not route between the net­works, and shall not change the rout­ing based upon RIP mes­sages from oth­er ma­chines. Un­for­tu­nately I do not know enough about it to be sure, and I do not get the time to play around with this. I have seen some in­ter­st­ing op­tions re­gard­ing this in the man-​page, but play­ing around with this and sniff­ing the net­work to see what hap­pens, is not an op­tion ATM. Any­one with a config/​tutorial for this “do not broad­cast any­thing, do not ac­cept any­thing from outside”-case (if pos­sible)?

Why are game console/​TV com­pan­ies not im­ple­ment­ing this?

At the week­end a friend vis­ited me. We have not seen since each oth­er since a long time. As we stud­ied both com­puter sci­ence, parts of our dis­cus­sion where off course tech­no­logy re­lated. Parts of the dis­cus­sion where about cur­rent TV’s and game con­soles (he par­ti­cip­ated in the design of the PS3 CPU, so he is well aware about the tech­nic­al lim­it­a­tions of the hard­ware the cur­rent game con­soles use).

Dur­ing our dis­cus­sion we talked about the soft­ware lim­it­a­tions of such hard­ware.

Cur­rent TV’s come for ex­ample with some pre­defined in­ter­net chan­nels, but not with a real web browser. We think that people which keep a TV for 10 years or longer (like for ex­ample our par­ents and prob­ably both of us too) this will res­ult in a loss of fea­tures after some years, be­cause those chan­nels will get less at­ten­tion of case to ex­ist at all. There is also no way to switch to al­tern­at­ives then, ex­cept by buy­ing a new TV (we ex­pect that there will be no firm­ware up­date in such a case). With a real web browser this would not be an is­sue (it may be more easy to enter URL’s with a real key­board than with a re­mote con­trol, but let us do small steps here). Game con­soles are a bit bet­ter in this re­gard, but there we have the prob­lem that some web­sites are too much memory hungry (they do not in­clude the user agent of the game con­sole browsers in the same class as smart phones or tab­let PCs… from the size as­pect they are not, but from the memory and com­put­ing power as­pect they are more sim­il­ar).

I would ex­pect that the TV sta­tions do not want to have TVs with really good browsers, be­cause then you may not need a TV sta­tion any­more. But this is what users would use if it would be there.

An­oth­er de­fi­cit is that there is not a mail pro­gram in game con­soles and TV’s. For writ­ing mails you need a real key­board, but for a quick check if there is mail (e.g. X un­read mails, or maybe even dis­play­ing the sub­ject line of the emails) or maybe to just read without an­swer­ing a solu­tion without a key­board con­nec­ted would already be enough.

I ex­pect that con­sole man­u­fac­tur­ers do not want to spend money for some­thing people are not will­ing to give much money for, re­spect­ively for some­thing where they can not make money with (an email ser­vice from the con­sole com­pany would be an­oth­er mail ser­vice ad­di­tion­al to the one for the PC and maybe ad­di­tion­al to the one of the smart phone… people do not need 10 email ac­counts, one is enough).

An­oth­er over­looked fea­ture is some kind of VoIP+Video fea­ture (at least for the game con­soles which have op­tion­ally a cam­era, but IMO this is also pos­sible for the next gen­er­a­tion of TV’s with build-​in web­cams). At least the of­fer­ings from Sony and Mi­crosoft are power­ful enough to come with some kind of video con­fer­en­cing soft­ware. It does not mat­ter much if this is Skype or the Google ver­sion of this, or some oth­er wide­spread one (MS surely wants to use their own stuff), it just has to be one which is in wide­spread use to be ad­op­ted by the people.This does not need to be in HD, even a small video would already be much more than what is avail­able ATM.

Ba­sic­ally I gave the an­swer to my ques­tion (the title of this post­ing) my­self (ex­cept for the video con­fer­en­cing stuff)… but on the oth­er hand this would be some­thing which could set a product apart from oth­ers. For the PS3 this may be now one of the things which could show up in the Homebrew scene, now that the se­cur­ity of the PS3 is com­prom­ised. For the Wii at least the email part could be eas­ily done. The rest… would have to catch up in case some­thing like this shows up for the PS3 and is used ex­tens­ively.

Fight­ing with the Or­acle Dir­ect­ory Serv­er 7 (DSEE7) on Sol­ar­is 10 up­date 9

After mov­ing our sec­ond­ary man­age­ment site (our team is split up in­to 2 dif­fer­ent loc­a­tions) to a new build­ing, we de­cided to clean-​up some things. One of those things in­volves mov­ing the LDAP to a dif­fer­ent ma­chine (more or less a new serv­er for the new site, it is in­de­pend­ent re­gard­ing LDAP/​homes/​… from the primary site). While I am at it, I take the op­por­tun­ity to move from DSEE5 to DSEE7 (my pre­vi­ous post about the DSEE6 mi­gra­tion was at the primary site). This time I took the pack­age dis­tri­bu­tion in­stead of the zip dis­tri­bu­tion (the main reas­on is that I can get patch-​listings with an auto­mat­ic tool, and the sec­ond­ary man­age­ment site has no disaster-​recovery re­quire­ments for the ap­plic­a­tions… we just will setup a new sec­ond­ary site some­where else if ne­ces­sary).

Here my ex­per­i­ences with the in­stall­a­tion in­struc­tions of DSEE7.

  • The in­stall in­struc­tions refer to the web in­ter­face for the DSEE7 man­age­ment, but I have not seen some­thing which tells you first have to setup an ap­plic­a­tion serv­er (this was bet­ter in the DSEE6 in­struc­tions).
  • When us­ing the Glassfish ap­plic­a­tion serv­er which comes with Sol­ar­is 10 for the web in­ter­face, you will get an ex­cep­tion after de­ploy­ing the dscc7.war, as it is us­ing an out­dated JVM. After some fight­ing and Googling, I found that I have to change the AS_​JAVA value in /​usr/​appserv­er/​con­fig/asenv.conf to a more re­cent JVM as it is point­ing to the very out­dated j2se 1.4.x. I poin­ted it to /​usr/​java (which is a sym­link to the most re­cent ver­sion in­stalled as a pack­age). In­stead of the ori­gin­al ex­cep­tion I got an­oth­er one now (after a re­dir­ec­tion in the web–browser), some­thing that it can not find the Ant­Main class (Glassfish uses ANT from /​usr/​sfw, this is the one which comes with Sol­ar­is 10 up­date 9). I tried with Java 5 in­stead of Java 6, but I get the same er­ror. In the net there are some dis­cus­sions about such er­rors (it is even a FAQ at the ANT site), but this Glassfish/​DSEE7 thing is a black box for me, so what am I sup­posed to do here (I do not want to put the sys­tem in­to an un­of­fi­cial state by in­stalling my own ANT for Glassfish/​DSEE7)?
    It was not men­tioned in the Ap­pendix of the DSEE7 in­stall in­struc­tions which ex­plains how to in­stall the .war in Glassfish that you have to change to a more re­cent JVM, and I still fight with the Ant­Main prob­lem (hey Or­acle, there is room for im­prove­ment in the product com­pat­ib­il­ity test­ing and doc­u­ment­a­tion veri­fic­a­tion pro­cess).

I will up­date this post­ing when I make some ad­vance­ments. For now I let the web in­ter­face in the bad state as it is and con­cen­trate on fin­ish­ing the LDAP move to the new sys­tem (in­stalling an DSEE on a backup sys­tem, con­fig­ur­ing rep­lic­a­tion, switch­ing the cli­ents to them). The web in­ter­face is in­de­pend­ent enough to handle it later (hints wel­come, that is the main pur­pose why I write this pos­ing in the middle of the work).

One-​Time-​Passwords for Horde/​IMP?

I search a way to use one-​time–pass­words for Horde/​IMP on FreeBSD. I do not want to use PAM (loc­al users on the ma­chine). Cur­rently I use the au­then­tic­a­tion via IMAP4 (link between the IMAP4-​server and post­fix via MySQL, to have the same PW for send­ing and re­ceiv­ing), and I ex­pect that not all users of Horde/​IMP will use OTP if avail­able, so the prob­lem case is not that easy. I can ima­gine a solu­tion which tries to au­then­tic­ate via OTP first, and if it suc­ceeds gets a pass­word for the lo­gin to the IMAP4 serv­er. If the OTP-​auth fails, it could try the entered pass­word for the lo­gin to the IMAP4 serv­er. Mi­grat­ing ex­ist­ing users to a new solu­tion can be done by telling them to enter the pass­word from the ma­chine of the per­son do­ing the mi­gra­tion. The solu­tion needs to auto­mat­ic­ally lo­gin to the IMAP4 serv­er, en­ter­ing a pass­word for the IMAP4 serv­er after the OTP-​login to Horde is not an op­tion.

Oh, yes, send­ing the pass­words over SSL is not an op­tion (that is already the only way to lo­gin there). The goals are to have

  • an easy to re­mem­ber pass­word for an OTP app on the mo­bile to gen­er­ate the real pass­word
  • the pass­word ex­pire fast, so that a stolen pass­word does not cause much harm
  • not the same login-​password for dif­fer­ent ser­vices (mail-​pw != jabber-​pw != user-​pw)