Month: August 2009

EMC^2/Legato Net­work­er 7.5.1.4 status

The update to 7.5.1.4 went fine. No major prob­lems encoun­tered. So far we did not see any regres­sions. The com­plete sys­tem feels a lit­tle bit more sta­ble (no restarts nec­es­sary so far, before some where nec­es­sary from time to time). We still have to test all our prob­lem cases:

  • restart NW-server direct­ly after delet­ing a client with index entries (man­u­al copy of /nsr need­ed before, in case the medi­adb cor­rup­tion bug is not fixed as promised)
  • shut­down a stor­age node to test if the NW-server still crash­es in this case
  • start with an emp­ty medi­adb but pop­u­lat­ed clients (emp­ty /nsr/mm, but untouched /nsr/res) and scan some tapes to check if “shad­ow clients” (my term for clients which have the same client ID but get new­ly cre­at­ed dur­ing the scan­ning with a new client ID and a name of “~<original-name>-<number>”) still get cre­at­ed instead of pop­u­lat­ing the index of the cor­rect client

The first two ones are sup­posed to be fixed, the last one is maybe not fixed.

Not fixed (accord­ing to the sup­port) is the prob­lem of need­ing a restart of the NW-server when mov­ing a tape library from one stor­age node to anoth­er stor­age node. It also seems that our prob­lem with the man­u­al cloning of save sets is not solved. There are still some clone process­es which do not get out of the “serv­er busy” loop, no mat­ter how idle the NW-server is. In this case it can be seen that nsr­clone is wait­ing in nanosleep (use pstack or dtrace to see it). The strange thing is, that a safe set which is “fail­ing” with such behav­ior will always cause this behav­ior. We need to have a deep­er look to see if we find sim­i­lar­i­ties between such safe sets and dif­fer­ences to safe sets which can be cloned with­out problems.

Share/Save

Dai­ly doxy­gen gen­er­at­ed docs of the FreeB­SD ker­nel (head)

I man­aged to get some time to set­up an auto­mat­ed gen­er­a­tion of the doxy­gen docs for ker­nel sub­sys­tems of FreeB­SD on my webserver.

Every night/morning (Ger­man time­zone) the sources will be updat­ed, and the docs get regen­er­at­ed (this takes some time). Cur­rent­ly this depends upon some patch­es to the make­file and doxy­gen con­fig files in tools/kerneldoc/subsys. Every­thing is gen­er­at­ed direct­ly in the place where the web­serv­er will look for to deliv­er the pages, so if you browse this in the mid­dle of the gen­er­a­tion, the con­tent may not be con­sis­tent (yet).

Please be nice to the web­serv­er and do not mir­ror this. You can gen­er­ate this your­self very easy. Assum­ing you have the FreeB­SD source on a local hard disk, you just need to down­load the patch from http://www.Leidinger.net/FreeBSD/current-patches/ (if you do not find dox.diff, update your FreeB­SD sources and every­thing will be OK), apply the patch, cd into tools/kerneldoc/subsys and run “make all” (or “make vm” or what­ev­er you are inter­est­ed in). You need doxy­gen installed, off course.

If you want to set­up some­thing like this your­self, just down­load the script which is doing all the work, change some vari­ables in the begin­ning, and cre­ate your own local ver­sion of the com­plete docs.

In case this is using sig­nif­i­cant traf­fic, I will ask core/admins if there is the pos­si­bil­i­ty to host it on FreeBSD.org resources.

Share/Save

EMC^2/Legato Net­work­er 7.5.1 problems

In July a col­league switched our back­up sys­tem from a Net­work­er (NW) serv­er 7.1.1 with a sin­gle phys­i­cal tape library to a NW serv­er 7.5.1 on a Solaris Clus­ter plus 2 stor­age nodes with 2 vir­tu­al tape libraries and 2 phys­i­cal tape libraries. He spe­cial­ly decid­ed to install 7.5.1 instead of 7.5 (going with 7.4 was not an option he had because of some require­ments for the Share­point back­up for our Win­dows peo­ple) in the begin­ning of the project sev­er­al months ago, as he did not trust a “.0‑release”. Every­thing went well until we crossed the point of no return. Then we got a cor­rupt media DB (meta data which tells which back­up for which client is on which tape) just 3 days before he went into hol­i­day. Yeah! 🙁

As I was agree­ing to be his back­up and pol­ish the NW set­up dur­ing his hol­i­day, this result­ed in a lot of over­time for me to get every­thing back into shape. The bad part of this was, that the inci­dent opened with the NW-support took about one a half day until it reached a point where some­one was look­ing at try­ing to repro­duce the core dumps we where get­ting. Until then the stan­dard trial&error prob­lem fix­ing pro­ce­dure was done my the over­seas sup­port peo­ple (we had one sup­port guy on-site, and he did real­ly good sup­port, and he was aston­ished about this behav­ior too). And this despite the fact the the help-request was on pri­or­i­ty one. We also had clear­ly stat­ed sev­er­al times in the tick­et, that our media DB was cor­rupt (with an emp­ty media DB every­thing was work­ing, just with the pop­u­lat­ed media DB the serv­er crashed on start­up). The prob­lem was, that you are not allowed to delete a client which has entries in the media DB, with­out run­ning “nsrim ‑X” by hand or tak­ing a back­up of the boot­strap (this runs “nsrim ‑X”). This is sup­posed to be fixed in 7.5.1.4 (released because of our prob­lem, it seems). The man­ag­er from EMC^2 which came because of this was a poor guy. Imag­ine him being there to rep­re­sent EMC^2, me and my boss to rep­re­sent the on-site admins of our client, one of  the Win­dows peo­ple of our client, and 5 peo­ple from our client in the same room (this means 1 vs. 7)… and our client was not happy.

As we did not had a work­ing boot­strap back­up (do not ask, a mix of bad luck and strange behav­ior of NW), we had to do a dis­as­ter recov­ery (this means scan­ning all tapes with back­ups we did so far). The bad part is, this will result in some clients which are sup­posed to have the same client ID to show up with a dif­fer­ent client ID. In plain eng­lish this means, the dis­as­ter recov­ery pro­ce­dure does not real­ly work as expect­ed. Yes, we still can recov­er the data, but it is not just start­ing the recov­er pro­gram and mak­ing your choice what to restore. You have to first find the data in the media DB (with the mmin­fo tool), print out the new client ID, cre­ate an entry in /etc/hosts on the NW-server with a dum­my name (I decid­ed to give them the IP 127.0.1.X, in case we need it), cre­ate a client in NW with the same dum­my name (e.g. z‑recover-<client-name>) and then to recov­er your data nor­mal­ly. This is a bug for which we do not know yet if it is fixed in 7.5.1.4 or not. I can tell that this pro­ce­dure works, we had to recov­er some data just after we where in a state where it was pos­si­ble to recov­er at least some data (set­up was OK, but not all tapes where scanned).

Addi­tion­al­ly to the oth­er bugs we stum­bled upon (e.g. mov­ing a VTL to the oth­er stor­age node needs a restart of the NW-server, NW will moan about the license not being OK and not giv­ing access to the VTL, if you do not restart; if the cloning con­tacts the media DB at the wrong time (when it is busy), the cloning will nev­er start and stay in “serv­er busy”-mode, even if it is not busy any­more; the postcmd is run­ning one minute after the precmd, it is not detect­ing that a back­up is still run­ning; …) I have to say that NW 7.5.1 is not real­ly pro­duc­tion ready. For me NW 7.5.1 is beta qual­i­ty soft­ware. Dur­ing the time when I took care about fix­ing the prob­lems with our back­up sys­tem, the one thing I told most often to oth­er peo­ple when they asked if I think the back­up will be OK in the night was: please cross your fingers.

My col­league is back from hol­i­day, and on Wednes­day he is going to install 7.5.1.4. It is sup­posed to fix most of our prob­lems. I cross my fingers.

I can also report some good things about NW. The GUI is much bet­ter than the one from 7.1.1. And when 7.5.x will be as sta­ble as our 7.1.1 ver­sion was (there where some prob­lems from time to time, but the workaround was to restart the serv­er and it was OK, as this did not hap­pen often, it was OK for us), it will be real­ly a nice back­up sys­tem which we have here. And boy, this thing is fast (OK, a part of this is because of the VTL instead of a phys­i­cal tape library, but NW is now also able to read from sev­er­al tapes in par­al­lel for the same restore – if the data is on mul­ti­ple tapes, off course).

Bot­tom line: if you use 7.5.x, update to 7.5.1.4 as soon as possible.

Share/Save

My own XMPP/Jabber server

In the last week I took some time to set­up my own jab­ber server.

I decid­ed to have a look at ejab­berd. Seems to be nice. The ini­tial con­fig  was done with­out a prob­lem. Then I tried to enable some addi­tion­al stuff and then the docs start­ed to be not clear enough. It seems you have to con­fig­ure addi­tion­al host names if you want to add e.g. the echo ser­vice and an ICQ trans­port. I tried first with just the name of the sys­tem, but this did not work out as expect­ed. Now I have sev­er­al DNS entries to the same sys­tem, just to be able to run a XMPP serv­er with some addi­tion­al fea­tures . The default set­tings of the trans­ports are also to lis­ten on local­host, instead of a real inter­face of the machine.

Not real­ly user­friend­ly, and the docs do not explic­it­ly tell that you have to have those addi­tion­al entries (it is some­how told implic­it­ly, but if you go this way for the first time, it may not be obvious).

To con­fig­ure access via an web fron­tend (via the http_bind/bochs exten­sion), the docs are also not very clear. It is easy to over­look that you have to make a change in the lis­ten part and in the mod­ules part. After get­ting every­thing right, all seems to work good. I already have sev­er­al users and it seems that they are sat­is­fied. The only “bad” thing is that pid­gin does not seem to give a nice UI to add your ICQ (or what­ev­er trans­port you have con­fig­ured addi­tion­al­ly) account. While pid­gin can do ICQ, peo­ple may use the webin­ter­face at some loca­tions, so it would be nice if pid­gin would sup­port this better.

Share/Save

WP prob­lems solved

I solved my WP problems.

The perma­link prob­lem was because of a miss­ing .htac­cess file. I cre­at­ed one with write per­mis­sions for WP, and changed the permal­imk set­ting back.

The white-on-white issue was solved by set­ting CONCATENATE_SCRIPTS to false in wp-config, so it seems to be a caching prob­lem some­where. Lat­er I will take some time to switch back and flush some caches (at least where I have the rights to do it).

Share/Save