Cheap process mon­i­tor­ing (no addi­tion­al soft­ware required)

I have an old sys­tem (only the hard­ware, it runs ‑cur­rent) which reboots itself from time to time (most­ly dur­ing the dai­ly periodic(8) run, but also dur­ing a lot of com­pil­ing (por­tup­grade)). There is no obvi­ous rea­son (no pan­ic) why it is doing this. It could be that there is some hard­ware defect, or some­thing else. It is not impor­tant enough to get a high enough pri­or­i­ty that I try hard to ana­lyze the prob­lem with this machine. The annoy­ing part is, that some­times after a restart apache does not start. So if this hap­pens, the solu­tion is to login and start the web­serv­er. If the web­serv­er would start each time, near­ly nobody would detect the reboot (root gets an EMail on each reboot via an @reboot crontab entry).

My prag­mat­ic solu­tion (for ser­vices start­ed via a good rc.d script which has a work­ing sta­tus com­mand) is a crontab entry which checks peri­od­i­cal­ly if it is run­ning and which restarts the ser­vice if not. As an exam­ple for apache and an inter­val of 10 minutes:

*/10 * * * *    /usr/local/etc/rc.d/apache22 status >/dev/null 2>&1 || /usr/local/etc/rc.d/apache22 restart

For the use case of this service/machine, this is enough. In case of a prob­lem with the ser­vice, a mail with the restart out­put would arrive each time it runs, else only after a reboot for which the ser­vice did not restart.

Inter­est­ing projects in the GSoC

I count­ed 18 projects which are giv­en to FreeB­SD in this years GSoC. For 3 of them I have some comments.

Very inter­est­ing to me is the project which is named Col­lec­tive lim­its on set of process­es (a.k.a. jobs). This looks a bit like the Solaris contract/project IDs. If this project results in some­thing which allows the user­land to query which PID belongs to which set, than this allows some nice improve­ment for start scripts. For exam­ple at work on Solaris each appli­ca­tion is a mix of sev­er­al projects (apache = “name:web” project, tom­cat = “name:app” project, Ora­cle DB = “name:ora” project). Our man­age­ment frame­work (writ­ten by a co-worker) allows to eas­i­ly do some­thing with those projects, a “show” dis­plays the prstat (sim­i­lar to top) info just for process­es which belong to the project, a “kill” sends a kill-signal to all process­es of the project, and so on. We could do some­thing sim­i­lar with our start scripts by declar­ing a name­space (FreeBSD:base:XXX / FreeBSD:ports:XXX?) and maybe num­ber space (depend­ing on the imple­men­ta­tion) as reserved and use it to see if process­es which belong to a par­tic­u­lar script are still run­ning or kill them or whatever.

The oth­er two projects I want to com­ment upon here are Com­plete libp­kg and cre­ate new pkg tools and Com­plete Pack­age sup­port in the pkg_install tools and cleanup. Both projects ref­er­ence libp­kg in their descrip­tion. I hope the men­tors of both projects pay some atten­tion to what is going on in the oth­er project to not cause dependencies/clashes between the students.

That I do not men­tion oth­er projects does not mean that they are not inter­est­ing or sim­i­lar, it is just that I do not have to say some­thing valu­able about them…

HOWTO men­tor in the GSoC (ini­tial com­mu­ni­ca­tion with the student)

Every men­tor in the GSoC has a dif­fer­ent way of han­dling stu­dents. Here is what I do.

The stu­dent intro­duced him­self to me as request­ed by our soc-admins in the ini­tial mail to our stu­dents. He looked up in which time­zone I am (pub­lic info) and pre­sent­ed his time­zone (and rough loca­tion) to me. That is nice. He also offered dif­fer­ent com­mu­ni­ca­tion chan­nels (basi­cal­ly EMail and IM).

I con­firmed what he looked up, and pre­sent­ed what I did in the past GSoC in which I par­tic­i­pat­ed so that he has an idea if am new to the game or not. I told him that quick/short ques­tions are bet­ter asked via IM, while long expla­na­tions or ques­tions are bet­ter han­dled via EMail. I also gave him a rough overview when he can expect quick answers from me and when I am not available.

Fol­low­ing are some ques­tions I asked him, so that I get an impres­sion about what to expect and that I can plan a bit (some of those may already be told in stu­dent appli­ca­tion, but I pre­fer to have every­thing in one place):

  • From when to when do you intent to spend how much time for the GSoC?
  • Any hol­i­days / non-availability planned dur­ing the GSoC?
  • Any university-stuff (exams/lessons/…) dur­ing this time (the uni has high­er pri­or­i­ty than the GSoC for Google)?
  • Any­thing else in par­al­lel of the GSoC (some paid work, tak­ing care about ill (grand-)parents, …)?
  • At what lev­el of knowl­edge do you see your­self regard­ing computer-science/programming/OS-concepts (rel­a­tive to oth­er stu­dents and rel­a­tive to the topic)?
  • How do you want to start about the project (where do you want to start, what do you intent to do… just a quick overview… a bit more than say­ing “I add X”, but not as far as copy&paste of code examples)?

More impor­tant than that (IMO), is to give an idea what is expect­ed from the student:

  • you have FreeBSD-current installed (on a real PC or in a vir­tu­al machine)
  • you give me a report about the sta­tus each week (“did noth­ing” is also a valid report, it gives me the info that you are still alive and did not lose inter­est in the GSoC)
  • if your sched­ule changes in a sig­nif­i­cant way, give me a lit­tle noti­fi­ca­tion (e.g. “I can not do any­thing next week”)
  • if you spend more than 30 min­utes with a prob­lem, pre­pare an email with the prob­lem descrip­tion; if this prepa­ra­tion did not solve your prob­lem, send me the mail (if you solve the prob­lem 5 min­utes lat­er, no prob­lem, I pre­fer to get a mail too much than to have you stuck with some­thing for an incred­i­ble amount of time)

A men­tor does not know every­thing, off course, so the stu­dent should be sub­scribed to hackers@ and current@, and if there is a spe­cif­ic list which match­es good to the project he is work­ing on, then to this mail­ing list too. This allows the men­tor to tell the stu­dent to send a mail with the ques­tions to one of those lists with­out much prepa­ra­tion to receive all answers.

Anoth­er help­ful resource is the FreeB­SD ker­nel cross-reference. For some peo­ple my doxy­gen gen­er­at­ed docs of parts of the FreeB­SD ker­nel may be help­ful (put unfor­tu­nate­ly not a lot of doxygen-markup is with­in our source code).

I also told that he shall pre­pare him­self that I will ask him to send a ref­er­ence to a patch of his work long enough before the GSoC ends to an appro­pri­ate mail­ing list, and that com­ments from there regard­ing changes he must or shall do are not some­thing bad, but a way to improve the result and/or his skills.

Men­tor­ing again in the GSoC

Seems that I will active­ly men­tor again in this Google Sum­mer of Code (as opposed to just review the sub­mis­sions from stu­dents and/or act­ing as a fall-back mentor).

The project I will men­tor is the “Make option­al ker­nel sub­sys­tems reg­is­ter them­selves via sysctl”-one from the FreeB­SD ideas page.

The stu­dent already got into con­tact with me and it looks like he is moti­vat­ed (he is already sub­scribed to sev­er­al FreeB­SD mail­inglists, which is not a require­ment we have in our GSoC docs).

One-Time-Passwords for Horde/IMP?

I search a way to use one-time-passwords for Horde/IMP on FreeB­SD. I do not want to use PAM (local users on the machine). Cur­rent­ly I use the authen­ti­ca­tion via IMAP4 (link between the IMAP4-server and post­fix via MySQL, to have the same PW for send­ing and receiv­ing), and I expect that not all users of Horde/IMP will use OTP if avail­able, so the prob­lem case is not that easy. I can imag­ine a solu­tion which tries to authen­ti­cate via OTP first, and if it suc­ceeds gets a pass­word for the login to the IMAP4 serv­er. If the OTP-auth fails, it could try the entered pass­word for the login to the IMAP4 serv­er. Migrat­ing exist­ing users to a new solu­tion can be done by telling them to enter the pass­word from the machine of the per­son doing the migra­tion. The solu­tion needs to auto­mat­i­cal­ly login to the IMAP4 serv­er, enter­ing a pass­word for the IMAP4 serv­er after the OTP-login to Horde is not an option.

Oh, yes, send­ing the pass­words over SSL is not an option (that is already the only way to login there). The goals are to have

  • an easy to remem­ber pass­word for an OTP app on the mobile to gen­er­ate the real password
  • the pass­word expire fast, so that a stolen pass­word does not cause much harm
  • not the same login-password for dif­fer­ent ser­vices (mail-pw != jabber-pw != user-pw)