FreeBSD is on its way to move from CVS to SVN  for the ver­sion con­trol sys­tem for the Ports Col­lec­tion. The deci­sion was made to keep the com­plete his­tory, so the com­plete CVS repos­i­tory has to be con­verted to SVN.

As CVS has no way to record a copy or move of files inside the repos­i­tory, we copied the CVS files inside the repos­i­tory in case we wanted to copy or move a file (the so called “repocopy”). While this allows to see the full his­tory of a file, the draw­back is that you do not really know when a file was copied/moved if you are not strict at record­ing this info after doing a copy. Guess what, we where not.

Now with the move to SVN which has a build-in way for copies/moves, it would be nice if we could record this info. In an inter­nal dis­cus­sion some­one told its not pos­si­ble to detect a repocopy reliably.

Well, I thought oth­er­wise and an hour later my mail went out how to detect one. The longest time was needed to write how to do it, not to come up with a solu­tion. I do not know if some­one picked up this algo­rithm and imple­mented some­thing for the cvs2svn con­verter, but I decided to pub­lish the algo­rithm here if some­one needs a sim­i­lar func­tion­al­ity some­where else. Note, the fol­low­ing is tai­lored to the struc­ture of the Ports Col­lec­tion. This allows to speed up some things (no need to do all steps on all files). If you want to use this in a generic repos­i­tory where the struc­ture is not as reg­u­lar as in our Ports Col­lec­tion, you have to run this algo­rithm on all files.

It also detects com­mits where mul­ti­ple files where com­mit­ted at once in one com­mit (sweep­ing commits).

Prepa­ra­tion

• check only category/name/Make­file
• gen­er­ate a hash of each commitlog+committer
• if you are memory-limited use ha/sh/ed/dirs/cvs-rev and store path­name in the list cvs-rev (path­name = “category-name”) as storage
• store the hash also in pathname/cvs-rev

If you have only one item in ha/sh/ed/dirs/cvs-rev in the end, there was no repocopy and no sweep­ing com­mit, you can delete this ha/sh/ed/dirs/cvs-rev.

If you have more than … let’s say … 10 (sub­ject to tun­ing) path­names in ha/sh/ed/dirs/cvs-rev you found a sweep­ing com­mit and you can delete the ha/sh/ed/dirs/cvs-rev.

The meat

The remain­ing ha/sh/ed/dirs/cvs-rev are prob­a­bly repocopies. Take one ha/sh/ed/dirs/cvs-rev and for each path­name (there may be more than 2 path­names) in there have a look at pathname/. Take the first cvs-rev of each and check if they have the same hash. Con­tinue with the next rev-number for each until you found a cvs-rev which does not con­tain the same hash. If the num­ber of cvs-revs since the begin­ning is >= … let’s say … 3 (sub­ject to tun­ing), you have a can­di­date for a repocopy. If it is >=  … 10 (sub­ject to tun­ing), you have a very good indi­ca­tor for a repocopy. You have to pro­ceed until you have only one path­name left.

You may detect mul­ti­ple repocopies like A->B->C->D or A->B + A->D + A->C here.

Write out the repocopy can­di­date to a list and delete the ha/sh/ed/dirs/cvs-rev for each cvs-rev in a detected sequence.

This finds repocopy can­di­dates for category/name/Makefile. To detect the cor­rect repocopy-date (there are maybe cases where another file was changed after the Make­file but before the repocopy), you now have to look at all the files for a given repocopy-pair and check if there is a match­ing com­mit after the Makefile-commit-date. If you want to be 100% sure, you com­pare the com­plete commit-history of all files for a given repocopy-pair.

In sev­eral pre­vi­ous posts I wrote about my quest for the right source for­mat to stream video to my Sony BRAVIA TV (build in 2009). The last week-end I finally found some­thing which sat­is­fies me.

What I found was serviio, a free UPnP-AV (DLNA) server. It is writ­ten in java and runs on Win­dows, Linux and FreeBSD (it is not listed on the web­site, but we have an not-so-up-to-date ver­sion in the ports tree). If nec­es­sary it transcodes the input to an appro­pri­ate for­mat for the DLNA ren­derer (in my case the TV).

I tested it with my slow Net­book, so that I was able to see with which input for­mat it will just remux the input con­tainer to a MPEG trans­port stream, and which input for­mat would be really re-encoded to a for­mat the TV understands.

The bot­tom line of the tests is, that I just need to use a sup­ported con­tainer (like MKV or MP4 or AVI) with H.264-encoded video (e.g. encoded by x264) and AC3 audio.

The TV is able to chose between sev­eral audio streams, but I have not tested if serviio is able to serve files with mul­ti­ple audio streams (my wife has a dif­fer­ent mother lan­guage than me, so it is inter­est­ing for us to have mul­ti­ple audio streams for a movie), and I do not know if DLNA sup­ports some­thing like this.

Now I just have to replace minidlna (which only works good with my TV for MP3s and Pic­tures) with serviio on my FreeBSD file server and we can for­get about the disk-juggling.

Michael W. Lucas pub­lished his new book “SSH Mas­tery” (no link to an online store, get it from your pre­ferred online or offline one in your part of the world).

Do you think you know a lot about SSH? I thought I did when Michael searched tech­ni­cal proof-readers for this book. I offered to have a look at his work in progress and he gen­tly accepted (while I do not get money for this, I am one of the per­sons he thanks for  the tech­ni­cal review in the begin­ning, so I am involved some­how and as such you should take the fol­low­ing with a grain of salt).

I already had user restric­tions in place before the review, but now I nar­rowed down some restric­tions based upon some con­di­tion­als. I already used SSH tun­nels for var­i­ous things before (where legally applic­a­ble), but I learned some addi­tional VPN tech­niques with SSH. I already used mul­ti­ple ssh-keys for var­i­ous things, but Michael pro­vides some inter­est­ing ways of han­dling a large-volume of ssh-keys over mul­ti­ple machines. … I really hope that my review was as valu­able for Michael, as it was for me to do the review.

He ends the book with “You now know more about SSH, OpenSSH and Putty than the vast major­ity of IT pro­fes­sion­als! Con­grat­u­la­tions”, and this is true, and all that in his writ­ing style where you can come with a prob­lem, read about it, and leave with a solu­tion (nor­mally with a lit­tle bit of enter­tain­ment in between).

I know a lot of peo­ple which work daily with SSH, and they know only a small part of what is pre­sented in this book. In my opin­ion this book is a must-have for every System/Database/Application/Whatever Admin­is­tra­tor in charge of some­thing on an UNIX-like sys­tem, and even “nor­mal users” of SSH (no mat­ter if they use PuTTY, or a ssh com­mand line pro­gram on an UNIX-like sys­tem (most prob­a­bly it will be OpenSSH or a clone of it)) will get some help­ful infor­ma­tion from this book.

I can only rec­om­mend it.

In the light of the recent bench­mark dis­cus­sion, a vol­un­teer imported the tun­ing man-page into the wiki. Some com­ments at some places for pos­si­ble improve­ments are already made. Please go over there, have a look, and par­tic­i­pate please (testing/verification/discussion/improvements/…).

As always, feel free to reg­is­ter with First­name­Last­name and tell a FreeBSD com­mit­ter to add you to the con­trib­u­tors group for write access (you also get the ben­e­fit to be able to reg­is­ter for an email noti­fi­ca­tion for spe­cific pages).

I have the habit to chmod with the rel­a­tive nota­tion (e.g. g+w or a+r or go-w or sim­i­lar) instead of the absolute one (e.g. 0640 or u=rw,g=r,o=). Recently I had to chmod a lot of files. As usual I was using the rel­a­tive nota­tion. With a lot of files, this took a lot of time. Time was not really an issue, so I did not stop it to restart with a bet­ter per­form­ing com­mand (e.g. find /path –type f –print0 | xargs –0 chmod 0644; find /path –type d –print0 | xargs –0 chmod 0755), but I thought a lit­tle tips&tricks post­ing may be in order, as not every­one knows the difference.

The rel­a­tive notation

When you spec­ify g+w, it means to remove the write access for the group, but keep every­thing else like it is. Nat­u­rally this means that chmod first has to lookup the cur­rent access rights. So for each async write request, there has to be a read-request first.

The absolute notation

The absolute nota­tion is what most peo­ple are used to (at least the numeric one). It does not need to read the access rights before chang­ing them, so there is less I/O to be done to get what you want. The draw­back is that it is not so nice for recur­sive changes. You do not want to have the x-bit for data files, but you need it for direc­to­ries. If you only have a tree with data files where you want to have an uni­form access, the exam­ple above via find is prob­a­bly faster (for sure if the direc­tory meta-data is still in RAM).

If you have a mix of bina­ries and data, it is a lit­tle bit more tricky to come up with a way which is faster. If the data has a name-pattern, you could use it in the find.

And if you have a non-uniform access for the group bits and want to make sure the owner has write access to every­thing, it may be faster to use the rel­a­tive nota­tion than to find a replace­ment command-sequence with the absolute notation.