Alexander Leidinger

Just another weblog

Jun
23

Lin­ux­u­la­tor explained: How to cre­ate Linux bina­ries on FreeBSD

There may by cases where you want to gen­er­ate a Linux binary on a FreeBSD machine. This is not a prob­lem with the lin­ux­u­la­tor, but not with the default linux_base port.

As you may know, the linux_base port is designed to deliver an inte­grated expe­ri­ence with FreeBSD native pro­grams. As such some parts of the native FreeBSD infra­struc­ture is used. If you would try to use a Linux–com­piler to gen­er­ate Linux–bina­ries, you would run into the prob­lem that by default the FreeBSD includes are used.

Pre­req­ui­sites

To have a fully fea­tured and non-integrated Linux envi­ron­ment on your FreeBSD sys­tem either mount an exist­ing (and com­pat­i­ble) Linux instal­la­tion some­where into your FreeBSD sys­tem, or install a linux_dist port. This can be done addi­tion­ally to an already installed linux_base port.

Prepa­ra­tion

When you have a com­plete Linux envi­ron­ment avail­able, you need to mount the FreeBSD devfs to /path/to/complete_linux/dev, lin­procfs to /path/to/complete_linux/proc and lin­sysfs to /path/to/complete_linux/sys to have a com­plete setup.

Use it

Now you just need to chroot into this  /path/to/complete_linux and you configure/make/install or what­ever you need to do to gen­er­ate your desired Linux binary.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…
Share/Save

Dec
23

Cal­cu­lat­ing the tar­get size of H264 videos

From time to time I con­vert videos to H264. When I do this I want to get the best qual­ity out of a give file­size. This means I cre­ate VBR videos. The ques­tion here is, how big the tar­get video file shall be.

After search­ing around a lit­tle bit in the net I found a for­mula which is sup­posed to give a hint about the tar­get file­size. Nat­u­rally this depends on the encoder (or even the encoder-version), the encoder set­tings and even the video.

The for­mula is at least a good hint for my use, so I wrote a script which cal­cu­lates sev­eral file­size val­ues for a given video (based upon the out­put of medi­ainfo, which the scripts expects in a file with the file­name as an argu­ment to the script). It cal­cu­lates a CBR and a VBR value for a given video based upon the width, height and dura­tion. It should work on all sys­tem with a POSIX com­pat­i­ble shell.

Exam­ple out­put for a video from my HD-ready cam, orig­i­nal file­size 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per sec­ond: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per sec­ond: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per sec­ond: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sec­tions, the dif­fer­ence is the “motion” value. It is a kind of mul­ti­pli­ca­tor depend­ing on the amount of motion in the video. For the videos I made myself (fam­ily videos, and even some videos of vol­ley ball games), the first sec­tion seems to be just fine. So I reduced the orig­i­nal MP4 file to about 50% (not vis­i­ble here is the audio size, nor­mally I copy the orig­i­nal audio unmodified).

For the curi­ous ones, the for­mula is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps value. The CBR value is

bps * playtime_in_seconds / 8

and the VBR value is

3 / 4 * CBR_value.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…

Nov
25

Which crypto card to use with FreeBSD (ssh/gpg)

The recent secu­rity inci­dent trig­gered a dis­cus­sion how to secure ssh/gpg keys.

One way I want to focus on here (because it is the way I want to use at home), is to store the keys on a crypto card. I did some research for suit­able crypto cards and found one which is called Feit­ian PKI Smart­card, and one which is called OpenPGP card. The OpenPGP card also exists in a USB ver­sion (basi­cally a small ver­sion of the card is already inte­grated into a small USB card reader).

The Feit­ian card is reported to be able to han­dle RSA keys upto 2048 bits. They do not seem to han­dle DSA (or ECDSA) keys. The smart­card quick starter guide they have  (the Tun­ing smart­card file sys­tem part) tells how to change the para­me­ters of the card to store upto 9 keys on it.

The spec of the OpenPGP card tells that it sup­ports RSA keys upto 3072 bits, but there are reports that it is able to han­dle RSA keys upto 4096 bits (you need to have at least GPG 2.0.18 to han­dle that big keys on the crypto card). It looks to me like the card is not han­dle DSA (or ECDSA) cards. There are only slots for upto 3 keys on it.

If I go this way, I would also need a card reader. It seems a class 3 one (hard­ware PIN pad and dis­play) would be the most “future-proof” way to go ahead. I found a Reiner SCT cyber­Jack sec­oder card reader, which is believed to be sup­ported by OpenSC and seems to be a good bal­ance between cost and fea­tures of the Reiner SCT card readers.

If any­one read­ing this can sug­gest a bet­ter crypto card (keys upto 4096 bits, more than 3 slots, and/or DSA/ECDSA  sup­port), or a bet­ter card reader, or has any prac­ti­cal expe­ri­ence with any of those com­po­nents on FreeBSD, please add a comment.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…

Jul
13

Book review: FreeBSD Device Drivers

In mid-April a woman from the mar­ket­ing depart­ment of No Starch Press con­tacted me and asked if I am inter­ested to do a pub­lic review of the FreeBSD Device Dri­vers book by Joseph Kong (no link to a book shop, go and have a look in your pre­ferred one). Just this sim­ple ques­tion, no strings attached.

I had my nose in some device dri­vers in the past, but I never wrote one, and never had a look at the big pic­ture. I was inter­ested to know how every­thing fits together, so this made me a good vic­tim for a review (novice enough to learn some­thing new and to have a look if enough is explained, and expe­ri­enced enough to under­stand what is going on in the FreeBSD ker­nel).

Some min­utes after I agreed to review it (but with a lit­tle notice that I do not know how long I need to review it), I had the PDF ver­sion of the book. That was faster than I expected (maybe I am too old-school and used to have paper ver­sions of books in my hands).

Let the review begin… but bear with me, this is the first time I do a real pub­lic review of a book (instead of a tech­ni­cal review for an author). And as this is my very own per­sonal opin­ion, I will not allow com­ments here. This page is all about my opin­ion while read­ing the book, ques­tions I have while read­ing the book shall serve as a hint about the qual­ity of the book and they should be answered in the book, not here.

In short, the book is not per­fect, but it is a good book. There is room for improve­ment, but on a very high level. If you want to write a device dri­ver for FreeBSD, this book is a must. I sug­gest to read it com­pletely, even chap­ters which do not belong to the type of dri­ver you want to write (spe­cially the case stud­ies of real dri­vers). The rea­son is that each chap­ter has some notes which may not only apply to the chap­ter in ques­tion, but to all kinds of device dri­vers. The long review fol­lows now.

The first chap­ter is titled “Build­ing and run­ning mod­ules”. The author begins with descrip­tion of the usual device dri­ver types (NIC dri­ver, pseudo-device, …) and how they can be added to the ker­nel (sta­t­i­cally linked in or as a mod­ule). The first code exam­ple is a small and easy ker­nel mod­ule, so that we do not have to reboot the sys­tem we use to develop a dri­ver (except we make a fault dur­ing dri­ver devel­op­ment which causes the machine to panic or hang). Every part of the exam­ple is well explained. This is fol­lowed by an overview about char­ac­ter devices (e.g. disks) and a sim­ple character-device dri­ver (so far a pseudo-device, as we do not have real hard­ware we access) which is not only as-well explained as the module-example, but there is also a note where the code was sim­pli­fied and what should be done instead.

After read­ing this chap­ter you should be able to write your own ker­nel mod­ule in 5 min­utes (well, after 5 min­utes it will not be able to do a lot — just a “hello world” – but at least you can already load/unload/execute some code into/from/in the kernel).

I have not tried any exam­ple myself, but I com­piled a lot of mod­ules and dri­vers I mod­i­fied in the past and remem­ber to have seen the described parts.

The sec­ond chap­ter explains how to allo­cate and free mem­ory in the ker­nel. There is the pos­si­bil­ity to allo­cate maybe-contiguous mem­ory (the nor­mal case, when your hard­ware does not do DMA or does not have the require­ment that the mem­ory region it makes DMA from/too needs to be con­tigu­ous), and really con­tigu­ous. For the size argu­ment of the free­ing of the the con­tigu­ous mem­ory there is the sen­tence “Gen­er­ally, size should be equal the amount allo­cated.”. Imme­di­ately I wanted to know what hap­pens if you spec­ify a dif­fer­ent size (as a non-native eng­lish speaker I under­stand this sen­tence in a way that I am allowed to spec­ify a dif­fer­ent size and as such are able to free only parts of the allo­cated mem­ory). Unfor­tu­nately this is not answered. I had a look into the source, the ker­nel frees mem­ory pages, so the size argu­ment (and addr argu­ment) will be rounded to include a full page. This means the­o­ret­i­cally I am able to free parts of the allo­cated mem­ory, but this is a source-maintenance night­mare (needs knowl­edge about the machine spe­cific page bound­aries and you need to make sure that you do the absolutely cor­rect size cal­cu­la­tions).  To me this looks more like as long as nobody is point­ing a gun at my head and tells me to use a dif­fer­ent size, spec­i­fy­ing the same size as made dur­ing the allo­ca­tion of this mem­ory region is the way to go.

After read­ing this chap­ter you should know how to kill the sys­tem by allo­cat­ing all the RAM in the kernel.

Again, I did not try to com­pile the exam­ples in this chap­ter, but the dif­fer­ence of the mem­ory allo­ca­tion in the ker­nel com­pared with mem­ory allo­ca­tion in the user­land is not that big.

The third chap­ter explains the device com­mu­ni­ca­tion and con­trol inter­faces (ioctl/sysctl) of a dri­ver. The ioctl part teached me some parts I always wanted to know when I touched some ioctls, but never both­ered to find out before. Unfor­tu­nately this makes me a lit­tle bit ner­vous about the way ioctls are han­dled in the FreeBSD lin­ux­u­la­tor, but this is not urgent ATM (and can prob­a­bly be han­dled by a com­mend in the right place). The sysctl part takes a lit­tle bit longer to fol­low through, but there is also more to learn about it. If you just mod­ify an exist­ing dri­ver with an exist­ing sysctl inter­face, it prob­a­bly just comes down to copy&paste with lit­tle mod­i­fi­ca­tions, but if you need to make more com­plex changes or want to add a sysctl inter­face to a dri­ver, this part of the book is a good way to under­stand what is pos­si­ble and how every­thing fits together. Per­son­ally I would have wished for a more detailed guide when to pick the ioctl inter­face and when the sysctl inter­face than what was writ­ten in the con­clu­sion of the chap­ter, but it is prob­a­bly not that easy to come up with a good list which fits most drivers.

After read­ing this chap­ter you should be able to get data in and out of the ker­nel in 10 minutes.

As before, I did not com­pile the exam­ples in this chap­ter. I already added ioctls and sysctls in var­i­ous places in the FreeBSD kernel.

Chap­ter 4 is about thread syn­chro­niza­tion – mutexes, shared/exclusive locks, reader/writer locks and con­di­tion vari­ables. For me this chap­ter is not as good as the pre­vi­ous ones. While I got a good expla­na­tion of every­thing, I missed a nice overview table which com­pares the var­i­ous meth­ods of thread syn­chro­niza­tion. Bren­dan Gregg did a nice table to give an overview of DTrace vari­able types and when to use them. Some­thing like this would have been nice in this chap­ter too. Apart from this I got all the info I need (but hey, I already wrote a NFS client for an exper­i­men­tal com­puter with more than 200000 CPUs in 1998, so I’m famil­iar with such syn­chro­niza­tion primitives).

Delayed exe­cu­tion is explained in chap­ter 5. Most of the infor­ma­tion pre­sented there was new to me. While there where not much exam­ples pre­sented (there will be some in a later chap­ter), I got a good overview about what exists. This time there was even an overview when to use which type of delayed exe­cu­tion infra­struc­ture. I would have pre­ferred to have this overview in the begin­ning of the chap­ter, but that is maybe some kind of per­sonal preference.

In chap­ter 6 a com­plete device dri­ver is dis­sected. It is the vir­tual null modem ter­mi­nal dri­ver. The chap­ter pro­vides real-world exam­ples of event-handlers, call­outs and taskqueues which where not demon­strated in chap­ter five. At the same time the chap­ter serves as a descrip­tion of the func­tions a TTY dri­ver needs to have.

Auto­mated device detec­tion with New­bus and the cor­re­spond­ing resource allo­ca­tion (I/O ports, device mem­ory and inter­rupts) are explained in chap­ter 7. It is easy… if you have a real device to play with. Unfor­tu­nately the chap­ter missed a para­graph or two about the sus­pend and resume meth­ods. If you think about it, it is not hard to come up with what they are sup­posed to do, but a lit­tle explicit descrip­tion of what they shall do, in what state the hard­ware should be put and what to assume when being called would have been nice.

Chap­ter 8 is about inter­rupts. It is easy to add an inter­rupt han­dler (or to remove one), the hard part is to gen­er­ate an inter­rupt. The exam­ple code uses the par­al­lel port, and the chap­ter also con­tains a lit­tle expla­na­tion how to gen­er­ate an inter­rupt… if you are not afraid to touch real hard­ware (the par­al­lel port) with a resistor.

In chap­ter 9 the lpt(4) dri­ver is explained, as most of the top­ics dis­cussed so far are used inside. The expla­na­tion how every­thing is used is good, but what I miss some­times is why they are used. The most promi­nent (and only) exam­ple here for me is why are call­outs used to catch stray inter­rupts? That call­outs are a good way of han­dling this is clear to me, the big ques­tion is why can there be stray inter­rupts. Can this hap­pen only for the par­al­lel port (respec­tively a lim­ited amount of devices), or does every dri­ver for real inter­rupt dri­ven hard­ware need to come with some­thing like this? I assume this is some­thing spe­cific to the device, but a lit­tle expla­na­tion regard­ing this would have been nice.

Access­ing I/O ports and I/O mem­ory for devices are explained in chap­ter 10 based upon a dri­ver for a LED device (turn on and off 2 LEDs on an ISA bus). All the func­tions to read and write data are well explained, just the part about the mem­ory bar­rier is a lit­tle bit short. It is not clear why the CPU reorder­ing of mem­ory accesses mat­ter to what looks like func­tion calls. Those func­tion calls may be macros, but this is not explained in the text. Some lit­tle exam­ples when to use the bar­ri­ers instead of an abstract descrip­tion would also have been nice at this point.

Chap­ter 11 is sim­i­lar to chap­ter 10, just that a PCI bus dri­ver is dis­cussed instead of an ISA bus dri­ver. The dif­fer­ences are not that big, but important.

In chap­ter 12 it is explained how to do DMA in a dri­ver. This part is not easy to under­stand. I would have wanted to have more exam­ples and expla­na­tions of the DMA tag and DMA map parts. I am also sur­prised to see dif­fer­ent sup­ported archi­tec­tures for the flags BUS_DMA_COHERENT and BUS_DMA_NOCACHE for dif­fer­ent func­tions. Either this means FreeBSD is not coher­ent in those parts, or it is a bug in the book, or it is sup­posed to be like this and the rea­sons are not explained in the book. As there is no explicit note about this, it prob­a­bly leads to con­fu­sion of read­ers which pay enough atten­tion here. It would also have been nice to have an expla­na­tion when to use those flags which are only imple­mented on a sub­set of the archi­tec­tures FreeBSD sup­ports. Any­way, the expla­na­tions give enough infor­ma­tion to under­stand what is going on and to be able to have a look at other device dri­vers for real-live exam­ples and to get a deeper under­stand­ing of this topic.

Disk dri­vers and block I/O (bio) requests are described in chap­ter 13. With this chap­ter I have a lit­tle prob­lem. The author used the word “unde­fined” in sev­eral places where I as a non-native speaker would have used “not set” or “set to 0″. The word “unde­fined” implies for me that there may be garbage inside, whereas from a tech­ni­cal point of view I can not imag­ine that some ran­dom value in those places would have the desired result. In my opin­ion each such place is obvi­ous, so I do not expect that an expe­ri­enced pro­gram­mer would lose time/hairs/sanity over it, but inex­pe­ri­enced pro­gram­mers which try to assem­ble the cor­re­spond­ing struc­tures on the (unini­tial­ized) heap (for what­ever rea­son), may strug­gle with this.

Chap­ter 14 is about the CAM layer. While the pre­vi­ous chap­ter showed how to write a dri­ver for a disk device, chap­ter 14 gave an overview about how to an HBA to the CAM layer. It is just an overview, it looks like CAM needs a book on its own to be fully described. The sim­ple (and most impor­tant) cases are described, with the hardware-specific parts being an exer­cise for the per­son writ­ing the device dri­ver. I have the impres­sion it gives enough details to let some­one with hard­ware (or pro­to­col), and more impor­tantly doc­u­men­ta­tion for this device, start writ­ing a driver.

It would have been nice if chap­ter 13 and 14 would have had a lit­tle schematic which describes at which level of the kernel-subsystems the cor­re­spond­ing dri­ver sits. And while I am at it, a schematic with all the dri­ver com­po­nents dis­cussed in this book at the begin­ning as an overview, or in the end as an annex, would be great too.

An overview of USB dri­vers is given in chap­ter 15 with the USB printer dri­ver as an exam­ple for the expla­na­tion of the USB dri­ver inter­faces. If USB would not be as com­plex as it is, it would be a nice chap­ter to start driver-writing exper­i­ments (due to the avail­abil­ity of var­i­ous USB devices). Well… bad luck for curi­ous peo­ple. BTW, the author gives point­ers to the offi­cial USB docs, so if you are really curi­ous, feel free to go ahead. :)

Chap­ter 16 is the first part about net­work dri­vers. It deals with ifnet (e.g. stuff needed for ifcon­fig), ifme­dia (sim­pli­fied: which kind of cable and speed is sup­ported), mbufs and MSI(-X). As in other chap­ters before, a lit­tle overview and a lit­tle pic­ture in the begin­ning would have been nice.

Finally, in chap­ter 17, the packet recep­tion and trans­mis­sion of net­work dri­vers is described. Large exam­ple code is bro­ken up into sev­eral pieces here, for more easy dis­cus­sion of related information.

One thing I miss after reach­ing the end of the book is a dis­cus­sion of sound dri­vers. And this is surely not the only type of dri­vers which is not dis­cussed, I can come up with crypto, firewire, gpio, watch­dog, smb and iic devices within a few sec­onds. While I think that it is much more easy to under­stand all those dri­vers now after read­ing the book, it would have been nice to have at least a lit­tle overview of other dri­ver types and maybe even a short descrip­tion of their dri­ver methods.

Con­clu­sion: As I wrote already in the begin­ning, the book is not per­fect, but it is good. While I have not writ­ten a device dri­ver for FreeBSD, the book pro­vided enough insight to be able to write one and to under­stand exist­ing dri­vers. I really hope there will be a sec­ond edi­tion which addresses the minor issues I had while read­ing it to make it a per­fect book.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…

Tags: , , , , , , , , ,
May
31

Free DLNA server which works good with my Sony BRAVIA TV

In sev­eral pre­vi­ous posts I wrote about my quest for the right source for­mat to stream video to my Sony BRAVIA TV (build in 2009). The last week-end I finally found some­thing which sat­is­fies me.

What I found was serviio, a free UPnP-AV (DLNA) server. It is writ­ten in java and runs on Win­dows, Linux and FreeBSD (it is not listed on the web­site, but we have an not-so-up-to-date ver­sion in the ports tree). If nec­es­sary it transcodes the input to an appro­pri­ate for­mat for the DLNA ren­derer (in my case the TV).

I tested it with my slow Net­book, so that I was able to see with which input for­mat it will just remux the input con­tainer to a MPEG trans­port stream, and which input for­mat would be really re-encoded to a for­mat the TV understands.

The bot­tom line of the tests is, that I just need to use a sup­ported con­tainer (like MKV or MP4 or AVI) with H.264-encoded video (e.g. encoded by x264) and AC3 audio.

The TV is able to chose between sev­eral audio streams, but I have not tested if serviio is able to serve files with mul­ti­ple audio streams (my wife has a dif­fer­ent mother lan­guage than me, so it is inter­est­ing for us to have mul­ti­ple audio streams for a movie), and I do not know if DLNA sup­ports some­thing like this.

Now I just have to replace minidlna (which only works good with my TV for MP3s and Pic­tures) with serviio on my FreeBSD file server and we can for­get about the disk-juggling.

GD Star Rat­ing
load­ing…
GD Star Rat­ing
load­ing…

Tags: , , , , , , , , ,