Cal­cu­lat­ing the tar­get size of H264 videos

From time to time I con­vert videos to H264. When I do this I want to get the best qual­ity out of a give files­ize. This means I cre­ate VBR videos. The ques­tion here is, how big the tar­get video file shall be.

After search­ing around a little bit in the net I found a for­mula which is sup­posed to give a hint about the tar­get files­ize. Nat­ur­ally this de­pends on the en­coder (or even the encoder-​version), the en­coder set­tings and even the video.

The for­mula is at least a good hint for my use, so I wrote a script which cal­cu­lates sev­er­al files­ize val­ues for a giv­en video (based upon the out­put of me­di­ainfo, which the scripts ex­pects in a file with the fi­le­name as an ar­gu­ment to the script). It cal­cu­lates a CBR and a VBR value for a giv­en video based upon the width, height and dur­a­tion. It should work on all sys­tem with a POSIX com­pat­ible shell.

Ex­ample out­put for a video from my HD-​ready cam, ori­gin­al files­ize 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 2
Per second: 6451200.000 bps /​ 6300 Kibps
Total CBR: 1148313600 bytes /​ 1121400 KiB /​ 1095 MiB
Total VBR: 861235200 bytes /​ 841050 KiB /​ 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 3
Per second: 9676800.000 bps /​ 9450 Kibps
Total CBR: 1722470400 bytes /​ 1682100 KiB /​ 1642 MiB
Total VBR: 1291852800 bytes /​ 1261575 KiB /​ 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Mo­tion: 4
Per second: 12902400.000 bps /​ 12600 Kibps
Total CBR: 2296627200 bytes /​ 2242800 KiB /​ 2190 MiB
Total VBR: 1722470400 bytes /​ 1682100 KiB /​ 1642 MiB

There are 3 sec­tions, the dif­fer­ence is the “mo­tion” value. It is a kind of mul­ti­plic­at­or de­pend­ing on the amount of mo­tion in the video. For the videos I made my­self (fam­ily videos, and even some videos of vol­ley ball games), the first sec­tion seems to be just fine. So I re­duced the ori­gin­al MP4 file to about 50% (not vis­ible here is the au­dio size, nor­mally I copy the ori­gin­al au­dio un­mod­i­fied).

For the curi­ous ones, the for­mula is

width_​in_​pixels * height_​in_​pixels * fps * motion_​value * 0.07

for the bps value. The CBR value is

bps * playtime_​in_​seconds /​ 8

and the VBR value is

3 /​ 4 * CBR_​value.

Which crypto card to use with FreeBSD (ssh/​gpg)

The re­cent se­cur­ity in­cid­ent triggered a dis­cus­sion how to se­cure ssh/​gpg keys.

One way I want to fo­cus on here (be­cause it is the way I want to use at home), is to store the keys on a crypto card. I did some re­search for suit­able crypto cards and found one which is called Fei­tian PKI Smart­card, and one which is called Open­P­GP card. The Open­P­GP card also ex­ists in a USB ver­sion (ba­sic­ally a small ver­sion of the card is already in­teg­rated in­to a small USB card read­er).

The Fei­tian card is re­por­ted to be able to handle RSA keys upto 2048 bits. They do not seem to handle DSA (or ECDSA) keys. The smart­card quick starter guide they have  (the Tun­ing smart­card file sys­tem part) tells how to change the para­met­ers of the card to store upto 9 keys on it.

The spec of the Open­P­GP card tells that it sup­ports RSA keys upto 3072 bits, but there are re­ports that it is able to handle RSA keys upto 4096 bits (you need to have at least GPG 2.0.18 to handle that big keys on the crypto card). It looks to me like the card is not handle DSA (or ECDSA) cards. There are only slots for upto 3 keys on it.

If I go this way, I would also need a card read­er. It seems a class 3 one (hard­ware PIN pad and dis­play) would be the most “future-​proof” way to go ahead. I found a Rein­er SCT cy­ber­Jack secoder card read­er, which is be­lieved to be sup­por­ted by OpenSC and seems to be a good bal­ance between cost and fea­tures of the Rein­er SCT card read­ers.

If any­one read­ing this can sug­gest a bet­ter crypto card (keys upto 4096 bits, more than 3 slots, and/​or DSA/​ECDSA  sup­port), or a bet­ter card read­er, or has any prac­tic­al ex­per­i­ence with any of those com­pon­ents on FreeBSD, please add a com­ment.

Book re­view: FreeBSD Device Drivers

In mid-​April a wo­man from the mar­ket­ing de­part­ment of No Starch Press con­tac­ted me and asked if I am in­ter­ested to do a pub­lic re­view of the FreeBSD Device Drivers book by Joseph Kong (no link to a book shop, go and have a look in your pre­ferred one). Just this simple ques­tion, no strings at­tached.

I had my nose in some device drivers in the past, but I nev­er wrote one, and nev­er had a look at the big pic­ture. I was in­ter­ested to know how everything fits to­geth­er, so this made me a good vic­tim for a re­view (novice enough to learn some­thing new and to have a look if enough is ex­plained, and ex­per­i­enced enough to un­der­stand what is go­ing on in the FreeBSD ker­nel).

Some minutes after I agreed to re­view it (but with a little no­tice that I do not know how long I need to re­view it), I had the PDF ver­sion of the book. That was faster than I ex­pec­ted (maybe I am too old-​school and used to have pa­per ver­sions of books in my hands).

Let the re­view be­gin… but bear with me, this is the first time I do a real pub­lic re­view of a book (in­stead of a tech­nic­al re­view for an au­thor). And as this is my very own per­son­al opin­ion, I will not al­low com­ments here. This page is all about my opin­ion while read­ing the book, ques­tions I have while read­ing the book shall serve as a hint about the qual­ity of the book and they should be answered in the book, not here.

In short, the book is not per­fect, but it is a good book. There is room for im­prove­ment, but on a very high level. If you want to write a device driver for FreeBSD, this book is a must. I sug­gest to read it com­pletely, even chapters which do not be­long to the type of driver you want to write (spe­cially the case stud­ies of real drivers). The reas­on is that each chapter has some notes which may not only ap­ply to the chapter in ques­tion, but to all kinds of device drivers. The long re­view fol­lows now.

The first chapter is titled “Build­ing and run­ning mod­ules”. The au­thor be­gins with de­scrip­tion of the usu­al device driver types (NIC driver, pseudo-​device, …) and how they can be ad­ded to the ker­nel (stat­ic­ally linked in or as a mod­ule). The first code ex­ample is a small and easy ker­nel mod­ule, so that we do not have to re­boot the sys­tem we use to de­vel­op a driver (ex­cept we make a fault dur­ing driver de­vel­op­ment which causes the ma­chine to pan­ic or hang). Every part of the ex­ample is well ex­plained. This is fol­lowed by an over­view about char­ac­ter devices (e.g. disks) and a simple character-​device driver (so far a pseudo-​device, as we do not have real hard­ware we ac­cess) which is not only as-​well ex­plained as the module-​example, but there is also a note where the code was sim­pli­fied and what should be done in­stead.

After read­ing this chapter you should be able to write your own ker­nel mod­ule in 5 minutes (well, after 5 minutes it will not be able to do a lot – just a “hello world” – but at least you can already load/​unload/​execute some code into/​from/​in the ker­nel).

I have not tried any ex­ample my­self, but I com­piled a lot of mod­ules and drivers I mod­i­fied in the past and re­mem­ber to have seen the de­scribed parts.

The second chapter ex­plains how to al­loc­ate and free memory in the ker­nel. There is the pos­sib­il­ity to al­loc­ate maybe-​contiguous memory (the nor­mal case, when your hard­ware does not do DMA or does not have the re­quire­ment that the memory re­gion it makes DMA from/​too needs to be con­tigu­ous), and really con­tigu­ous. For the size ar­gu­ment of the free­ing of the the con­tigu­ous memory there is the sen­tence “Gen­er­ally, size should be equal the amount al­loc­ated.”. Im­me­di­ately I wanted to know what hap­pens if you spe­cify a dif­fer­ent size (as a non-​native eng­lish speak­er I un­der­stand this sen­tence in a way that I am al­lowed to spe­cify a dif­fer­ent size and as such are able to free only parts of the al­loc­ated memory). Un­for­tu­nately this is not answered. I had a look in­to the source, the ker­nel frees memory pages, so the size ar­gu­ment (and ad­dr ar­gu­ment) will be roun­ded to in­clude a full page. This means the­or­et­ic­ally I am able to free parts of the al­loc­ated memory, but this is a source-​maintenance night­mare (needs know­ledge about the ma­chine spe­cif­ic page bound­ar­ies and you need to make sure that you do the ab­so­lutely cor­rect size cal­cu­la­tions).  To me this looks more like as long as nobody is point­ing a gun at my head and tells me to use a dif­fer­ent size, spe­cify­ing the same size as made dur­ing the al­loc­a­tion of this memory re­gion is the way to go.

After read­ing this chapter you should know how to kill the sys­tem by al­loc­at­ing all the RAM in the ker­nel.

Again, I did not try to com­pile the ex­amples in this chapter, but the dif­fer­ence of the memory al­loc­a­tion in the ker­nel com­pared with memory al­loc­a­tion in the user­land is not that big.

The third chapter ex­plains the device com­mu­nic­a­tion and con­trol in­ter­faces (ioctl/​sysctl) of a driver. The ioctl part teached me some parts I al­ways wanted to know when I touched some ioctls, but nev­er bothered to find out be­fore. Un­for­tu­nately this makes me a little bit nervous about the way ioctls are handled in the FreeBSD linuxu­lat­or, but this is not ur­gent ATM (and can prob­ably be handled by a com­mend in the right place). The sy­sctl part takes a little bit longer to fol­low through, but there is also more to learn about it. If you just modi­fy an ex­ist­ing driver with an ex­ist­ing sy­sctl in­ter­face, it prob­ably just comes down to copy&paste with little modi­fic­a­tions, but if you need to make more com­plex changes or want to add a sy­sctl in­ter­face to a driver, this part of the book is a good way to un­der­stand what is pos­sible and how everything fits to­geth­er. Per­son­ally I would have wished for a more de­tailed guide when to pick the ioctl in­ter­face and when the sy­sctl in­ter­face than what was writ­ten in the con­clu­sion of the chapter, but it is prob­ably not that easy to come up with a good list which fits most drivers.

After read­ing this chapter you should be able to get data in and out of the ker­nel in 10 minutes.

As be­fore, I did not com­pile the ex­amples in this chapter. I already ad­ded ioctls and sy­sctls in vari­ous places in the FreeBSD ker­nel.

Chapter 4 is about thread syn­chron­iz­a­tion – mu­texes, shared/​exclusive locks, reader/​writer locks and con­di­tion vari­ables. For me this chapter is not as good as the pre­vi­ous ones. While I got a good ex­plan­a­tion of everything, I missed a nice over­view table which com­pares the vari­ous meth­ods of thread syn­chron­iz­a­tion. Brendan Gregg did a nice table to give an over­view of DTrace vari­able types and when to use them. Some­thing like this would have been nice in this chapter too. Apart from this I got all the info I need (but hey, I already wrote a NFS cli­ent for an ex­per­i­ment­al com­puter with more than 200000 CPUs in 1998, so I’m fa­mil­i­ar with such syn­chron­iz­a­tion prim­it­ives).

Delayed ex­e­cu­tion is ex­plained in chapter 5. Most of the in­form­a­tion presen­ted there was new to me. While there where not much ex­amples presen­ted (there will be some in a later chapter), I got a good over­view about what ex­ists. This time there was even an over­view when to use which type of delayed ex­e­cu­tion in­fra­struc­ture. I would have pre­ferred to have this over­view in the be­gin­ning of the chapter, but that is maybe some kind of per­son­al pref­er­ence.

In chapter 6 a com­plete device driver is dis­sec­ted. It is the vir­tu­al null mo­dem ter­min­al driver. The chapter provides real-​world ex­amples of event-​handlers, cal­louts and taskqueues which where not demon­strated in chapter five. At the same time the chapter serves as a de­scrip­tion of the func­tions a TTY driver needs to have.

Auto­mated device de­tec­tion with New­bus and the cor­res­pond­ing re­source al­loc­a­tion (I/​O ports, device memory and in­ter­rupts) are ex­plained in chapter 7. It is easy… if you have a real device to play with. Un­for­tu­nately the chapter missed a para­graph or two about the sus­pend and re­sume meth­ods. If you think about it, it is not hard to come up with what they are sup­posed to do, but a little ex­pli­cit de­scrip­tion of what they shall do, in what state the hard­ware should be put and what to as­sume when be­ing called would have been nice.

Chapter 8 is about in­ter­rupts. It is easy to add an in­ter­rupt hand­ler (or to re­move one), the hard part is to gen­er­ate an in­ter­rupt. The ex­ample code uses the par­al­lel port, and the chapter also con­tains a little ex­plan­a­tion how to gen­er­ate an in­ter­rupt… if you are not afraid to touch real hard­ware (the par­al­lel port) with a res­ist­or.

In chapter 9 the lpt(4) driver is ex­plained, as most of the top­ics dis­cussed so far are used in­side. The ex­plan­a­tion how everything is used is good, but what I miss some­times is why they are used. The most prom­in­ent (and only) ex­ample here for me is why are cal­louts used to catch stray in­ter­rupts? That cal­louts are a good way of hand­ling this is clear to me, the big ques­tion is why can there be stray in­ter­rupts. Can this hap­pen only for the par­al­lel port (re­spect­ively a lim­ited amount of devices), or does every driver for real in­ter­rupt driv­en hard­ware need to come with some­thing like this? I as­sume this is some­thing spe­cif­ic to the device, but a little ex­plan­a­tion re­gard­ing this would have been nice.

Ac­cess­ing I/​O ports and I/​O memory for devices are ex­plained in chapter 10 based upon a driver for a LED device (turn on and off 2 LEDs on an ISA bus). All the func­tions to read and write data are well ex­plained, just the part about the memory bar­ri­er is a little bit short. It is not clear why the CPU re­order­ing of memory ac­cesses mat­ter to what looks like func­tion calls. Those func­tion calls may be mac­ros, but this is not ex­plained in the text. Some little ex­amples when to use the bar­ri­ers in­stead of an ab­stract de­scrip­tion would also have been nice at this point.

Chapter 11 is sim­il­ar to chapter 10, just that a PCI bus driver is dis­cussed in­stead of an ISA bus driver. The dif­fer­ences are not that big, but im­port­ant.

In chapter 12 it is ex­plained how to do DMA in a driver. This part is not easy to un­der­stand. I would have wanted to have more ex­amples and ex­plan­a­tions of the DMA tag and DMA map parts. I am also sur­prised to see dif­fer­ent sup­por­ted ar­chi­tec­tures for the flags BUS_​DMA_​COHERENT and BUS_​DMA_​NOCACHE for dif­fer­ent func­tions. Either this means FreeBSD is not co­her­ent in those parts, or it is a bug in the book, or it is sup­posed to be like this and the reas­ons are not ex­plained in the book. As there is no ex­pli­cit note about this, it prob­ably leads to con­fu­sion of read­ers which pay enough at­ten­tion here. It would also have been nice to have an ex­plan­a­tion when to use those flags which are only im­ple­men­ted on a sub­set of the ar­chi­tec­tures FreeBSD sup­ports. Any­way, the ex­plan­a­tions give enough in­form­a­tion to un­der­stand what is go­ing on and to be able to have a look at oth­er device drivers for real-​live ex­amples and to get a deep­er un­der­stand­ing of this top­ic.

Disk drivers and block I/​O (bio) re­quests are de­scribed in chapter 13. With this chapter I have a little prob­lem. The au­thor used the word “un­defined” in sev­er­al places where I as a non-​native speak­er would have used “not set” or “set to 0”. The word “un­defined” im­plies for me that there may be garbage in­side, where­as from a tech­nic­al point of view I can not ima­gine that some ran­dom value in those places would have the de­sired res­ult. In my opin­ion each such place is ob­vi­ous, so I do not ex­pect that an ex­per­i­enced pro­gram­mer would lose time/​hairs/​sanity over it, but in­ex­per­i­enced pro­gram­mers which try to as­semble the cor­res­pond­ing struc­tures on the (un­ini­tial­ized) heap (for whatever reas­on), may struggle with this.

Chapter 14 is about the CAM lay­er. While the pre­vi­ous chapter showed how to write a driver for a disk device, chapter 14 gave an over­view about how to an HBA to the CAM lay­er. It is just an over­view, it looks like CAM needs a book on its own to be fully de­scribed. The simple (and most im­port­ant) cases are de­scribed, with the hardware-​specific parts be­ing an ex­er­cise for the per­son writ­ing the device driver. I have the im­pres­sion it gives enough de­tails to let someone with hard­ware (or pro­tocol), and more im­port­antly doc­u­ment­a­tion for this device, start writ­ing a driver.

It would have been nice if chapter 13 and 14 would have had a little schem­at­ic which de­scribes at which level of the kernel-​subsystems the cor­res­pond­ing driver sits. And while I am at it, a schem­at­ic with all the driver com­pon­ents dis­cussed in this book at the be­gin­ning as an over­view, or in the end as an an­nex, would be great too.

An over­view of USB drivers is giv­en in chapter 15 with the USB print­er driver as an ex­ample for the ex­plan­a­tion of the USB driver in­ter­faces. If USB would not be as com­plex as it is, it would be a nice chapter to start driver-​writing ex­per­i­ments (due to the avail­ab­il­ity of vari­ous USB devices). Well… bad luck for curi­ous people. BTW, the au­thor gives point­ers to the of­fi­cial USB docs, so if you are really curi­ous, feel free to go ahead. 🙂

Chapter 16 is the first part about net­work drivers. It deals with ifnet (e.g. stuff needed for if­con­fig), if­me­dia (sim­pli­fied: which kind of cable and speed is sup­por­ted), mbufs and MSI(-X). As in oth­er chapters be­fore, a little over­view and a little pic­ture in the be­gin­ning would have been nice.

Fi­nally, in chapter 17, the pack­et re­cep­tion and trans­mis­sion of net­work drivers is de­scribed. Large ex­ample code is broken up in­to sev­er­al pieces here, for more easy dis­cus­sion of re­lated in­form­a­tion.

One thing I miss after reach­ing the end of the book is a dis­cus­sion of sound drivers. And this is surely not the only type of drivers which is not dis­cussed, I can come up with crypto, firewire, gpio, watch­dog, smb and iic devices with­in a few seconds. While I think that it is much more easy to un­der­stand all those drivers now after read­ing the book, it would have been nice to have at least a little over­view of oth­er driver types and maybe even a short de­scrip­tion of their driver meth­ods.

Con­clu­sion: As I wrote already in the be­gin­ning, the book is not per­fect, but it is good. While I have not writ­ten a device driver for FreeBSD, the book provided enough in­sight to be able to write one and to un­der­stand ex­ist­ing drivers. I really hope there will be a second edi­tion which ad­dresses the minor is­sues I had while read­ing it to make it a per­fect book.

Free DLNA serv­er which works good with my Sony BRAVIA TV

In sev­er­al pre­vi­ous posts I wrote about my quest for the right source format to stream video to my Sony BRAVIA TV (build in 2009). The last week-​end I fi­nally found some­thing which sat­is­fies me.

What I found was ser­viio, a free UPnP-​AV (DLNA) serv­er. It is writ­ten in java and runs on Win­dows, Linux and FreeBSD (it is not lis­ted on the web­site, but we have an not-​so-​up-​to-​date ver­sion in the ports tree). If ne­ces­sary it transcodes the in­put to an ap­pro­pri­ate format for the DLNA ren­der­er (in my case the TV).

I tested it with my slow Net­book, so that I was able to see with which in­put format it will just re­mux the in­put con­tain­er to a MPEG trans­port stream, and which in­put format would be really re-​encoded to a format the TV un­der­stands.

The bot­tom line of the tests is, that I just need to use a sup­por­ted con­tain­er (like MKV or MP4 or AVI) with H.264-encoded video (e.g. en­coded by x264) and AC3 au­dio.

The TV is able to chose between sev­er­al au­dio streams, but I have not tested if ser­viio is able to serve files with mul­tiple au­dio streams (my wife has a dif­fer­ent moth­er lan­guage than me, so it is in­ter­est­ing for us to have mul­tiple au­dio streams for a movie), and I do not know if DLNA sup­ports some­thing like this.

Now I just have to re­place min­idlna (which only works good with my TV for MP3s and Pic­tures) with ser­viio on my FreeBSD file serv­er and we can for­get about the disk-​juggling.

Linuxu­lat­or D-​Trace probes com­mit­ted to cur­rent

A while ago I com­mit­ted the linuxu­lat­or D-​Trace probes I talked about earli­er. I waited a little bit for this an­nounce­ment to make sure I have not broken any­thing. Nobody com­plained so far, so I as­sume noth­ing ob­vi­ously bad crept in.

The >500 probes I com­mit­ted do not cov­er the en­tire linuxu­lat­or, but are a good start. Adding new ones is straight for­ward, if someone is in­ter­ested in a ju­ni­or–ker­nel-hack­er task, this would be one. Just ask me (or ask on emu­la­tion@), and I can guide you through it.

DTrace in GENERIC (-cur­rent)

In case you have not no­ticed yet, KDTRACE_​HOOKS is now in the GENERIC ker­nel in FreeBSD-cur­rent. This means you just need to load the DTrace mod­ules and can use DTrace with the GENERIC ker­nel.

In case you do not know what you can do with DTrace, take the time to have a look at the DTrace blog. It is worth any minute you in­vest read­ing it.

Linuxu­lat­or pro­gress

This week­end I made some pro­gress in the linuxu­lat­or:

  • I MFCed the re­port­ing of some linux-​syscalls to 9-​stable and 8-​stable.
  • I up­dated my linuxu­lat­or-dtrace patch to a re­cent -cur­rent. I already com­piled it on i386 and arundel@ has it com­piled on amd64. I coun­ted more than 500 new DTrace probes. Now that DTrace res­cans for SDT probes when a ker­nel mod­ule is loaded, there is no ker­nel pan­ic any­more when the linux mod­ule is loaded after the DTrace mod­ules and you want to use DTrace. I try to com­mit this at a morn­ing of a day where I can fix things dur­ing the day in case some prob­lems show up which I did not no­tice dur­ing my test­ing.
  • I cre­ated a PR for portmgr@ to re­po­copy a new linux_​base port.
  • I set the ex­pir­a­tion date of linux_​base-​fc4 (only used by 7.x and up­stream way past its EoL) and all de­pend­ent ports. It is set to the EoL of the last 7.x re­lease, which can not use a later linux_​base port. I also ad­ded a com­ment which ex­plains that the date is the EoL of the last 7.x re­lease.