Alexander Leidinger

Just another weblog


Book review: FreeBSD Device Drivers

In mid-April a woman from the mar­ket­ing depart­ment of No Starch Press con­tacted me and asked if I am inter­ested to do a pub­lic review of the FreeBSD Device Dri­vers book by Joseph Kong (no link to a book shop, go and have a look in your pre­ferred one). Just this sim­ple ques­tion, no strings attached.

I had my nose in some device dri­vers in the past, but I never wrote one, and never had a look at the big pic­ture. I was inter­ested to know how every­thing fits together, so this made me a good vic­tim for a review (novice enough to learn some­thing new and to have a look if enough is explained, and expe­ri­enced enough to under­stand what is going on in the FreeBSD ker­nel).

Some min­utes after I agreed to review it (but with a lit­tle notice that I do not know how long I need to review it), I had the PDF ver­sion of the book. That was faster than I expected (maybe I am too old-school and used to have paper ver­sions of books in my hands).

Let the review begin… but bear with me, this is the first time I do a real pub­lic review of a book (instead of a tech­ni­cal review for an author). And as this is my very own per­sonal opin­ion, I will not allow com­ments here. This page is all about my opin­ion while read­ing the book, ques­tions I have while read­ing the book shall serve as a hint about the qual­ity of the book and they should be answered in the book, not here.

In short, the book is not per­fect, but it is a good book. There is room for improve­ment, but on a very high level. If you want to write a device dri­ver for FreeBSD, this book is a must. I sug­gest to read it com­pletely, even chap­ters which do not belong to the type of dri­ver you want to write (spe­cially the case stud­ies of real dri­vers). The rea­son is that each chap­ter has some notes which may not only apply to the chap­ter in ques­tion, but to all kinds of device dri­vers. The long review fol­lows now.

The first chap­ter is titled “Build­ing and run­ning mod­ules”. The author begins with descrip­tion of the usual device dri­ver types (NIC dri­ver, pseudo-device, …) and how they can be added to the ker­nel (sta­t­i­cally linked in or as a mod­ule). The first code exam­ple is a small and easy ker­nel mod­ule, so that we do not have to reboot the sys­tem we use to develop a dri­ver (except we make a fault dur­ing dri­ver devel­op­ment which causes the machine to panic or hang). Every part of the exam­ple is well explained. This is fol­lowed by an overview about char­ac­ter devices (e.g. disks) and a sim­ple character-device dri­ver (so far a pseudo-device, as we do not have real hard­ware we access) which is not only as-well explained as the module-example, but there is also a note where the code was sim­pli­fied and what should be done instead.

After read­ing this chap­ter you should be able to write your own ker­nel mod­ule in 5 min­utes (well, after 5 min­utes it will not be able to do a lot — just a “hello world” – but at least you can already load/unload/execute some code into/from/in the kernel).

I have not tried any exam­ple myself, but I com­piled a lot of mod­ules and dri­vers I mod­i­fied in the past and remem­ber to have seen the described parts.

The sec­ond chap­ter explains how to allo­cate and free mem­ory in the ker­nel. There is the pos­si­bil­ity to allo­cate maybe-contiguous mem­ory (the nor­mal case, when your hard­ware does not do DMA or does not have the require­ment that the mem­ory region it makes DMA from/too needs to be con­tigu­ous), and really con­tigu­ous. For the size argu­ment of the free­ing of the the con­tigu­ous mem­ory there is the sen­tence “Gen­er­ally, size should be equal the amount allo­cated.”. Imme­di­ately I wanted to know what hap­pens if you spec­ify a dif­fer­ent size (as a non-native eng­lish speaker I under­stand this sen­tence in a way that I am allowed to spec­ify a dif­fer­ent size and as such are able to free only parts of the allo­cated mem­ory). Unfor­tu­nately this is not answered. I had a look into the source, the ker­nel frees mem­ory pages, so the size argu­ment (and addr argu­ment) will be rounded to include a full page. This means the­o­ret­i­cally I am able to free parts of the allo­cated mem­ory, but this is a source-maintenance night­mare (needs knowl­edge about the machine spe­cific page bound­aries and you need to make sure that you do the absolutely cor­rect size cal­cu­la­tions).  To me this looks more like as long as nobody is point­ing a gun at my head and tells me to use a dif­fer­ent size, spec­i­fy­ing the same size as made dur­ing the allo­ca­tion of this mem­ory region is the way to go.

After read­ing this chap­ter you should know how to kill the sys­tem by allo­cat­ing all the RAM in the kernel.

Again, I did not try to com­pile the exam­ples in this chap­ter, but the dif­fer­ence of the mem­ory allo­ca­tion in the ker­nel com­pared with mem­ory allo­ca­tion in the user­land is not that big.

The third chap­ter explains the device com­mu­ni­ca­tion and con­trol inter­faces (ioctl/sysctl) of a dri­ver. The ioctl part teached me some parts I always wanted to know when I touched some ioctls, but never both­ered to find out before. Unfor­tu­nately this makes me a lit­tle bit ner­vous about the way ioctls are han­dled in the FreeBSD lin­ux­u­la­tor, but this is not urgent ATM (and can prob­a­bly be han­dled by a com­mend in the right place). The sysctl part takes a lit­tle bit longer to fol­low through, but there is also more to learn about it. If you just mod­ify an exist­ing dri­ver with an exist­ing sysctl inter­face, it prob­a­bly just comes down to copy&paste with lit­tle mod­i­fi­ca­tions, but if you need to make more com­plex changes or want to add a sysctl inter­face to a dri­ver, this part of the book is a good way to under­stand what is pos­si­ble and how every­thing fits together. Per­son­ally I would have wished for a more detailed guide when to pick the ioctl inter­face and when the sysctl inter­face than what was writ­ten in the con­clu­sion of the chap­ter, but it is prob­a­bly not that easy to come up with a good list which fits most drivers.

After read­ing this chap­ter you should be able to get data in and out of the ker­nel in 10 minutes.

As before, I did not com­pile the exam­ples in this chap­ter. I already added ioctls and sysctls in var­i­ous places in the FreeBSD kernel.

Chap­ter 4 is about thread syn­chro­niza­tion – mutexes, shared/exclusive locks, reader/writer locks and con­di­tion vari­ables. For me this chap­ter is not as good as the pre­vi­ous ones. While I got a good expla­na­tion of every­thing, I missed a nice overview table which com­pares the var­i­ous meth­ods of thread syn­chro­niza­tion. Bren­dan Gregg did a nice table to give an overview of DTrace vari­able types and when to use them. Some­thing like this would have been nice in this chap­ter too. Apart from this I got all the info I need (but hey, I already wrote a NFS client for an exper­i­men­tal com­puter with more than 200000 CPUs in 1998, so I’m famil­iar with such syn­chro­niza­tion primitives).

Delayed exe­cu­tion is explained in chap­ter 5. Most of the infor­ma­tion pre­sented there was new to me. While there where not much exam­ples pre­sented (there will be some in a later chap­ter), I got a good overview about what exists. This time there was even an overview when to use which type of delayed exe­cu­tion infra­struc­ture. I would have pre­ferred to have this overview in the begin­ning of the chap­ter, but that is maybe some kind of per­sonal preference.

In chap­ter 6 a com­plete device dri­ver is dis­sected. It is the vir­tual null modem ter­mi­nal dri­ver. The chap­ter pro­vides real-world exam­ples of event-handlers, call­outs and taskqueues which where not demon­strated in chap­ter five. At the same time the chap­ter serves as a descrip­tion of the func­tions a TTY dri­ver needs to have.

Auto­mated device detec­tion with New­bus and the cor­re­spond­ing resource allo­ca­tion (I/O ports, device mem­ory and inter­rupts) are explained in chap­ter 7. It is easy… if you have a real device to play with. Unfor­tu­nately the chap­ter missed a para­graph or two about the sus­pend and resume meth­ods. If you think about it, it is not hard to come up with what they are sup­posed to do, but a lit­tle explicit descrip­tion of what they shall do, in what state the hard­ware should be put and what to assume when being called would have been nice.

Chap­ter 8 is about inter­rupts. It is easy to add an inter­rupt han­dler (or to remove one), the hard part is to gen­er­ate an inter­rupt. The exam­ple code uses the par­al­lel port, and the chap­ter also con­tains a lit­tle expla­na­tion how to gen­er­ate an inter­rupt… if you are not afraid to touch real hard­ware (the par­al­lel port) with a resistor.

In chap­ter 9 the lpt(4) dri­ver is explained, as most of the top­ics dis­cussed so far are used inside. The expla­na­tion how every­thing is used is good, but what I miss some­times is why they are used. The most promi­nent (and only) exam­ple here for me is why are call­outs used to catch stray inter­rupts? That call­outs are a good way of han­dling this is clear to me, the big ques­tion is why can there be stray inter­rupts. Can this hap­pen only for the par­al­lel port (respec­tively a lim­ited amount of devices), or does every dri­ver for real inter­rupt dri­ven hard­ware need to come with some­thing like this? I assume this is some­thing spe­cific to the device, but a lit­tle expla­na­tion regard­ing this would have been nice.

Access­ing I/O ports and I/O mem­ory for devices are explained in chap­ter 10 based upon a dri­ver for a LED device (turn on and off 2 LEDs on an ISA bus). All the func­tions to read and write data are well explained, just the part about the mem­ory bar­rier is a lit­tle bit short. It is not clear why the CPU reorder­ing of mem­ory accesses mat­ter to what looks like func­tion calls. Those func­tion calls may be macros, but this is not explained in the text. Some lit­tle exam­ples when to use the bar­ri­ers instead of an abstract descrip­tion would also have been nice at this point.

Chap­ter 11 is sim­i­lar to chap­ter 10, just that a PCI bus dri­ver is dis­cussed instead of an ISA bus dri­ver. The dif­fer­ences are not that big, but important.

In chap­ter 12 it is explained how to do DMA in a dri­ver. This part is not easy to under­stand. I would have wanted to have more exam­ples and expla­na­tions of the DMA tag and DMA map parts. I am also sur­prised to see dif­fer­ent sup­ported archi­tec­tures for the flags BUS_DMA_COHERENT and BUS_DMA_NOCACHE for dif­fer­ent func­tions. Either this means FreeBSD is not coher­ent in those parts, or it is a bug in the book, or it is sup­posed to be like this and the rea­sons are not explained in the book. As there is no explicit note about this, it prob­a­bly leads to con­fu­sion of read­ers which pay enough atten­tion here. It would also have been nice to have an expla­na­tion when to use those flags which are only imple­mented on a sub­set of the archi­tec­tures FreeBSD sup­ports. Any­way, the expla­na­tions give enough infor­ma­tion to under­stand what is going on and to be able to have a look at other device dri­vers for real-live exam­ples and to get a deeper under­stand­ing of this topic.

Disk dri­vers and block I/O (bio) requests are described in chap­ter 13. With this chap­ter I have a lit­tle prob­lem. The author used the word “unde­fined” in sev­eral places where I as a non-native speaker would have used “not set” or “set to 0″. The word “unde­fined” implies for me that there may be garbage inside, whereas from a tech­ni­cal point of view I can not imag­ine that some ran­dom value in those places would have the desired result. In my opin­ion each such place is obvi­ous, so I do not expect that an expe­ri­enced pro­gram­mer would lose time/hairs/sanity over it, but inex­pe­ri­enced pro­gram­mers which try to assem­ble the cor­re­spond­ing struc­tures on the (unini­tial­ized) heap (for what­ever rea­son), may strug­gle with this.

Chap­ter 14 is about the CAM layer. While the pre­vi­ous chap­ter showed how to write a dri­ver for a disk device, chap­ter 14 gave an overview about how to an HBA to the CAM layer. It is just an overview, it looks like CAM needs a book on its own to be fully described. The sim­ple (and most impor­tant) cases are described, with the hardware-specific parts being an exer­cise for the per­son writ­ing the device dri­ver. I have the impres­sion it gives enough details to let some­one with hard­ware (or pro­to­col), and more impor­tantly doc­u­men­ta­tion for this device, start writ­ing a driver.

It would have been nice if chap­ter 13 and 14 would have had a lit­tle schematic which describes at which level of the kernel-subsystems the cor­re­spond­ing dri­ver sits. And while I am at it, a schematic with all the dri­ver com­po­nents dis­cussed in this book at the begin­ning as an overview, or in the end as an annex, would be great too.

An overview of USB dri­vers is given in chap­ter 15 with the USB printer dri­ver as an exam­ple for the expla­na­tion of the USB dri­ver inter­faces. If USB would not be as com­plex as it is, it would be a nice chap­ter to start driver-writing exper­i­ments (due to the avail­abil­ity of var­i­ous USB devices). Well… bad luck for curi­ous peo­ple. BTW, the author gives point­ers to the offi­cial USB docs, so if you are really curi­ous, feel free to go ahead. :)

Chap­ter 16 is the first part about net­work dri­vers. It deals with ifnet (e.g. stuff needed for ifcon­fig), ifme­dia (sim­pli­fied: which kind of cable and speed is sup­ported), mbufs and MSI(-X). As in other chap­ters before, a lit­tle overview and a lit­tle pic­ture in the begin­ning would have been nice.

Finally, in chap­ter 17, the packet recep­tion and trans­mis­sion of net­work dri­vers is described. Large exam­ple code is bro­ken up into sev­eral pieces here, for more easy dis­cus­sion of related information.

One thing I miss after reach­ing the end of the book is a dis­cus­sion of sound dri­vers. And this is surely not the only type of dri­vers which is not dis­cussed, I can come up with crypto, firewire, gpio, watch­dog, smb and iic devices within a few sec­onds. While I think that it is much more easy to under­stand all those dri­vers now after read­ing the book, it would have been nice to have at least a lit­tle overview of other dri­ver types and maybe even a short descrip­tion of their dri­ver methods.

Con­clu­sion: As I wrote already in the begin­ning, the book is not per­fect, but it is good. While I have not writ­ten a device dri­ver for FreeBSD, the book pro­vided enough insight to be able to write one and to under­stand exist­ing dri­vers. I really hope there will be a sec­ond edi­tion which addresses the minor issues I had while read­ing it to make it a per­fect book.

GD Star Rat­ing
GD Star Rat­ing

Tags: , , , , , , , , ,

Are USB mem­ory sticks really that bad?

Last week my ZFS cache device — an USB mem­ory stick — showed xxxM write errors. I got this stick for free as a promo, so I do not expect it to be of high qual­ity (or wear-leveling or sim­i­lar life-saving things). The stick sur­vived about 9 months, dur­ing which it pro­vided a nice speed-up for the access to the cor­re­spond­ing ZFS stor­age pool. I replaced it by another stick which I got for free as a promo. This new stick sur­vived… one long week­end. It has now 8xxM write errors and the USB sub­sys­tem is not able to speak to it any­more. 30 min­utes ago I issued an “usb­con­fig reset” to this device, which is still not fin­ished. This leads me to the ques­tion if such sticks are really that bad, or if some prob­lem crept into the USB subsystem?

If this is a prob­lem with the mem­ory stick itself, I should be able to repro­duce such a prob­lem on a dif­fer­ent machine with a dif­fer­ent OS. I could test this with FreeBSD 8.1, Solaris 10u9, or Win­dows XP. What I need is an auto­mated test. This rules out the Win­dows XP machine for me, I do not want to spend time to search a suit­able test which is avail­able for free and allows to be run in an auto­mated way. For FreeBSD and Solaris it prob­a­bly comes down to use some disk-I/O bench­mark (I think there are enough to chose from in the FreeBSD Ports Col­lec­tion) and run it in a shell-loop.

GD Star Rat­ing
GD Star Rat­ing

Tags: , , , , , , , , ,

Round-up of recent FreeBSD work

I had a look at some USB PRs and wrote a list of those with patches to Warner (as he is work­ing on USB stuff cur­rently). I also cat­e­go­rized them (easy, not easy, maybe already fixed, …). The easy ones he han­dled already, for the rest I don’t know his cur­rent plans.

Regard­ing lin­ux­u­la­tor stuff I’m work­ing on a MFC patch (no TLS, no futexes). As I don’t have a -sta­ble box I need some help test­ing it before I can com­mit it. I only com­pile tested this on –cur­rent with the new gcc 4.2. What I need is:

  • test­ing on i386, amd64 (if I for­got some­thing, it may panic your system)
  • make uni­verse” test (you have to grep all the logs for “Error 1″ and inves­ti­gate the error if there’s one)
  • LTP test run, see the wiki for more (best would be a diff of the logs in the result direc­tory of no-patch/patch runs)
  • nor­mal linux appli­ca­tion use-tests

What the patch pro­vides is:

  • mmap fixes
  • fix mem­leaks
  • add mprotect/iopl/lstat/ftruncate/statfs64/timer_*/mq_*
  • more errno value mapping
  • don’t limit num­ber of syscalls to 255
  • allow to exec libs
  • ioctl TIOCGPTN
  • han­dle more socket options
  • de-COMPAT_43-ify
  • add dummy syscalls so that we know what is needed (reports from users)
  • style(9)
  • lin­procfs enhance­ments
GD Star Rat­ing
GD Star Rat­ing

Tags: , , , , , , , , ,

USB PR shooting

I did shoot down some USB related PR’s today. We have now sup­port for more devices in uscan­ner, umo­dem and one new PDA in uvi­sor. Addi­tion­ally all peo­ple which use the con­troller of a XBOX 360 will now notice that the LEDs on them stop blink­ing, when the uhid dri­ver attaches to it (like on the XBOX 360).

This affects -cur­rent and -sta­ble.

GD Star Rat­ing
GD Star Rat­ing

Tags: , , ,