Book re­view: FreeBSD Device Drivers

In mid-​April a wo­man from the mar­ket­ing de­part­ment of No Starch Press con­tac­ted me and asked if I am in­ter­ested to do a pub­lic re­view of the FreeBSD Device Drivers book by Joseph Kong (no link to a book shop, go and have a look in your pre­ferred one). Just this sim­ple ques­tion, no strings at­tached.

I had my nose in some device drivers in the past, but I never wrote one, and never had a look at the big pic­ture. I was in­ter­ested to know how everything fits to­gether, so this made me a good vic­tim for a re­view (novice enough to learn some­thing new and to have a look if enough is ex­plained, and ex­per­i­enced enough to un­der­stand what is go­ing on in the FreeBSD ker­nel).

Some minutes af­ter I agreed to re­view it (but with a little no­tice that I do not know how long I need to re­view it), I had the PDF ver­sion of the book. That was faster than I ex­pec­ted (maybe I am too old-​school and used to have pa­per ver­sions of books in my hands).

Let the re­view be­gin… but bear with me, this is the first time I do a real pub­lic re­view of a book (in­stead of a tech­nical re­view for an au­thor). And as this is my very own per­sonal opin­ion, I will not al­low com­ments here. This page is all about my opin­ion while read­ing the book, ques­tions I have while read­ing the book shall serve as a hint about the qual­ity of the book and they should be answered in the book, not here.

In short, the book is not per­fect, but it is a good book. There is room for im­prove­ment, but on a very high level. If you want to write a device driver for FreeBSD, this book is a must. I sug­gest to read it com­pletely, even chapters which do not be­long to the type of driver you want to write (spe­cially the case stud­ies of real drivers). The reason is that each chapter has some notes which may not only ap­ply to the chapter in ques­tion, but to all kinds of device drivers. The long re­view fol­lows now.

The first chapter is titled “Build­ing and run­ning mod­ules”. The au­thor be­gins with de­scrip­tion of the usual device driver types (NIC driver, pseudo-​device, …) and how they can be ad­ded to the ker­nel (stat­ic­ally linked in or as a mod­ule). The first code ex­ample is a small and easy ker­nel mod­ule, so that we do not have to re­boot the sys­tem we use to de­velop a driver (ex­cept we make a fault dur­ing driver de­vel­op­ment which causes the ma­chine to panic or hang). Every part of the ex­ample is well ex­plained. This is fol­lowed by an over­view about char­ac­ter devices (e.g. disks) and a sim­ple character-​device driver (so far a pseudo-​device, as we do not have real hard­ware we ac­cess) which is not only as-​well ex­plained as the module-​example, but there is also a note where the code was sim­pli­fied and what should be done in­stead.

Af­ter read­ing this chapter you should be able to write your own ker­nel mod­ule in 5 minutes (well, af­ter 5 minutes it will not be able to do a lot — just a “hello world” — but at least you can already load/​unload/​execute some code into/​from/​in the ker­nel).

I have not tried any ex­ample my­self, but I com­piled a lot of mod­ules and drivers I mod­i­fied in the past and re­mem­ber to have seen the de­scribed parts.

The second chapter ex­plains how to al­loc­ate and free memory in the ker­nel. There is the pos­sib­il­ity to al­loc­ate maybe-​contiguous memory (the nor­mal case, when your hard­ware does not do DMA or does not have the re­quire­ment that the memory re­gion it makes DMA from/​too needs to be con­tigu­ous), and really con­tigu­ous. For the size ar­gu­ment of the free­ing of the the con­tigu­ous memory there is the sen­tence “Gen­er­ally, size should be equal the amount al­loc­ated.”. Im­me­di­ately I wanted to know what hap­pens if you spe­cify a dif­fer­ent size (as a non-​native eng­lish speaker I un­der­stand this sen­tence in a way that I am al­lowed to spe­cify a dif­fer­ent size and as such are able to free only parts of the al­loc­ated memory). Un­for­tu­nately this is not answered. I had a look into the source, the ker­nel frees memory pages, so the size ar­gu­ment (and addr ar­gu­ment) will be roun­ded to in­clude a full page. This means the­or­et­ic­ally I am able to free parts of the al­loc­ated memory, but this is a source-​maintenance night­mare (needs know­ledge about the ma­chine spe­cific page bound­ar­ies and you need to make sure that you do the ab­so­lutely cor­rect size cal­cu­la­tions).  To me this looks more like as long as nobody is point­ing a gun at my head and tells me to use a dif­fer­ent size, spe­cify­ing the same size as made dur­ing the al­loc­a­tion of this memory re­gion is the way to go.

Af­ter read­ing this chapter you should know how to kill the sys­tem by al­loc­at­ing all the RAM in the ker­nel.

Again, I did not try to com­pile the ex­amples in this chapter, but the dif­fer­ence of the memory al­loc­a­tion in the ker­nel com­pared with memory al­loc­a­tion in the user­land is not that big.

The third chapter ex­plains the device com­mu­nic­a­tion and con­trol in­ter­faces (ioctl/​sysctl) of a driver. The ioctl part teached me some parts I al­ways wanted to know when I touched some ioctls, but never bothered to find out be­fore. Un­for­tu­nately this makes me a little bit nervous about the way ioctls are handled in the FreeBSD linuxu­lator, but this is not ur­gent ATM (and can prob­ably be handled by a com­mend in the right place). The sy­sctl part takes a little bit longer to fol­low through, but there is also more to learn about it. If you just modify an ex­ist­ing driver with an ex­ist­ing sy­sctl in­ter­face, it prob­ably just comes down to copy&paste with little modi­fic­a­tions, but if you need to make more com­plex changes or want to add a sy­sctl in­ter­face to a driver, this part of the book is a good way to un­der­stand what is pos­sible and how everything fits to­gether. Per­son­ally I would have wished for a more de­tailed guide when to pick the ioctl in­ter­face and when the sy­sctl in­ter­face than what was writ­ten in the con­clu­sion of the chapter, but it is prob­ably not that easy to come up with a good list which fits most drivers.

Af­ter read­ing this chapter you should be able to get data in and out of the ker­nel in 10 minutes.

As be­fore, I did not com­pile the ex­amples in this chapter. I already ad­ded ioctls and sy­sctls in vari­ous places in the FreeBSD ker­nel.

Chapter 4 is about thread syn­chron­iz­a­tion — mu­texes, shared/​exclusive locks, reader/​writer locks and con­di­tion vari­ables. For me this chapter is not as good as the pre­vi­ous ones. While I got a good ex­plan­a­tion of everything, I missed a nice over­view table which com­pares the vari­ous meth­ods of thread syn­chron­iz­a­tion. Brendan Gregg did a nice table to give an over­view of DTrace vari­able types and when to use them. Some­thing like this would have been nice in this chapter too. Apart from this I got all the info I need (but hey, I already wrote a NFS cli­ent for an ex­per­i­mental com­puter with more than 200000 CPUs in 1998, so I’m fa­mil­iar with such syn­chron­iz­a­tion prim­it­ives).

Delayed ex­e­cu­tion is ex­plained in chapter 5. Most of the in­form­a­tion presen­ted there was new to me. While there where not much ex­amples presen­ted (there will be some in a later chapter), I got a good over­view about what ex­ists. This time there was even an over­view when to use which type of delayed ex­e­cu­tion in­fra­struc­ture. I would have pre­ferred to have this over­view in the be­gin­ning of the chapter, but that is maybe some kind of per­sonal pref­er­ence.

In chapter 6 a com­plete device driver is dis­sec­ted. It is the vir­tual null mo­dem ter­minal driver. The chapter provides real-​world ex­amples of event-​handlers, cal­louts and taskqueues which where not demon­strated in chapter five. At the same time the chapter serves as a de­scrip­tion of the func­tions a TTY driver needs to have.

Auto­mated device de­tec­tion with New­bus and the cor­res­pond­ing re­source al­loc­a­tion (I/​O ports, device memory and in­ter­rupts) are ex­plained in chapter 7. It is easy… if you have a real device to play with. Un­for­tu­nately the chapter missed a para­graph or two about the sus­pend and re­sume meth­ods. If you think about it, it is not hard to come up with what they are sup­posed to do, but a little ex­pli­cit de­scrip­tion of what they shall do, in what state the hard­ware should be put and what to as­sume when be­ing called would have been nice.

Chapter 8 is about in­ter­rupts. It is easy to add an in­ter­rupt hand­ler (or to re­move one), the hard part is to gen­er­ate an in­ter­rupt. The ex­ample code uses the par­al­lel port, and the chapter also con­tains a little ex­plan­a­tion how to gen­er­ate an in­ter­rupt… if you are not afraid to touch real hard­ware (the par­al­lel port) with a res­istor.

In chapter 9 the lpt(4) driver is ex­plained, as most of the top­ics dis­cussed so far are used in­side. The ex­plan­a­tion how everything is used is good, but what I miss some­times is why they are used. The most prom­in­ent (and only) ex­ample here for me is why are cal­louts used to catch stray in­ter­rupts? That cal­louts are a good way of hand­ling this is clear to me, the big ques­tion is why can there be stray in­ter­rupts. Can this hap­pen only for the par­al­lel port (re­spect­ively a lim­ited amount of devices), or does every driver for real in­ter­rupt driven hard­ware need to come with some­thing like this? I as­sume this is some­thing spe­cific to the device, but a little ex­plan­a­tion re­gard­ing this would have been nice.

Ac­cess­ing I/​O ports and I/​O memory for devices are ex­plained in chapter 10 based upon a driver for a LED device (turn on and off 2 LEDs on an ISA bus). All the func­tions to read and write data are well ex­plained, just the part about the memory bar­rier is a little bit short. It is not clear why the CPU re­order­ing of memory ac­cesses mat­ter to what looks like func­tion calls. Those func­tion calls may be mac­ros, but this is not ex­plained in the text. Some little ex­amples when to use the bar­ri­ers in­stead of an ab­stract de­scrip­tion would also have been nice at this point.

Chapter 11 is sim­ilar to chapter 10, just that a PCI bus driver is dis­cussed in­stead of an ISA bus driver. The dif­fer­ences are not that big, but im­port­ant.

In chapter 12 it is ex­plained how to do DMA in a driver. This part is not easy to un­der­stand. I would have wanted to have more ex­amples and ex­plan­a­tions of the DMA tag and DMA map parts. I am also sur­prised to see dif­fer­ent sup­por­ted ar­chi­tec­tures for the flags BUS_​DMA_​COHERENT and BUS_​DMA_​NOCACHE for dif­fer­ent func­tions. Either this means FreeBSD is not co­her­ent in those parts, or it is a bug in the book, or it is sup­posed to be like this and the reas­ons are not ex­plained in the book. As there is no ex­pli­cit note about this, it prob­ably leads to con­fu­sion of read­ers which pay enough at­ten­tion here. It would also have been nice to have an ex­plan­a­tion when to use those flags which are only im­ple­men­ted on a sub­set of the ar­chi­tec­tures FreeBSD sup­ports. Any­way, the ex­plan­a­tions give enough in­form­a­tion to un­der­stand what is go­ing on and to be able to have a look at other device drivers for real-​live ex­amples and to get a deeper un­der­stand­ing of this topic.

Disk drivers and block I/​O (bio) re­quests are de­scribed in chapter 13. With this chapter I have a little prob­lem. The au­thor used the word “un­defined” in sev­eral places where I as a non-​native speaker would have used “not set” or “set to 0”. The word “un­defined” im­plies for me that there may be garbage in­side, whereas from a tech­nical point of view I can not ima­gine that some ran­dom value in those places would have the de­sired res­ult. In my opin­ion each such place is ob­vi­ous, so I do not ex­pect that an ex­per­i­enced pro­gram­mer would lose time/​hairs/​sanity over it, but in­ex­per­i­enced pro­gram­mers which try to as­semble the cor­res­pond­ing struc­tures on the (un­ini­tial­ized) heap (for whatever reason), may struggle with this.

Chapter 14 is about the CAM layer. While the pre­vi­ous chapter showed how to write a driver for a disk device, chapter 14 gave an over­view about how to an HBA to the CAM layer. It is just an over­view, it looks like CAM needs a book on its own to be fully de­scribed. The sim­ple (and most im­port­ant) cases are de­scribed, with the hardware-​specific parts be­ing an ex­er­cise for the per­son writ­ing the device driver. I have the im­pres­sion it gives enough de­tails to let someone with hard­ware (or pro­to­col), and more im­port­antly doc­u­ment­a­tion for this device, start writ­ing a driver.

It would have been nice if chapter 13 and 14 would have had a little schem­atic which de­scribes at which level of the kernel-​subsystems the cor­res­pond­ing driver sits. And while I am at it, a schem­atic with all the driver com­pon­ents dis­cussed in this book at the be­gin­ning as an over­view, or in the end as an an­nex, would be great too.

An over­view of USB drivers is given in chapter 15 with the USB printer driver as an ex­ample for the ex­plan­a­tion of the USB driver in­ter­faces. If USB would not be as com­plex as it is, it would be a nice chapter to start driver-​writing ex­per­i­ments (due to the avail­ab­il­ity of vari­ous USB devices). Well… bad luck for curi­ous people. BTW, the au­thor gives point­ers to the of­fi­cial USB docs, so if you are really curi­ous, feel free to go ahead. 🙂

Chapter 16 is the first part about net­work drivers. It deals with ifnet (e.g. stuff needed for if­con­fig), if­me­dia (sim­pli­fied: which kind of cable and speed is sup­por­ted), mbufs and MSI(-X). As in other chapters be­fore, a little over­view and a little pic­ture in the be­gin­ning would have been nice.

Fi­nally, in chapter 17, the packet re­cep­tion and trans­mis­sion of net­work drivers is de­scribed. Large ex­ample code is broken up into sev­eral pieces here, for more easy dis­cus­sion of re­lated in­form­a­tion.

One thing I miss af­ter reach­ing the end of the book is a dis­cus­sion of sound drivers. And this is surely not the only type of drivers which is not dis­cussed, I can come up with crypto, firewire, gpio, watch­dog, smb and iic devices within a few seconds. While I think that it is much more easy to un­der­stand all those drivers now af­ter read­ing the book, it would have been nice to have at least a little over­view of other driver types and maybe even a short de­scrip­tion of their driver meth­ods.

Con­clu­sion: As I wrote already in the be­gin­ning, the book is not per­fect, but it is good. While I have not writ­ten a device driver for FreeBSD, the book provided enough in­sight to be able to write one and to un­der­stand ex­ist­ing drivers. I really hope there will be a second edi­tion which ad­dresses the minor is­sues I had while read­ing it to make it a per­fect book.

Are USB memory sticks really that bad?

Last week my ZFS cache device — an USB memory stick — showed xxxM write er­rors. I got this stick for free as a promo, so I do not ex­pect it to be of high qual­ity (or wear-​leveling or sim­ilar life-​saving things). The stick sur­vived about 9 months, dur­ing which it provided a nice speed-​up for the ac­cess to the cor­res­pond­ing ZFS stor­age pool. I re­placed it by an­other stick which I got for free as a promo. This new stick sur­vived… one long week­end. It has now 8xxM write er­rors and the USB sub­sys­tem is not able to speak to it any­more. 30 minutes ago I is­sued an “us­b­con­fig re­set” to this device, which is still not fin­ished. This leads me to the ques­tion if such sticks are really that bad, or if some prob­lem crept into the USB sub­sys­tem?

If this is a prob­lem with the memory stick it­self, I should be able to re­pro­duce such a prob­lem on a dif­fer­ent ma­chine with a dif­fer­ent OS. I could test this with FreeBSD 8.1, Sol­aris 10u9, or Win­dows XP. What I need is an auto­mated test. This rules out the Win­dows XP ma­chine for me, I do not want to spend time to search a suit­able test which is avail­able for free and al­lows to be run in an auto­mated way. For FreeBSD and Sol­aris it prob­ably comes down to use some disk-​I/​O bench­mark (I think there are enough to chose from in the FreeBSD Ports Col­lec­tion) and run it in a shell–loop.

Round-​up of re­cent FreeBSD work

I had a look at some USB PRs and wrote a list of those with patches to Warner (as he is work­ing on USB stuff cur­rently). I also cat­egor­ized them (easy, not easy, maybe already fixed, …). The easy ones he handled already, for the rest I don’t know his cur­rent plans.

Re­gard­ing linuxu­lator stuff I’m work­ing on a MFC patch (no TLS, no fu­texes). As I don’t have a -stable box I need some help test­ing it be­fore I can com­mit it. I only com­pile tested this on –cur­rent with the new gcc 4.2. What I need is:

  • test­ing on i386, amd64 (if I for­got some­thing, it may panic your sys­tem)
  • “make uni­verse” test (you have to grep all the logs for “Er­ror 1” and in­vest­ig­ate the er­ror if there’s one)
  • LTP test run, see the wiki for more (best would be a diff of the logs in the res­ult dir­ect­ory of no-​patch/​patch runs)
  • nor­mal linux ap­plic­a­tion use-​tests

What the patch provides is:

  • mmap fixes
  • fix mem­leaks
  • add mprotect/​iopl/​lstat/ftruncate/statfs64/timer_*/mq_*
  • more er­rno value map­ping
  • don’t limit num­ber of sy­scalls to 255
  • al­low to exec libs
  • ioctl TIOCGPTN
  • handle more socket op­tions
  • de-​COMPAT_​43-​ify
  • add dummy sy­scalls so that we know what is needed (re­ports from users)
  • style(9)
  • lin­procfs en­hance­ments

USB PR shoot­ing

I did shoot down some USB re­lated PR’s today. We have now sup­port for more devices in uscan­ner, umo­dem and one new PDA in uvisor. Ad­di­tion­ally all people which use the con­trol­ler of a XBOX 360 will now no­tice that the LEDs on them stop blink­ing, when the uhid driver at­taches to it (like on the XBOX 360).

This af­fects -cur­rent and -stable.