ADSL RAM… the­o­ry and real­i­ty do not match

I am now wait­ing since Decem­ber that my ADSL line is switched to the rate adap­tive mode (RAM). The­o­ret­i­cal­ly it is pos­si­ble. Unfor­tu­nate­ly the real­i­ty does not agree to this (yet).

Luck­i­ly I am not a nor­mal cus­tomer, I know a tech­ni­cian which works for my ISP. He could switch the line with­out prob­lems, but the next update of the sys­tem (which hap­pens from time to time) would can­cel this again, as each update “resets” the sta­tus to what is record­ed in the DB. The prob­lem is, that he can not switch my line to RAM in the DB (actu­al­ly it is not him, he is a net­work tech­ni­cian not one of the sales peo­ple with access to the DB-interface). I am not the only cus­tomer where this is not pos­si­ble. So far they where not able to see a pattern.

Cur­rent­ly there are two col­leagues of him, a friend of him and me which he has as good exam­ples where it does not work (there are more, but those are “just” reg­u­lar cus­tomers). We are now his toys, he wants to find out how to con­vince the sys­tem to switch to RAM in those cas­es. This needs a while, as parts of this need to go the offi­cial way until he sees if it works or not.

I am very hap­py that I am not just a nor­mal cus­tomer. This way it is much more trans­par­ent for me.

AQM/ECN in FreeBSD

After read­ing Jim Get­tys inves­ti­ga­tions about the prob­lems cur­rent buffer sizes of net­work equip­ment pro­voke (which may even have impli­ca­tions in the net neu­tral­i­ty debate), I had a look at which active queue man­age­ment (AQM) algo­rithms with or with­out explic­it con­ges­tion noti­fi­ca­tion (ECN) FreeB­SD supports.

It looks like there is not much imple­ment­ed (if the best solu­tion would be imple­ment­ed, it would not mat­ter how much there is, but unfor­tu­nate­ly there is no best solu­tion). Oth­er sys­tems offer more. RED is imple­ment­ed, but even the inventor/researcher of RED thinks the algo­rithm needs some improve­ments (he is in the process of prepar­ing a paper about this, as Jim Get­tys reveals). Blue/SFBlue is not imple­ment­ed (a more turnkey-solution than the cur­rent RED imple­men­ta­tion). PID con­troller (which may or may not be some­thing some­one wants to use in this case… no idea about its pros/cons in this regard, but it is ref­er­enced in the AQM arti­cle on Wikipedia) is also not implemented.

Regard­ing ECN for FreeB­SD you can find more or less no real doc­u­men­ta­tion in the net (at least with a sim­ple “ECN FreeB­SD” search). It is imple­ment­ed for the RED algo­rithm, but as the RED algo­rithm needs some tuning/setup, this is not a turnkey solu­tion. There is a ECN relat­ed sysctl, but I do not have the impres­sion that this is a turnkey-solution which mag­i­cal­ly gen­er­ates ECN mes­sages with­out using dum­mynet for AQM.

From my cur­rent under­stand­ing (but I think I do not know a lot about this top­ic) it looks like AQM is a fea­ture most peo­ple would like to have acti­vat­ed by default (with an appro­pri­ate algo­rithm which does not need tun­ing to pro­duce a good enough result). If this is cor­rect, it is a shame that FreeB­SD does not acti­vate AQM with an algo­rithm which is not bad for most cas­es by default (with the option to change the algo­rithm and to dis­able com­plete­ly). If my under­stand­ing is not cor­rect, I would like to get a hit with the clue bat please.

How big are the buffers in FreeB­SD drivers?

Today I have read an inter­est­ing inves­ti­ga­tion and prob­lem analy­sis from Jim Get­tys.

It is a set of arti­cles he wrote over sev­er­al months and is not fin­ished writ­ing as of this writ­ing (if you are deeply inter­est­ed in it go and read them, the most inter­est­ing ones are from Decem­ber and Jan­u­ary and the com­ments to the arti­cles are also con­tribut­ing to the big pic­ture). Basi­cal­ly he is telling that a lot of net­work prob­lems users at home (with ADSL/cable or WLAN) expe­ri­ence  are because buffers in the net­work hard­ware or in oper­at­ing sys­tems are too big. He also pro­pos­es workarounds until this prob­lem is attacked by OS ven­dors and equip­ment manufacturers.

Basi­cal­ly he is telling the net­work con­ges­tion algo­rithms can not do their work good, because the net­work buffers which are too big come into the way of their work (not report­ing pack­et loss time­ly enough respec­tive­ly try to not lose pack­ets in sit­u­a­tions where pack­et loss would be bet­ter because it would trig­ger action in the con­ges­tion algorithms).

He inves­ti­gat­ed the behav­ior of Lin­ux, OS X and Win­dows (the sys­tem he had avail­able). I want­ed to have a quick look at the sit­u­a­tion in FreeB­SD regard­ing this, but it seems at least with my net­work card I am not able to see/find the cor­re­spond­ing size of the buffers in dri­vers in 30 seconds.

I think it would be very good if this issue is inves­ti­gat­ed in FreeB­SD, and apart from maybe tak­ing some action in the source also write some sec­tion for the hand­book which explains the issue (one prob­lem here is, that there are sit­u­a­tions where you want/need to have such big buffers and as such we can not just down­size them) and how to bench­mark and tune this.

Unfor­tu­nate­ly I even have too much on my plate to even fur­ther look into this. 🙁 I hope one of the net­work peo­ple in FreeB­SD is pick­ing up the ball and starts playing.

Sony Bravia and HD videos (via DLNA)

I made some more tests which video res­o­lu­tions my TV accepts via DLNA. While I was look­ing before a SD res­o­lu­tions, this time I took care about some HD resolutions.

As the Sin­tel video in the 1024×436 res­o­lu­tion did not play, I tried to reen­code it to 1024×720 (for the enabled x264 options see below). This did not work either. After that I went to the offi­cial res­o­lu­tion of 1280×720, and this works. Ini­tial­ly this video was encod­ed as High@L3.1, but with this the TV pro­duced some arti­facts on play­back. After chang­ing this to High@L4.0 (sim­ply by remux­ing instead of reen­cod­ing), the play­back was fine (warn­ing: increas­ing the H.264 lev­el is OK, decreas­ing it if the video does not com­ply to the low­ered lev­el, may cause prob­lems). I miss a set­ting in avide­mux for the lev­el, it would be nice if there would be the pos­si­bil­i­ty to set it.

I also test­ed if the 1280×544 ver­sion of the Sin­tel video plays fine on the TV or not. It does not play fine, so there is prob­a­bly a hard require­ment on the com­plete res­o­lu­tion for HD video.

While doing this I noticed that tsMux­eR is trun­cat­ing the audio, instead of the 6 chan­nel audio it was before, the remuxed file has only two channels.

As I did not want to always go through all the set­tings to enter what I want, I made a lit­tle avidemux-script to set­up (ECMA script + xml) every­thing for me. This was easy, I just took an exist­ing one (the Sony PSP one) as a base and changed the encod­ing options and the tar­get con­tain­er (unluck­i­ly avide­mux 2.5.4 does not sup­port H.264 in MPEG-TS yet, so I have to use a MP4 con­tain­er and remux it into the MPEG-TS stream afterwards).

The options I used for the x264-reencoding are –8x8dct –analyse all –mixed-refs –bime –weightb –subme 9 –b‑rdo –ref 4 –b‑adapt 2 –bframes 4 –direct auto –me umh (this includes b‑pyramid, for which there are reports that it does not work).

Stream­ing video to a Sony Bravia (via DLNA)

I have a Sony Bravia TV with a net­work (eth­er­net) con­nec­tion. Let­ting aside the fact that he is in anoth­er sub­net than my NAS and as such I can not use a DLNA (UPnP-AV) serv­er (it is using mul­ti­cast and the Sim­ple Ser­vice Dis­cov­ery Pro­to­col (SSDP) which is not tra­vers­ing sub­nets in the WLAN-LAN-ADSL-router I use), there is no good expla­na­tion from Sony what I can feed to the TV.

When search­ing the net I can find some obscure sug­ges­tions and descrip­tions, but not all of them work for all peo­ple which try them. So it seems I have to research this myself. Luck­i­ly my router has a build-in UPnP-AV serv­er which I can use to play around (the file size is lim­it­ed to the size of the USB mem­o­ry stick I have con­nect­ed to the router, as he can not stream con­tent which is avail­able in a NAS in the network).

Sony tells the TV is able to receive MPEG2 TS and PS con­tain­ers. MPEG2 PS is more or less what you have on DVD. It is able to play SD and HD con­tent from the PS con­tain­er, and the video for­mat needs to be MPEG2. They do not explain what SD or HD means in this con­text (the valid res­o­lu­tions), and they do not tell what kind of audio is allowed.

For the MPEG2 TS con­tain­er they addi­tion­al­ly allow H.264 video, and they spec­i­fy EU, EU‑T and EU-ISO as sup­port­ed in this case. Again, they do not explain what those EU* parts are sup­posed to mean.

For the unspec­i­fied audio I assume this means AC‑3, AAC and MPEG Audio Lay­er 2 (some peo­ple use MP2 to describe this audio for­mat). I suc­cess­ful­ly test­ed AAC and AC‑3, and I have read that MP2 works too. Based upon my expe­ri­ences with the video part (more below) I assume the sam­pling rate and bitwidth mat­ter. So far I tried with 48 kHz and 16 bit per channel.

For the EU* parts I have not found any trust­wor­thy ref­er­ence what this could mean, but it looks this refers to some aspects of DVB/DVB‑T/DVB‑S(2) as it is used in Europe. I guess this is a bit linked with valid res­o­lu­tions the TV is able to handle.

For the video part I have found mixed reports. From hat I have read in the Wikipedia page of the H.264 video for­mat, I assumed the fail­ures are relat­ed to a wrong res­o­lu­tion and maybe the fact that some parts of the video do not con­form to spe­cif­ic “lev­els” of the H.264 for­mat. Some tell you are not allowed to use more than X ref­er­ence frames, some tell you can not use advanced fea­ture Y.

The first test I did was to take the Sin­tel video from Blender. I down­loaded the MP4 ver­sion and remuxed it into a MPEG2-TS con­tain­er (I used tsMux­eR to do this). The TV was able to play the AAC audio, but it did not show the video. When I look at the video prop­er­ties, I see that it has a res­o­lu­tion of 1280×544 at 24 FPS. For H.264 videos which use the “High” pro­file and are com­pat­i­ble up to lev­el 4.1, I do not see this res­o­lu­tion list­ed as valid in the Wikipedia page. Valid res­o­lu­tions are 1280×720 at 30, 60 and 68.3 FPS, and 1280×1024 at 42.2 FPS. This could explain why it is not working.

As a sec­ond test I took a video resized it to 624×256 (I did not pay much atten­tion to the scale in the pro­gram I used to test this, I just hoped it takes a good one, now that I try to sum­ma­rize what I inves­ti­gat­ed so far, I see that this size is not one of the valid sizes list­ed in the lev­els for H.264) at 23.976 FPS and reen­cod­ed it with the default options of the x264 encoder. The result­ing video played just fine on the TV.

My third test was to encode a resized (from 640×272 to 640×480) video at 23.976 FPS with weight­ed, upto 4 adap­tive B‑frames, mixed ref­er­ences, 4 ref­er­ence frames, and some oth­er options (this includes B‑pyramid, which seems to be enabled by default). And again, the result­ing video played just fine (despite the fact that I found com­ments in the net which sug­gest that B‑pyramid needs to be disabled…).

I still have to test some HD sizes, but it looks like one key aspect for com­pat­i­bil­i­ty is that the video is encod­ed with the right res­o­lu­tion (I have to admit, I do not real­ly know what this means, as one video had a res­o­lu­tion which was dif­fer­ent from what the valid sizes for the H.264 lev­els are) and com­plies to only lev­el 4.1 (or 4) and below (basi­cal­ly this means to obey some bitrate lim­i­ta­tions and the num­ber of max. ref­er­ence frames for the giv­en res­o­lu­tion). The 23.976 FPS I men­tioned above are not list­ed as one of the valid FPS in the lev­els for H.264, so I do not think the FPS have to strict­ly con­form to what is spec­i­fied for the lev­els. It looks more this is just an upper lim­it so that the video could also use some low­er FPS.

So far I took the sin­tel video and added some black bor­ders on top and below to get to the 1080×720 res­o­lu­tion. I used avide­mux 2.5.4 for this. The ver­sion I tried can only pro­duce a MP4 con­tain­er with this video/audio com­bi­na­tion, and my hope was to remux it with tsMux­eR to a MPEG2-TS, but tsMux­eR does not find a valid video or audio stream inside the MP4 con­tain­er. I am still search­ing for a pro­gram which is able to remux the result­ing MP4 into a MPEG2-TS. I found a tool which extracts the streams from the MP4 con­tain­er, but the only free MPEG2-TS mux­er seems to be tsMux­eR, which I was not able to con­vince to mux the two streams into a MPEG2-TS file. It seems I have either to wait until avide­mux knows how to gen­er­ate MPEG2-TS streams with H.264 and AAC, or until it gen­er­ates a MP4 con­tain­er with H.264 and AAC which tsMux­eR is able to handle.

What I also want to test is, if I can use the open-gop options (either in “nor­mal” or in “blu­ray” mode), but I did not took the time to test this.

The only think I can tell defin­i­tive­ly ATM is, that despite to all the infor­ma­tion I found in the net about this, noth­ing can be told about the hard require­ments except, that the video encod­ing can not exceed lev­el 4.1 (or 4), because so far no hard­ware decoder chip seems to sup­port a high­er lev­el. It seems I can use more or less all options avail­able for the H.264 encod­ing, and the res­o­lu­tion and FPS does not seem to mat­ter that much. I looks more that you have to find a ver­sion of an H.264 encoder which works for you, and after that you can more or less do what you want.