Tran­si­tion to nginx: part 2 – con­vert­ing a gallery v2 installation

In my first tran­si­tion to nginx I wrote that I was hap­py about the speed increase I got for my Horde web­mail set­up. After­wards I con­vert­ed a Gallery v2 instal­la­tion (yes, old, not under active devel­op­ment any­more, but inter­nal and still work­ing). There I have not seen any obvi­ous speed difference.

I did not con­vert all .htac­cess rewrite rules, the one for the “easy and beau­ti­ful” URL names was too com­plex for the con­vert­er for rewrite I found. As it is just for inter­nal use, I just switched back to the not so nice “tech­ni­cal” URL names.

The impor­tant part of the apache 2.2 installation:

ExpiresActive On
ExpiresDefault "now plus 1 hour"
ExpiresByType image/* "now plus 1 month"
ExpiresByType text/javascript "now plus 1 month"
ExpiresByType application/x-javascript "now plus 1 month"
ExpiresByType text/css "now plus 1 month"

<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip|pdf)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

The nginx config:

worker_processes  1;

error_log  <filename>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  <filename>;

    sendfile    on;

        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout            300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 40M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

    server {
        listen       80;
        server_name  <hostname>;

        add_header   x-frame-options            "sameorigin";
        add_header   x-xss-protection           "1; mode=block";
        add_header   x-content-type-options     "nosniff";

        charset utf-8;

        #access_log  logs/host.access.log  main;
        if ($bad_client) { return 403; }

        location / {
            root   /usr/local/www/gallery2;
            index  index.php;
                location ~ \.php {
                        # Zero-day exploit defense.
                        # http://forum.nginx.org/read.php?2,88845,page=3
                        # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                        # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                        try_files $uri =404;

                        fastcgi_split_path_info ^(.+\.php)(/.+)$;
                        fastcgi_keep_conn       on;
                        fastcgi_index      index.php;
                        include          fastcgi_params;
                        fastcgi_param      SCRIPT_FILENAME $document_root$fastcgi_script_name;
                        fastcgi_pass        unix:/var/run/php.fcgi;
                }
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
            deny  all;
        }
        location ~ \.(inc|class)$ {
                deny all;
        }
        location ^~ /lib/tools/po/ {
                deny all;
        }
    }
}

Tran­si­tion to nginx: part 1, con­vert­ing Horde webmail

I am a long­time apache user. It may be that I touched apache 1.2 for the first time. Recent­ly I decid­ed to check out nginx. So I decid­ed to do it on my own web­mail sys­tem. The end result is, I replaced apache+mod_php with nginx+php-fpm there, and so far it does not look like I want to go back (it feels faster on direct com­pare, same serv­er, either apache or nginx start­ed to com­pare the speed, pure sub­jec­tive “mea­sure­ment”, no numbers).

The long sto­ry now.

The web­mail sys­tem uses horde. There I had apache 2.4 (pre­fork mpm) and php 5.6 via mod_php. With nginx I used php-fpm. I used the same php flags and val­ues in php_fpm like I used with mod_php. I con­fig­ured less php-fpm max-processes than I had allowed apache+mod_php to use. As nginx is not spawn­ing process­es for each con­nec­tion, I have less process­es and also less mem­o­ry allo­cat­ed as a result. Con­vert­ing the rewrite rules and mod_rewrite based black­list­ing took a while (and I have not con­vert­ed all black­lists I had before). So yes, this is not real­ly com­par­ing apples with apples (I could have tried a dif­fer­ent mpm for apache, and I could have used an fcgi based php approach instead of mod_php, I could have moved the rewrite rules out from .htac­cess files to the main config).

The result was on first login that the pages appeared notice­able faster. I direct­ly switched back to apache to con­firm. This was over a WLAN con­nec­tion. A lit­tle bit lat­er I had the same impres­sion when I test­ed this over a slow DSL link (2 MBit/s).

Here the impor­tant parts of the apache config:

ExpiresActive On
ExpiresDefault "now plus 3 hours"
ExpiresByType image/* "now plus 2 months"
ExpiresByType text/javascript "now plus 2 months"
ExpiresByType application/x-javascript "now plus 2 months"
ExpiresByType text/css "now plus 2 months"

SetEnvIfNoCase Request_URI "\.gif$" cache_me=1
SetEnvIfNoCase Request_URI "\.png$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpg$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpeg$" cache_me=1
SetEnvIfNoCase Request_URI "\.ico$" cache_me=1
SetEnvIfNoCase Request_URI "\.css$" cache_me=1
# Allow caching on media files
<IfModule mod_headers.c>
Header merge Cache-Control "public" env=cache_me
  <IfModule ssl_module.c>
    Header add Strict-Transport-Security "max-age=15768000"
  </IfModule>
</IfModule>
<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>
SetEnvIfNoCase Request_URI "\.js$" cache_me=1
Alias /Microsoft-Server-ActiveSync /usr/local/www/horde/rpc.php
RedirectPermanent /.well-known/carddav /horde/rpc.php
AcceptPathInfo on

And here most parts of my nginx.conf suit­able for Horde, includ­ing the rewrite rules from .htac­cess files:

worker_processes  1;

error_log  <file>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
        include    mime.types;
        default_type  application/octet-stream;

        access_log  <file>;

        sendfile        on;
        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout        300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 10M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

        server {
                listen     443 ssl spdy;
                server_name  <hostname>;

                ssl_certificate         <file>;
                ssl_certificate_key     <file>;

                ssl_session_cache       shared:SSL:10m;
                ssl_session_timeout     15m;
                ssl_ciphers     <ciper_list>;
                ssl_prefer_server_ciphers  on;

                # optional: see https://www.owasp.org/index.php/List_of_useful_HTTP_headers
                add_header                        strict-transport-security "max-age=31536000";
                add_header                        x-frame-options          "sameorigin";
                add_header                        x-xss-protection        "1; mode=block";
                add_header                        x-content-type-options        "nosniff";

                root                            /usr/local/www/horde;
                index                              index.php;

                charset utf8;

                access_log  <logfile>;
                if ($bad_client) { return 403; }

                location / {
                        location /Microsoft-Server-ActiveSync {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /autodiscover/autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /Autodiscover/Autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ^~ /(static|themes)/ {
                                expires                 1w;
                                add_header              Cache-Control public;
                        }
                        location ^~ /services/ajax.php {
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ~ \.php {
                                # Zero-day exploit defense.
                                # http://forum.nginx.org/read.php?2,88845,page=3
                                # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                                # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                                try_files $uri =404;

                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                fastcgi_index           index.php;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        try_files                          $uri $uri/ /rampage.php?$args;
                }
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
                location ~ /\.ht {
                        deny all;
                }
                location ~ /(config|lib|locale|po|scripts|templates)/ {
                        deny all;
                }
                location ^~ /rpc/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /rpc/index.php/$1 break;
                        }
                }
                location ^~ /kronolith/feed/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /kronolith/feed/index.php?c=$1 break;
                        }
                }
                location ^~ /content/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /content/index.php break;
                        }
                }
                location ^~ /whups/(queue|query)/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/queue/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/queue/rss.php?id=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/?$ /whups/queue/index.php?slug=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/rss/?$ /whups/queue/rss.php?slug=$1;
                }
                location ^~ /whups/ticket/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/ticket/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/ticket/rss.php?id=$1;
                        rewrite ^/([0-9]+)/([a-z]+)(\.php)?$ /whups/ticket/$2.php?id=$1 break;
                }
                location ^~ /.well-known/carddav {
                        return 301 https://webmail.Leidinger.net/rpc.php;
                }
                location ^~ /admin/ {
                        allow <local>;
                        deny all;
                }
                location ~ /test.php$ {
                        allow <local>;
                        deny all;
                }

                # Media: images, icons, video, audio, HTC, archives
                location ~* \.(?:jpe?g|gif|png|ico|cur|gz|bz2|tbz|tgz|svg|svgz|mp4|ogg|ogv|webm|htc|css|js|pdf|zip|rar|tar|txt|pl|conf)$ {
                        try_files $uri =404;

                        expires 1w;
                        access_log off;
                        add_header Cache-Control "public";
                }
        }
}

Gain­ing space on Android after ART->Dalvik switch (root access required)

I (still) use a Nexus S phone. I am using Cyanogen­mod on it. After an arti­cle in a com­put­er mag­a­zine I decid­ed to give the ART-runtime a try instead of the default Dalvic-runtime. Unfor­tu­nate­ly I do not have enough free space free (and all what I can is moved to the USB stor­age already) to real­ly use the ART-runtime.

After switch­ing back to the Dalvic-runtime, I had only 23 of the pre­vi­ous­ly avail­able space free. After a lit­tle bit of look­ing around I found /data/dalvik-cache. I delet­ed with a file man­ag­er the con­tent of the direc­to­ry (you will get some “app crashed/died” mes­sages) and reboot­ed the phone (this is not the same as for­mat­ting the cache par­ti­tion in the recov­ery system).

Dur­ing boot it pop­u­lat­ed the direc­to­ry again and now I have more than 43 of free space on the inter­nal storage.

Lin­ux­u­la­tor explained: How to cre­ate Lin­ux bina­ries on FreeBSD

There may by cas­es where you want to gen­er­ate a Lin­ux bina­ry on a FreeB­SD machine. This is not a prob­lem with the lin­ux­u­la­tor, but not with the default linux_base port.

As you may know, the linux_base port is designed to deliv­er an inte­grat­ed expe­ri­ence with FreeB­SD native pro­grams. As such some parts of the native FreeB­SD infra­struc­ture is used. If you would try to use a Linux-compiler to gen­er­ate Linux-binaries, you would run into the prob­lem that by default the FreeB­SD includes are used.

Pre­req­ui­sites

To have a ful­ly fea­tured and non-integrated Lin­ux envi­ron­ment on your FreeB­SD sys­tem either mount an exist­ing (and com­pat­i­ble) Lin­ux instal­la­tion some­where into your FreeB­SD sys­tem, or install a linux_dist port. This can be done addi­tion­al­ly to an already installed linux_base port.

Prepa­ra­tion

When you have a com­plete Lin­ux envi­ron­ment avail­able, you need to mount the FreeB­SD devfs to /path/to/complete_linux/dev, lin­procfs to /path/to/complete_linux/proc and lin­sys­fs to /path/to/complete_linux/sys to have a com­plete setup.

Use it

Now you just need to chroot into this  /path/to/complete_linux and you configure/make/install or what­ev­er you need to do to gen­er­ate your desired Lin­ux binary.

Cal­cu­lat­ing the tar­get size of H264 videos

From time to time I con­vert videos to H264. When I do this I want to get the best qual­i­ty out of a give file­size. This means I cre­ate VBR videos. The ques­tion here is, how big the tar­get video file shall be.

After search­ing around a lit­tle bit in the net I found a for­mu­la which is sup­posed to give a hint about the tar­get file­size. Nat­u­ral­ly this depends on the encoder (or even the encoder-version), the encoder set­tings and even the video.

The for­mu­la is at least a good hint for my use, so I wrote a script which cal­cu­lates sev­er­al file­size val­ues for a giv­en video (based upon the out­put of medi­ain­fo, which the scripts expects in a file with the file­name as an argu­ment to the script). It cal­cu­lates a CBR and a VBR val­ue for a giv­en video based upon the width, height and dura­tion. It should work on all sys­tem with a POSIX com­pat­i­ble shell.

Exam­ple out­put for a video from my HD-ready cam, orig­i­nal file­size 1.8 GB:

Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 2
Per sec­ond: 6451200.000 bps / 6300 Kibps
Total CBR: 1148313600 bytes / 1121400 KiB / 1095 MiB
Total VBR: 861235200 bytes / 841050 KiB / 821 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 3
Per sec­ond: 9676800.000 bps / 9450 Kibps
Total CBR: 1722470400 bytes / 1682100 KiB / 1642 MiB
Total VBR: 1291852800 bytes / 1261575 KiB / 1232 MiB
Width: 1280, Height: 720, FPS: 50.000, Time: 1424, Motion: 4
Per sec­ond: 12902400.000 bps / 12600 Kibps
Total CBR: 2296627200 bytes / 2242800 KiB / 2190 MiB
Total VBR: 1722470400 bytes / 1682100 KiB / 1642 MiB

There are 3 sec­tions, the dif­fer­ence is the “motion” val­ue. It is a kind of mul­ti­pli­ca­tor depend­ing on the amount of motion in the video. For the videos I made myself (fam­i­ly videos, and even some videos of vol­ley ball games), the first sec­tion seems to be just fine. So I reduced the orig­i­nal MP4 file to about 50% (not vis­i­ble here is the audio size, nor­mal­ly I copy the orig­i­nal audio unmodified).

For the curi­ous ones, the for­mu­la is

width_in_pixels * height_in_pixels * fps * motion_value * 0.07

for the bps val­ue. The CBR val­ue is

bps * playtime_in_seconds / 8

and the VBR val­ue is

34 * CBR_value.