Tran­si­tion to nginx: part 2 – con­vert­ing a gallery v2 instal­la­tion

In my first tran­si­tion to nginx I wrote that I was hap­py about the speed increase I got for my Horde web­mail set­up. After­wards I con­vert­ed a Gallery v2 instal­la­tion (yes, old, not under active devel­op­ment any­more, but inter­nal and still work­ing). There I have not seen any obvi­ous speed dif­fer­ence.

I did not con­vert all .htac­cess rewrite rules, the one for the “easy and beau­ti­ful” URL names was too com­plex for the con­vert­er for rewrite I found. As it is just for inter­nal use, I just switched back to the not so nice “tech­ni­cal” URL names.

The impor­tant part of the apache 2.2 instal­la­tion:

ExpiresActive On
ExpiresDefault "now plus 1 hour"
ExpiresByType image/* "now plus 1 month"
ExpiresByType text/javascript "now plus 1 month"
ExpiresByType application/x-javascript "now plus 1 month"
ExpiresByType text/css "now plus 1 month"

<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip|pdf)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

The nginx con­fig:

worker_processes  1;

error_log  <filename>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  <filename>;

    sendfile    on;

        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout            300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 40M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

    server {
        listen       80;
        server_name  <hostname>;

        add_header   x-frame-options            "sameorigin";
        add_header   x-xss-protection           "1; mode=block";
        add_header   x-content-type-options     "nosniff";

        charset utf-8;

        #access_log  logs/host.access.log  main;
        if ($bad_client) { return 403; }

        location / {
            root   /usr/local/www/gallery2;
            index  index.php;
                location ~ \.php {
                        # Zero-day exploit defense.
                        # http://forum.nginx.org/read.php?2,88845,page=3
                        # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                        # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                        try_files $uri =404;

                        fastcgi_split_path_info ^(.+\.php)(/.+)$;
                        fastcgi_keep_conn       on;
                        fastcgi_index      index.php;
                        include          fastcgi_params;
                        fastcgi_param      SCRIPT_FILENAME $document_root$fastcgi_script_name;
                        fastcgi_pass        unix:/var/run/php.fcgi;
                }
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
            deny  all;
        }
        location ~ \.(inc|class)$ {
                deny all;
        }
        location ^~ /lib/tools/po/ {
                deny all;
        }
    }
}
Send to Kin­dle

Tran­si­tion to nginx: part 1, con­vert­ing Horde web­mail

I am a long­time apache user. It may be that I touched apache 1.2 for the first time. Recent­ly I decid­ed to check out nginx. So I decid­ed to do it on my own web­mail sys­tem. The end result is, I replaced apache+mod_php with nginx+php-fpm there, and so far it does not look like I want to go back (it feels faster on direct com­pare, same serv­er, either apache or nginx start­ed to com­pare the speed, pure sub­jec­tive “mea­sure­ment”, no num­bers).

The long sto­ry now.

The web­mail sys­tem uses horde. There I had apache 2.4 (pre­fork mpm) and php 5.6 via mod_php. With nginx I used php-fpm. I used the same php flags and val­ues in php_fpm like I used with mod_php. I con­fig­ured less php-fpm max-processes than I had allowed apache+mod_php to use. As nginx is not spawn­ing process­es for each con­nec­tion, I have less process­es and also less mem­o­ry allo­cat­ed as a result. Con­vert­ing the rewrite rules and mod_rewrite based black­list­ing took a while (and I have not con­vert­ed all black­lists I had before). So yes, this is not real­ly com­par­ing apples with apples (I could have tried a dif­fer­ent mpm for apache, and I could have used an fcgi based php approach instead of mod_php, I could have moved the rewrite rules out from .htac­cess files to the main con­fig).

The result was on first login that the pages appeared notice­able faster. I direct­ly switched back to apache to con­firm. This was over a WLAN con­nec­tion. A lit­tle bit lat­er I had the same impres­sion when I test­ed this over a slow DSL link (2 MBit/s).

Here the impor­tant parts of the apache con­fig:

ExpiresActive On
ExpiresDefault "now plus 3 hours"
ExpiresByType image/* "now plus 2 months"
ExpiresByType text/javascript "now plus 2 months"
ExpiresByType application/x-javascript "now plus 2 months"
ExpiresByType text/css "now plus 2 months"

SetEnvIfNoCase Request_URI "\.gif$" cache_me=1
SetEnvIfNoCase Request_URI "\.png$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpg$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpeg$" cache_me=1
SetEnvIfNoCase Request_URI "\.ico$" cache_me=1
SetEnvIfNoCase Request_URI "\.css$" cache_me=1
# Allow caching on media files
<IfModule mod_headers.c>
Header merge Cache-Control "public" env=cache_me
  <IfModule ssl_module.c>
    Header add Strict-Transport-Security "max-age=15768000"
  </IfModule>
</IfModule>
<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>
SetEnvIfNoCase Request_URI "\.js$" cache_me=1
Alias /Microsoft-Server-ActiveSync /usr/local/www/horde/rpc.php
RedirectPermanent /.well-known/carddav /horde/rpc.php
AcceptPathInfo on

And here most parts of my nginx.conf suit­able for Horde, includ­ing the rewrite rules from .htac­cess files:

worker_processes  1;

error_log  <file>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
        include    mime.types;
        default_type  application/octet-stream;

        access_log  <file>;

        sendfile        on;
        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout        300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 10M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

        server {
                listen     443 ssl spdy;
                server_name  <hostname>;

                ssl_certificate         <file>;
                ssl_certificate_key     <file>;

                ssl_session_cache       shared:SSL:10m;
                ssl_session_timeout     15m;
                ssl_ciphers     <ciper_list>;
                ssl_prefer_server_ciphers  on;

                # optional: see https://www.owasp.org/index.php/List_of_useful_HTTP_headers
                add_header                        strict-transport-security "max-age=31536000";
                add_header                        x-frame-options          "sameorigin";
                add_header                        x-xss-protection        "1; mode=block";
                add_header                        x-content-type-options        "nosniff";

                root                            /usr/local/www/horde;
                index                              index.php;

                charset utf8;

                access_log  <logfile>;
                if ($bad_client) { return 403; }

                location / {
                        location /Microsoft-Server-ActiveSync {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /autodiscover/autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /Autodiscover/Autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ^~ /(static|themes)/ {
                                expires                 1w;
                                add_header              Cache-Control public;
                        }
                        location ^~ /services/ajax.php {
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ~ \.php {
                                # Zero-day exploit defense.
                                # http://forum.nginx.org/read.php?2,88845,page=3
                                # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                                # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                                try_files $uri =404;

                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                fastcgi_index           index.php;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        try_files                          $uri $uri/ /rampage.php?$args;
                }
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
                location ~ /\.ht {
                        deny all;
                }
                location ~ /(config|lib|locale|po|scripts|templates)/ {
                        deny all;
                }
                location ^~ /rpc/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /rpc/index.php/$1 break;
                        }
                }
                location ^~ /kronolith/feed/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /kronolith/feed/index.php?c=$1 break;
                        }
                }
                location ^~ /content/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /content/index.php break;
                        }
                }
                location ^~ /whups/(queue|query)/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/queue/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/queue/rss.php?id=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/?$ /whups/queue/index.php?slug=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/rss/?$ /whups/queue/rss.php?slug=$1;
                }
                location ^~ /whups/ticket/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/ticket/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/ticket/rss.php?id=$1;
                        rewrite ^/([0-9]+)/([a-z]+)(\.php)?$ /whups/ticket/$2.php?id=$1 break;
                }
                location ^~ /.well-known/carddav {
                        return 301 https://webmail.Leidinger.net/rpc.php;
                }
                location ^~ /admin/ {
                        allow <local>;
                        deny all;
                }
                location ~ /test.php$ {
                        allow <local>;
                        deny all;
                }

                # Media: images, icons, video, audio, HTC, archives
                location ~* \.(?:jpe?g|gif|png|ico|cur|gz|bz2|tbz|tgz|svg|svgz|mp4|ogg|ogv|webm|htc|css|js|pdf|zip|rar|tar|txt|pl|conf)$ {
                        try_files $uri =404;

                        expires 1w;
                        access_log off;
                        add_header Cache-Control "public";
                }
        }
}
Send to Kin­dle

Essen Hack­thon 2015 – last day sta­tus

I com­mit­ted the 64bit sup­port for the lin­ux base ports (dis­abled by default, check the com­mit mes­sage), but this broke the INDEX build. Port­m­gr was faster than me to revert it. All errors are mine. I think most of the work is done, I just need to find out what the cor­rect way is to han­dle this make/fmake dif­fer­ence (mal­formed con­di­tion­al).

Send to Kin­dle

Essen Hackathon 2015 – How it looks like

For those curi­ous how it looks like:

FreeBSD developers with laptops outside in the sunClock­wise start­ing from the bot­tom left: flo, rodri­go, dru, Tux (in the back­ground, act­ing as a guard-dog^Wpenguin of the Lin­ux­ho­tel), bcr, lme, kp, bru­ef­fer.

From anoth­er angle:

FreeBSD developers with laptops outside in the sunClock­wise start­ing from the bot­tom: flo, rodri­go, dru, bcr, lme, kp, bru­ef­fer.

Not on the pic­tures is Sean Chit­ten­den.

Send to Kin­dle

Essen Hackathon Sta­tus report – 2nd day

I had a look at the open PR’s for a quick-win and found one where the depen­den­cies where incom­plete. Fixed.

Then I reviewed Alan Jude’s patch for 64bit linux_base-c6 ports (on amd64). Looks good so far. Just a few minor issues. I took the time to get famil­iar with reviews.FreeB­SD.org and the arc com­mand line tool, applied the patch to my source tree, worked a while on merge-conflicts, added some minor changes, and val­i­dat­ed the down­load of the 32bit RPM’s of the linux_base-c6 port.

In between I also discussed/reviewed some fix­es for docs with Dru, signed some PGP keys, and served as a source for a fun­ny pic­ture (at least what geeks/nerds con­sid­er a fun­ny pic­ture). I also checked how to allow multi-cast in jails. There is a PR with a patch inside, but it’s IPv6 only. I did some­thing sim­i­lar for IPv4 and com­piled a ker­nel. No com­pile time issues, but as the sys­tem where I can eas­i­ly test this is at home, I pre­fer to be in front of the box in case it pan­ics (that tells some­thing about my con­fi­dence lev­el of my patch… no idea if what I do there is actu­al­ly cor­rect… ENOCLUE about the net­work code in the ker­nel).

TODO for the last day of the Hackathon:

  • val­i­date all RPM’s (down­load / dis­tin­fo) of the ports which changed
  • val­i­date the install/deinstall of the 32bit ver­sion of the ports for regres­sion
  • val­i­date the 64bit install/deinstall for at least the lin­ux base port (more if time per­mits tomor­row)
Send to Kin­dle