HOWTO: “Blind” re­mote in­stall of FreeB­SD via tiny disk im­age (ZFS edition)

In a past post I described how to install a FreeB­SD remote­ly via a tiny UFS based disk image over a lin­ux sys­tem. In this post I describe how to do it with a ZFS based disk image.

Invari­ants

Giv­en a unix based remote sys­tem (in this case a lin­ux sys­tem) from which you know what kind of hard­ware it runs on (e.g. PCI IDs) and what the cor­re­spond­ing FreeB­SD dri­vers are.

HOWTO

In the title of this post I wrote “via a tiny disk im­age”. This is true for a suit­able defin­i­tion of tiny.

What we have in the root­server are two ~900 GB hard­disks. They shall be used in a soft­ware mir­ror. The ma­chine has 8 GB of RAM. I do not ex­pect much ker­nel pan­ics (= crash dumps) there, so we do not real­ly need >8 GB of swap (for­get the rule of hav­ing twice as much swap than RAM, with the cur­rent amount of RAM in a ma­chine you are in “trou­ble” when you need even the same amount of swap than RAM). I de­cided to go with 2 GB of swap.

Pushing/pulling a 900 GB im­age over the net­work to in­stall a sys­tem is not real­ly some­thing I want to do. I am OK to trans­fer 5 GB (that is 0.5% of the en­tire disk) to get this job done, and this is feasible.

First let us define some vari­ables in the shell, this way you just need to change the val­ues in one place and the rest is copy&paste (I use the SVN revi­sion of the source which I use to install the sys­tem as the name of the sysutils/beadm com­pat­i­ble boot-dataset in the rootfs, as such I also have the revi­sion num­ber avail­able in a variable):

ROOTFS_SIZE=5G
ROOTFS_NAME=root
FILENAME=rootfs
POOLNAME=mpool
VERSION=r$(cd /usr/src; svnliteversion)
SWAPSIZE=2G

Then change your cur­rent dir­ect­ory to a place where you have enough space for the im­age. There we will cre­ate a con­tainer for the im­age, and make it ready for partitioning:

truncate -s ${ROOTFS_SIZE} ${FILENAME}
mdconfig -a -t vnode -f ${FILENAME}
# if you want to fully allocate
# dd if=/dev/zero of=/dev/md0 bs=1m

Cre­ate the par­ti­tion table and the rootfs (in a sysutils/beadm com­pat­i­ble way – as I install FreeBSD-current there – and mount it tem­po­rary to /temppool):

gpart create -s GPT /dev/md0
gpart add -s 512K -t freebsd-boot -l bootcode0 /dev/md0
gpart add -a 4k -t freebsd-swap -s ${SWAPSIZE} -l swap0 /dev/md0
gpart add -a 1m -t freebsd-zfs -l ${POOLNAME}0 /dev/md0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 /dev/md0
# if not already the case and you want to have 4k physical sector size of the pool
# syscl vfs.zfs.min_auto_ashift=12
zpool create -o cachefile=/boot/zfs/zpool.cache_temp -o altroot=/temppool -O compress=lz4 -O atime=off -O utf8only=on ${POOLNAME} /dev/gpt/${POOLNAME}0
zfs create -o mountpoint=none ${POOLNAME}/ROOT
zfs create -o mountpoint=/ ${POOLNAME}/ROOT/${VERSION}
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off ${POOLNAME}/tmp
zfs create -o mountpoint=/usr -o canmount=off ${POOLNAME}/usr
zfs create -o mountpoint=/home ${POOLNAME}/home
zfs create -o setuid=off ${POOLNAME}/usr/ports
zfs create ${POOLNAME}/usr/src
zfs create -o mountpoint=/var -o canmount=off ${POOLNAME}/var
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/audit
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/crash
zfs create -o exec=off -o setuid=off ${POOLNAME}/var/log
zfs create -o atime=on ${POOLNAME}/var/mail
zfs create -o setuid=off ${POOLNAME}/var/tmp
zfs create ${POOLNAME}/var/ports
zfs create -o exec=off -o setuid=off -o mountpoint=/shared ${POOLNAME}/shared
zfs create -o exec=off -o setuid=off ${POOLNAME}/shared/distfiles
zfs create -o exec=off -o setuid=off ${POOLNAME}/shared/packages
zfs create -o exec=off -o setuid=off -o compression=lz4 ${POOLNAME}/shared/ccache
zfs create ${POOLNAME}/usr/obj
zpool set bootfs=${POOLNAME}/ROOT/${VERSION} ${POOLNAME}

In­stall FreeB­SD (from source):

cd /usr/src
#make buildworld >&! buildworld.log
#make buildkernel -j 8 KERNCONF=GENERIC >&! buildkernel_generic.log
make installworld DESTDIR=/temppool/ >& installworld.log
make distribution DESTDIR=/temppool/ >& distrib.log
make installkernel KERNCONF=GENERIC DESTDIR=/temppool/ >& installkernel.log

Copy the tem­po­rary zpool cache cre­at­ed above in the pool-creation part to the image (I have the impres­sion it is not real­ly need­ed and will work with­out, but I have not tried this):

cp /boot/zfs/zpool.cache_temp /temppool/boot/
cp /boot/zfs/zpool.cache_temp /temppool/boot/zpool.cache

Add the zfs mod­ule to loader.conf:

zfs_load="yes"
opensolaris_load="yes"

Now you need to cre­ate /temppool/etc/rc.conf (set the de­faultrouter, the IP ad­dress via ifconfig_IF (and do not for­get to use the right IF for it), the host­name, set sshd_enable to yes, zfs_enable=“YES”)  /temppool/boot/loader.conf (zfs_load=“yes”, opensolaris_load=“yes”, vfs.root.mountfrom=“zfs:${POOLNAME}/ROOT/r${VERSION}”)
/temppool/etc/hosts, /temppool/etc/resolv.conf and maybe /temppool/etc/sysctl.conf and /temppool/etc/periodic.conf.

Do not allow password-less root logins in single-user mode on the phys­i­cal con­sole, cre­ate a resolv.conf and an user:

cd /temppool/etc
sed -ie 's:console.*off.:&in:' ttys
cat >resolv.conf <<EOT
search YOURDOMAIN
nameserver 8.8.8.8
EOT
pw -V /temppool/etc groupadd YOURGROUP -g 1001
pw -V /temppool/etc useradd YOURUSER -u 1001 -d /home/YOURUSER -g YOURUSER -G wheel -s /bin/tcsh
pw -V /temppool/etc usermod YOURUSER -h 0
pw -V /temppool/etc usermod root -h 0
zfs create mpool/home/YOURUSER
chown YOURUSER:YOURGROUP /temppool/home/YOURUSER

Now you can make some more mod­i­fi­ca­tions to the sys­tem if want­ed, and then export the pool and detach the image:

zpool export ${POOLNAME}

mdconfig -d -u 0

Depend­ing on the upload speed you can achieve, it is ben­e­fi­cial to com­press the image now, e.g. with bzip2. Then trans­fer the image to the disk of the remote sys­tem. In my case I did this via:

ssh –C –o CompressionLevel=9 root@remote_host dd of=/dev/hda bs=1m < /path/to/${FILENAME}

Then reboot/power-cycle the remote system.

Post-install tasks

Now we have a new FreeB­SD sys­tem which uses only a frac­tion of the the hard­disk and is not resilient against harddisk-failures.

FreeB­SD will detect that the disk is big­ger than the image we used when cre­at­ing the GPT label and warn about it (cor­rupt GPT table). To fix this and to resize the par­ti­tion for the zpool to use the entire disk we first mir­ror the zpool to the sec­ond disk and resize the par­ti­tion of the first disk, and when the zpool is in-sync and then we resize the boot disk (atten­tion, you need to change the “-s” part in the fol­low­ing to match your disk size).

First back­up the label of the first disk, this makes it more easy to cre­ate the label of the sec­ond disk:

/sbin/gpart backup ada0 > ada0.gpart

Edit ada0.gpart (give dif­fer­ent names for the labels, main­ly change the num­ber 0 on the label-name to 1) and then use it to cre­ate the par­ti­tion of the sec­ond disk:

gpart restore -Fl ada1 < ada0.gpart
gpart resize -i 3 -a 4k -s 929g ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
zpool set autoexpand=on mpool

Fix the warn­ing about the GPT label and resize the partition:

gpart recover ada0
gpart resize -i 3 -a 4k -s 929g ada0

After­wards it should look sim­i­lar to this:

gpart show -l
=>        40  1953525088  ada0  GPT  (932G)
          40        1024     1  bootcode0  (512K)
        1064     4194304     2  swap0  (2.0G)
     4195368         984        - free -  (492K)
     4196352  1948254208     3  mpool0  (929G)
  1952450560     1074568        - free -  (525M)

=>        40  1953525088  ada1  GPT  (932G)
          40        1024     1  bootcode1  (512K)
        1064     4194304     2  swap1  (2.0G)
     4195368         984        - free -  (492K)
     4196352  1948254208     3  mpool1  (929G)
  1952450560     1074568        - free -  (525M)

Add the sec­ond disk to the zpool:

zpool attach mpool gpt/mpool0 gpt/mpool1

When the mir­ror is in sync (zpool sta­tus mpool), we can extend the size of the pool itself:

zpool offline mpool /dev/gpt/mpool0
zpool online mpool /dev/gpt/mpool0

As a last step we can add now an encrypt­ed swap (depend­ing on the impor­tance of the sys­tem maybe a gmirror-ed one – not explained here), and spec­i­fy where to dump (text-dumps) on.

/boot/loader.conf:

dumpdev="/dev/ada0p2"

/etc/rc.conf:

dumpdev="/dev/gpt/swap0"
crashinfo_enable="YES"
ddb_enable="yes"
encswap_enable="YES"
geli_swap_flags="-a hmac/sha256 -l 256 -s 4096 -d"

/etc/fstab:

# Device        Mountpoint      FStype  Options                 Dump    Pass#
/dev/ada1p2.eli none    swap    sw      0       0

Now the sys­tem is ready for some applications.

Tran­si­tion to nginx: part 4 – CGI scripts

I still have some CGI scripts on this web­site. They still work, and they are good enough for my needs. When I switched this web­site to nginx (the word­press set­up was a lit­tle bit more com­plex than what I wrote in part 1, part 2 and part 3… the con­fig will be one of my next blog posts) I was a lit­tle bit puz­zled how to do that with nginx. It took me some min­utes to get an idea how to do it and to find the right FreeB­SD port for this.

  • Install www/fcgiwrap
  • Add the fol­low­ing to rc.conf:

fcgiwrap_enable="YES"
fcgiwrap_user="www"

  • Run “ser­vice fcgi­wrap start”
  • Add the fol­low­ing to your nginx config:
location ^~ /cgi-bin/ {
    gzip off; #gzip makes scripts feel slower since they have to complete before getting gzipped
    fastcgi_pass  unix:/var/run/fcgiwrap/fcgiwrap.sock;
    fastcgi_index index.cgi;
    fastcgi_param SCRIPT_FILENAME /path/to/location$fastcgi_script_name;
    fastcgi_param GATEWAY_INTERFACE  CGI/1.1;
}

Tran­si­tion to nginx: part 3 — short and easy con­fig snippets

After some medium-difficoulty tran­si­tions in part 1 and part 2, here some easy ones:

php­MyAd­min: take the basics from one of the two oth­er blog posts (see above) with­out loca­tion direc­tives. For “loca­tion /” set the doc­u­ment root and copy the “loca­tion ~ \.php” from the con­fig of one of the parts above. Done.

TT-RSS: take the con­fig like for php­MyAd­min and add (assum­ing it is in the root of the serv­er, else you have to add the path in the front of the location)

location ^~ /(utils|templates|schema|cache|lock|locale|classes) {
     deny all;
}

Allow client-side caching for sta­t­ic content:

location ~* \.(?:jpe?g|gif|png|ico|cur|gz|bz2|xz|tbz|tgz|txz|svg|svgz|mp4|ogg|ogv|webm|htc|css|js|
pdf|zip|rar|tar|txt|conf)$ {
    try_files $uri =404;

    expires 1w;     # If you are not a big site,

                    # and don't change static content often,

                    # 1 week is not bad.
    access_log off; # If you don't need the logs
    add_header Cache-Control "public";
}

Secu­ri­ty: Despite the fact that the docs I’ve read tell that no-SSLv3 is the default, the first set­ting makes a dif­fer­ence (test­ed via SSLlabs’ SSLtest).

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # No SSLv 2/3
ssl_dhparam /path/to/dhparams.pem;   # generate via "openssl dhparam -out /path/to/dhparams.pem 2048"

 

Tran­si­tion to nginx: part 2 – con­vert­ing a gallery v2 installation

In my first tran­si­tion to nginx I wrote that I was hap­py about the speed increase I got for my Horde web­mail set­up. After­wards I con­vert­ed a Gallery v2 instal­la­tion (yes, old, not under active devel­op­ment any­more, but inter­nal and still work­ing). There I have not seen any obvi­ous speed difference.

I did not con­vert all .htac­cess rewrite rules, the one for the “easy and beau­ti­ful” URL names was too com­plex for the con­vert­er for rewrite I found. As it is just for inter­nal use, I just switched back to the not so nice “tech­ni­cal” URL names.

The impor­tant part of the apache 2.2 installation:

ExpiresActive On
ExpiresDefault "now plus 1 hour"
ExpiresByType image/* "now plus 1 month"
ExpiresByType text/javascript "now plus 1 month"
ExpiresByType application/x-javascript "now plus 1 month"
ExpiresByType text/css "now plus 1 month"

<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip|pdf)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

The nginx config:

worker_processes  1;

error_log  <filename>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log  <filename>;

    sendfile    on;

        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout            300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 40M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

    server {
        listen       80;
        server_name  <hostname>;

        add_header   x-frame-options            "sameorigin";
        add_header   x-xss-protection           "1; mode=block";
        add_header   x-content-type-options     "nosniff";

        charset utf-8;

        #access_log  logs/host.access.log  main;
        if ($bad_client) { return 403; }

        location / {
            root   /usr/local/www/gallery2;
            index  index.php;
                location ~ \.php {
                        # Zero-day exploit defense.
                        # http://forum.nginx.org/read.php?2,88845,page=3
                        # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                        # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                        try_files $uri =404;

                        fastcgi_split_path_info ^(.+\.php)(/.+)$;
                        fastcgi_keep_conn       on;
                        fastcgi_index      index.php;
                        include          fastcgi_params;
                        fastcgi_param      SCRIPT_FILENAME $document_root$fastcgi_script_name;
                        fastcgi_pass        unix:/var/run/php.fcgi;
                }
        }

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\.ht {
            deny  all;
        }
        location ~ \.(inc|class)$ {
                deny all;
        }
        location ^~ /lib/tools/po/ {
                deny all;
        }
    }
}

Tran­si­tion to nginx: part 1, con­vert­ing Horde webmail

I am a long­time apache user. It may be that I touched apache 1.2 for the first time. Recent­ly I decid­ed to check out nginx. So I decid­ed to do it on my own web­mail sys­tem. The end result is, I replaced apache+mod_php with nginx+php-fpm there, and so far it does not look like I want to go back (it feels faster on direct com­pare, same serv­er, either apache or nginx start­ed to com­pare the speed, pure sub­jec­tive “mea­sure­ment”, no numbers).

The long sto­ry now.

The web­mail sys­tem uses horde. There I had apache 2.4 (pre­fork mpm) and php 5.6 via mod_php. With nginx I used php-fpm. I used the same php flags and val­ues in php_fpm like I used with mod_php. I con­fig­ured less php-fpm max-processes than I had allowed apache+mod_php to use. As nginx is not spawn­ing process­es for each con­nec­tion, I have less process­es and also less mem­o­ry allo­cat­ed as a result. Con­vert­ing the rewrite rules and mod_rewrite based black­list­ing took a while (and I have not con­vert­ed all black­lists I had before). So yes, this is not real­ly com­par­ing apples with apples (I could have tried a dif­fer­ent mpm for apache, and I could have used an fcgi based php approach instead of mod_php, I could have moved the rewrite rules out from .htac­cess files to the main config).

The result was on first login that the pages appeared notice­able faster. I direct­ly switched back to apache to con­firm. This was over a WLAN con­nec­tion. A lit­tle bit lat­er I had the same impres­sion when I test­ed this over a slow DSL link (2 MBit/s).

Here the impor­tant parts of the apache config:

ExpiresActive On
ExpiresDefault "now plus 3 hours"
ExpiresByType image/* "now plus 2 months"
ExpiresByType text/javascript "now plus 2 months"
ExpiresByType application/x-javascript "now plus 2 months"
ExpiresByType text/css "now plus 2 months"

SetEnvIfNoCase Request_URI "\.gif$" cache_me=1
SetEnvIfNoCase Request_URI "\.png$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpg$" cache_me=1
SetEnvIfNoCase Request_URI "\.jpeg$" cache_me=1
SetEnvIfNoCase Request_URI "\.ico$" cache_me=1
SetEnvIfNoCase Request_URI "\.css$" cache_me=1
# Allow caching on media files
<IfModule mod_headers.c>
Header merge Cache-Control "public" env=cache_me
  <IfModule ssl_module.c>
    Header add Strict-Transport-Security "max-age=15768000"
  </IfModule>
</IfModule>
<Location />
# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|gz|bz2|zip)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>
SetEnvIfNoCase Request_URI "\.js$" cache_me=1
Alias /Microsoft-Server-ActiveSync /usr/local/www/horde/rpc.php
RedirectPermanent /.well-known/carddav /horde/rpc.php
AcceptPathInfo on

And here most parts of my nginx.conf suit­able for Horde, includ­ing the rewrite rules from .htac­cess files:

worker_processes  1;

error_log  <file>;

events {
        worker_connections      1024;
        use                     kqueue;
}


http {
        include    mime.types;
        default_type  application/octet-stream;

        access_log  <file>;

        sendfile        on;
        keepalive_timeout       15;
        client_body_timeout     300;
        client_header_timeout   12;
        send_timeout        300;
        client_body_in_file_only clean;
        client_body_buffer_size 128k;
        client_max_body_size 10M;

        gzip on;
        gzip_min_length 1000;
        gzip_types       text/plain text/xml text/css application/xml application/xhtml+xml application/rss+xml application/javascript application/x-javascript;
        gzip_disable     "msie6";

        include blacklist.conf;

        server {
                listen     443 ssl spdy;
                server_name  <hostname>;

                ssl_certificate         <file>;
                ssl_certificate_key     <file>;

                ssl_session_cache       shared:SSL:10m;
                ssl_session_timeout     15m;
                ssl_ciphers     <ciper_list>;
                ssl_prefer_server_ciphers  on;

                # optional: see https://www.owasp.org/index.php/List_of_useful_HTTP_headers
                add_header                        strict-transport-security "max-age=31536000";
                add_header                        x-frame-options          "sameorigin";
                add_header                        x-xss-protection        "1; mode=block";
                add_header                        x-content-type-options        "nosniff";

                root                            /usr/local/www/horde;
                index                              index.php;

                charset utf8;

                access_log  <logfile>;
                if ($bad_client) { return 403; }

                location / {
                        location /Microsoft-Server-ActiveSync {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /autodiscover/autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location /Autodiscover/Autodiscover.xml {
                                alias                   /usr/local/www/horde/rpc.php;
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME /usr/local/www/horde/rpc.php;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ^~ /(static|themes)/ {
                                expires                 1w;
                                add_header              Cache-Control public;
                        }
                        location ^~ /services/ajax.php {
                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        location ~ \.php {
                                # Zero-day exploit defense.
                                # http://forum.nginx.org/read.php?2,88845,page=3
                                # Won't work properly (404 error) if the file is not stored on this server, which is entirely possible with php-fpm/php-fcgi.
                                # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on another machine.  And then cross your fingers that you won't get hacked.
                                try_files $uri =404;

                                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                                fastcgi_keep_conn       on;
                                fastcgi_index           index.php;
                                include                 fastcgi_params;
                                fastcgi_param           SCRIPT_FILENAME $document_root$fastcgi_script_name;
                                fastcgi_pass            unix:/var/run/php.fcgi;
                        }

                        try_files                          $uri $uri/ /rampage.php?$args;
                }
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
                root   /usr/local/www/nginx-dist;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
                location ~ /\.ht {
                        deny all;
                }
                location ~ /(config|lib|locale|po|scripts|templates)/ {
                        deny all;
                }
                location ^~ /rpc/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /rpc/index.php/$1 break;
                        }
                }
                location ^~ /kronolith/feed/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /kronolith/feed/index.php?c=$1 break;
                        }
                }
                location ^~ /content/ {
                        if (!-e $request_filename){
                                rewrite ^(.*)$ /content/index.php break;
                        }
                }
                location ^~ /whups/(queue|query)/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/queue/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/queue/rss.php?id=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/?$ /whups/queue/index.php?slug=$1;
                        rewrite ^/([a-zA-Z0-9_]+)/rss/?$ /whups/queue/rss.php?slug=$1;
                }
                location ^~ /whups/ticket/ {
                        if (!-e $request_filename){
                                rewrite ^/([0-9]+)/?$ /whups/ticket/index.php?id=$1;
                        }
                        rewrite ^/([0-9]+)/rss/?$ /whups/ticket/rss.php?id=$1;
                        rewrite ^/([0-9]+)/([a-z]+)(\.php)?$ /whups/ticket/$2.php?id=$1 break;
                }
                location ^~ /.well-known/carddav {
                        return 301 https://webmail.Leidinger.net/rpc.php;
                }
                location ^~ /admin/ {
                        allow <local>;
                        deny all;
                }
                location ~ /test.php$ {
                        allow <local>;
                        deny all;
                }

                # Media: images, icons, video, audio, HTC, archives
                location ~* \.(?:jpe?g|gif|png|ico|cur|gz|bz2|tbz|tgz|svg|svgz|mp4|ogg|ogv|webm|htc|css|js|pdf|zip|rar|tar|txt|pl|conf)$ {
                        try_files $uri =404;

                        expires 1w;
                        access_log off;
                        add_header Cache-Control "public";
                }
        }
}