Migrating to LetsEncrypt for HTTPS Security Certificates

I have had HTTPS enabled on my sites for sites for four years. Up until now I have have StartSSL for the certificates.

StartSSL was fairly unique because you only paid for validations and then could issue as many certificates as you need. I had paid for class 2 and class 3 validation and issued 4 certifies:

  • SavvyRoo.com (and www) installed on an AWS load balancer
  • Ostermiller.org, Coinmill.com and all my personal sites installed on load balanced Apache servers. This certificate included a wildcard domain for Coinmill because there are lots of international subdomains
  • My development server which was a wildcard subdomain certificate.
  • My network attached storage on my home LAN

When I got new certificates from StartSSL and started to install them, I found they did not work. It turns out they are being un-trusted by browsers. Even if they fix their problems, the won’t be able to do so by the time my certificates expire, so I need to find a new certificate vendor.

It’s time to try LetsEncrypt.org. They have some advantages:

  • Their certificates are free.
  • The process of getting and renewing certificates can (and should) be automated.
  • You don’t have to upload documents or verify your identify
  • They have software that automates the processes of getting and installing certificates

And they also have some disadvantages:

  • They don’t offer wildcard certificates.
  • They only offer class 1 certificates.
  • Their certificates are valid for only 3 months.
  • Their SAN certificates allow only 100 names.
  • The automated software doesn’t handle some stuff: Multiple vhosts in one conf file, load balancers, hosts on a LAN
  • The automated software can break your configuration

I started with my development server. It is both my easiest case because it isn’t load balanced and it isn’t user facing so I can have down time there while I experiment.

I started by installing certbot, their automated tool. Under Ubuntu that is really easy:

sudo apt-get install certbot

Then I listed all the host names for the dev server (126 of them). I then requested their software automatically generate and install an Apache certificate for all of them:

# This was a BAD idea, never do this with over 100 host names
sudo certbot --apache -d devserver.example.com -d site1.devserver.example.com site2.devserver.example.com ...

Naturally it failed. I can’t say I expected it to succeed. I made some adjustments and ran it again. My account to hit rate limits and get locked. Reading the help, it would take a week for me to be able to use the account again. Ack!

Luckily there was nothing on that account that I actually needed. I was able to delete /etc/letsencrypt and /var/lib/letsencrypt and start over with a new account.

This time I just created a certificate for a single one of my sub-domains on the server. It worked! Let me examine what it did:

  • It contacted LetsEncrypt and got a nonce password
  • It shut down the webserver and configured /.well-known/acme-challenge/ to serve a nonce password on that virtual host
  • LetsEncrypt retrieved the nonce and issued the certificate
  • It shut down the webserver again, copied the virtual host directive, and added SSL directives to the copy
  • The SSL directives include changing how logging works!

I don’t like any of this. I don’t like all the shutdown and restarts. I would have to make configuration changes in two different files. I am already logging the vhost. I’m not sure why they would change logging for that as part of enabling SSL. Duplicating logging is the kind of change that will quickly run me out of disk space. I backed out their changes.

Instead of modifying the Apache configuration and wrecking my server, they have an option to tell it where the document root is. It will just stick the nonce files into a directory there and download the certificates for you to install yourself. I think I can make that work.

The challenge is that many of my virtual hosts are just reverse proxies. They either proxy requests to a backend server on another port (like Tomcat) or proxy to another server entirely (like my NAS). There aren’t document root directories locally for many of the hosts.

Let us configure Apache to put a /.well-known/acme-challenge/ directory on each and every host and serve them out of a central directory. First I create a configuration file for it:

Alias "/.well-known/acme-challenge" "/var/www/acme-challenge/.well-known/acme-challenge"
# Don't allow this directory to be reverse proxied
ProxyPass /.well-known/acme-challenge/ !
<Directory /var/www/acme-challenge/>
    AllowOverride All
    Require all granted
</Directory>

Then make the directory, put a test file in it, and enable the configuration:

sudo mkdir -p /var/www/acme-challenge/.well-known/acme-challenge/
sudo echo "Hello World" | sudo tee /var/www/acme-challenge/.well-known/acme-challenge/hello.txt
sudo a2enconf acme-challenge
sudo service apache2 restart

I test it and I can access http://example.com/.well-known/acme-challenge/hello.txt under most of my host names. The only ones that I can’t are proxies that require a basic authentication password. I had put the basic auth directives in a <Location /> configuration which would also apply to this directory. I just changed that configuration to <Proxy *> instead so only the proxied files were password protected.

Now I can use a command like this to generate the certificates:

certbot certonly --staging --agree-tos --webroot -w /var/www/acme-challenge/ -d 'devserver.example.com,www.devserver.example.com,site1.devservert.example.com'

This will generate a single SAN certificate with all the names listed. Note that I’m using --staging now to test all the commands I use so that I don’t hit the account limits again.

Then I can use that certificate in a single HTTPS virtual host that reverse proxies all the requests to HTTP.

<VirtualHost *:443>
    SSLEngine on
    SSLProtocol all -SSLv2 -SSLv3
    SSLCipherSuite ALL:!DH:!EXPORT:!RC4:+HIGH:+MEDIUM:!LOW:!aNULL:!eNULL

    SSLCertificateFile /etc/letsencrypt/live/devserver.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/devserver.example.com/privkey.pem

    RequestHeader set X-Forwarded-Proto "https"
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / http://localhost:80/
</VirtualHost>

Now the development server has a full set of certificates. On to the production web server.

The production server is load balanced. We have to get each of the web servers to respond with the appropriate challenge nonce when asked.

I’ve noticed LetsEncrypt allows this directory to redirect. Since we already have the development server set up for generating certificates, we will generate the certificates on the development server. The live site just needs to redirect the /.well-known/acme-challenge to it.

That can be accomplished by using the same acme-challenge.conf and /var/www/acme-challenge on each of the live servers. Then we just add /var/www/acme-challenge/.well-known/acme-challenge/.htaccess with the contents:

Redirect 302 / https://devserver.example.com/

With that in place I can verify that http://example.com/.well-known/acme-challenge/hello.txt redirects to https://devserver.example.com/.well-known/acme-challenge/hello.txt.

Certbot and LetsEncrypt are happy with this setup. Now I generate certificates for the live site on the development server using the same command line with a different set of host names.

Now that I have certificates generated for the live site, I can use scp to push them out to the live servers.

Now would be a good time to get renewals working. I can run a command to renew the certificates:

 sudo certbot --staging renew --force-renewal 

I was expecting to be able to use the --renew-hook option to have it call a script when it was done renewing, but I could not get that working. Ok, I’ll have to parse the output of the renew command and react appropriately. Because it will run from a cron job, I also want it to be silent unless it renews a cert or fails. The --staging needs to be removed from the script once it is working with live data. It can be tested with the --force-renew argument to certbot.

#!/bin/sh

OUT="$(mktemp)"

certbot --staging renew > $OUT
exitstat=$?

if ! grep --quiet 'No renewals were attempted' $OUT
then
    cat $OUT
fi

if grep --quiet /example.com.*success $OUT
then
    scp-live-script.sh
fi

if grep --quiet devserver.*success $OUT
then
    echo "Reloading apache conf on development server"
    service apache2 reload
fi

exit $exitstat

Just set that up to run nightly by creating /etc/cron.d/letsencryptrenew

22 1 * * * root   /path/to/lets-encrypt-renew.sh

Producing W3C Datetime Format with Timezone in Java

W3C Datetime (also known as ISO 8601) is a standard machine readable way of formatting dates. In its most specific form, dates can be printed with time and time zone in this format:

YYYY-MM-DDThh:mm:ss.sTZD

For example:

1997-07-16T19:20:30.45+01:00

It is not as easy to produce dates in this format in Java as it should be. I needed to format dates like this for creating an XML Sitemap file.

(more…)


Graphing the performance of sub-domains in Google Analytics

My currency conversion website coinmill.com has a subdomain for each language.  It supports over 40 languages.  The entire domain is tracked in a single Google Analytics account.    I wanted to know which subdomains were performing best and which were performing worst, and how that was changing over time. (more…)