Nowadays, an SSL is a necessity for any website in production, thus certificate issuing and renewal are routine tasks that can be easily automated by tools like Let’s Encrypt. These processes are performed using open-source certbot with the standalone authenticator. The main drawback here is the possible downtime of the web application server during the automatic certificate renewal and domain validation.
In order to overcome the problem with uptime during SSL certificates update, Jelastic created a built-in Let’s Encrypt Add-On with an advanced zero-downtime renewal algorithm.
Downtime Reasons while Updating SSL Certificate with certbot
The certbot command-line tool's main purpose is to configure an HTTPS server and automatically obtain a trusted certificate without human intervention. It performs the following tasks:
- proves that you are in control of the website
- receives a certificate and configures it on a web server
- tracks certificate expiration and updates it
- helps users revoke their certificate (if needed)
- forces certificate renewal when required
First, certbot initiates issuance of SSL certificate by sending Certificate Signing Requests (CSR). Then, it starts a Certificate Management Agent (CMA) which is a Python web server that listens to the specified port to handle a validation request from the Let's Encrypt CA. Before obtaining a certificate, it is necessary to prove that renewal was invoked by the domain administrator. The validation request always happens on port 80, which is usually occupied by the application listening to HTTP traffic. So, the web server which it is running on must be temporarily stopped to run CMA instead. And consequently such implementation causes application downtime.
Another issue that can cause downtime is that the Python web server can become stuck once it tries to handle a non-validation HTTP request that can come from any user. It happens since certbot runs the web server in a single-threaded mode and consequently it can’t handle multiple incoming requests.
Solution to Avoid Downtime during SSL Certificate Update
In order to omit the mentioned issues, Jelastic created a built-in Let’s Encrypt Add-On that includes an improved algorithm. It presupposes using iptables rules to integrate an intermediate reverse proxy layer into the traffic chain to filter HTTP traffic. The proxy is configured to route the request to the Python web server from Let’s Encrypt CA only. Other traffic is routed to the user’s application.
Iptables Routing to Proxy Port
The add-on does not stop the application's web server and redirects requests that come to website port 80 to the proxy port (e.g. 12345) via the iptables rules until validation is completed.
iptables -I INPUT -p tcp -m tcp --dport ${PROXY_PORT} -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport ${LE_PORT} -j ACCEPT
iptables -t nat -I PREROUTING -p tcp -m tcp ! -s 127.0.0.1/32 --dport 80 -j REDIRECT --to-ports ${PROXY_PORT}Where:
${PROXY_PORT} - 12345
${LE_PORT} - 12346
Intermediate Reverse Proxy
Tinyproxy is used as an intermediate proxy. It is a lightweight HTTP/HTTPS proxy-server built from scratch to be fast and compact. The startup speed is a key advantage of this software. Thus, we can ensure uninterruptible work for the user's application by switching traffic routes through the intermediate proxy layer. Its config file looks very simple and can be found in the Let’s Encrypt add-on repository:
User tinyproxy
Group tinyproxy
Port 12345
Timeout 6000
MaxClients 100
MinSpareServers 5
MaxSpareServers 20
StartServers 10
MaxRequestsPerChild 0
BindSame yes
DisableViaHeader Yes
ConnectPort 80
ConnectPort 12346
AddHeader "X-Forwarded-Proto" "http"
ReversePath "/" "http://127.0.0.1/"
ReversePath "/.well-known/acme-challenge/" "http://127.0.0.1:12346/.well-known/acme-challenge/"
ReverseOnly Yes
ReverseMagic YesHere the ReversePath directives enable reverse proxy support and define the HTTP traffic filtering rules. Any request that will come with the substring "/.well-known/acme-challenge/" will be routed to the Python web server that listens to port 12346 to get a validation file from the hidden .well-known/acme-challenge/ directory. Only Let's Encrypt CA generates such requests. And the rest of the HTTP traffic will be routed to the web application that listens to port 80. This is how the Jelastic Let’s Encrypt add-on proves that CSR was initiated by the domain administrator.
Use Case: Zero Downtime SSL Update in WordPress Websites
Now, let's see how it works in a real case scenario using a WordPress application as a sample. While WordPress package (Cluster or Standalone) installation, enable “Install Let’s Encrypt SSL with Auto-Renewal” option.
- WordPress Cluster
In the case of a WordPress clustered environment, traffic is redirected from all load balancers to the Master node (the first or initially created container in the layer), where the validation server is located.
- WordPress Standalone
Traffic flow in a WordPress standalone container (with LLSMP or LEMP certified templates) is shown in the illustration below.
According to the filtering rules, proxy outputs traffic either to the Python web server port 12346 or to port 80 of the user's web server.
Renewing Let’s Encrypt Certificate Using Webroot Authenticator
There is an additional method to issue new certificates or update existing ones which is called webroot. This method is preferable for highly loaded websites (thus soon will be added to WordPress cluster by default) as it does not interrupt web servers’ work and omit the risk of downtime or performance degradation.
The webroot can be used with certified PHP software stacks as in the WordPress packages. When launched, it obtains a path to the site's root directory. Then certbot creates a directory structure there for storing temporary file:
${webroot-path}/.well-known/acme-challenge
Let's Encrypt server checks for the presence of this file in the specified location with a request of this type:Let's Encrypt server checks for the presence of this file in the specified location with a request of this type:
http://example.com/.well-known/acme-challenge/KJr5D^ReTW4kY_Z6UIIkjlmlmOkyQgPr_7ArlLgtZE7Fg
If the requested temporary file (for example: "KJr5D^ReTW4kY_Z6UIIkjlmlmOkyQgPr_7ArlLgtZE7Fg") exists, then domain ownership will be successfully verified, and the certificate will be issued and uploaded to the users' website.
After completing the validation procedure, the certbot obtains the certificate file. Usually, the Let’s Encrypt SSL certificate file is stored in the /etc/letsencrypt/live directory where all the files are symlinks to the files in /etc/letsencrypt/archive directory.
So as you can see, there are options on how to achieve full uptime while updating SSL certificates. And to make it even more easy and automated, Jelastic implemented the required algorithm in Let's Encrypt add-on, as well as made this solution built-in to the WordPress cluster and standalone package. Get your own experience using free Let's Encrypt SSL certificates with zero downtime renewal procedure empowered by Jelastic PaaS.
