Recently I redesigned jasonneurohr.com to be more of a microservice architecture (for fun!) and in doing so initially used Cloudflare to handle the frontend TLS for the website. I used Let’s Encrypt when the site was more monolithic to provide the TLS certificates and had not had time to work out how to implement Let’s Encrypt into my CI/CD pipeline, and I wanted to solve that problem. Hence this article was written. The primary challenge to solve was persisting the Let’s Encrypt configuration (the certificates primarily) when the CI/CD pipeline was pushing new container images to the container registry and subsequently instructing the AWS Elastic Container Service (ECS) to terminate the existing containers (which is intentional).

Rather then go to the trouble of trying to automate the initial Let’s Encrypt enrollment with certbot, since I never plan to, at least for the foreseeable future, intend to do anything other then perform renewals, and it adds complexity to the problem which I don’t need to solve for my use case. Also, the website is already running, so I can take advantage of that fact to perform the initial enrollment as opposed to solving the problem for a new website yet to be deployed.

Note, this post is written specifically for an NGINX container running in the Amazon Web Services (AWS) Elastic Container Server (ECS).

Let’s get started.

The CI/CD pipeline includes an NGINX container build task which is where Let’s Encrypt will need to be implemented. First, we need to solve the data persistence problem. To do this log into the AWS console and navigate to ECS. Then locate your Task Definition and create a new revision. Scroll down to the Volumes section and click Add volume. Complete the following:

• Name - the volume name
• Specify a volume driver - checked
• Volume type - Docker
• Driver - local
• Scope - shared. This is important as it is what makes the volume persistent
• Enable auto-provisioning - checked

Next, edit the Container Definition for your NGINX container. Scroll down to the STORAGE AND LOGGING section then next to Mount points select the volume name created previously and specify the path as /etc/letsencrypt which is the path certbot will save the Let’s Encrypt data.

Click Update to save the configuration, and then click Create to save the Task Definition revision.

For more information on the ECS, volumes see Using Data Volumes in Tasks.

Once the revision has been created, if you are using a ECS Service (like me), you will need to update the service. See the Updating a Service page if you don’t know how to do this.

Now that the NGINX container is running with a persistent volume at /etc/letsencrypt and since the website is already live we need to run certbot in the environment to perform the enrollment and update the existing NGINX configuration. To do this, connect to the NGINX container shell and install certbot-nginx. In my case the container is using Alpine Linux, so the commands may have to be adjusted if you follow these steps with a different distro.

~ # apk add –no-cache certbot-nginx

Next, run certbot to perform the enrollment for the website.

~ # certbot –nginx -d www.jasonneurohr.com,jasonneurohr.com –agree-tos -m email@jasonneurohr.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let’s Encrypt project and the non-profit
organization that develops Certbot? We’d like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for jasonneurohr.com
http-01 challenge for www.jasonneurohr.com
2019/05/18 11:35:05 [notice] 39#39: signal process started
Waiting for verification…
Cleaning up challenges
2019/05/18 11:35:11 [notice] 41#41: signal process started
Deploying Certificate to VirtualHost /etc/nginx/conf.d/default.conf
Deploying Certificate to VirtualHost /etc/nginx/conf.d/default.conf
2019/05/18 11:35:17 [notice] 43#43: signal process started

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you’re confident your site works on HTTPS. You can undo this
change by editing your web server’s configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press ‘c’ to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/conf.d/default.conf
Redirecting all traffic on port 80 to ssl in /etc/nginx/conf.d/default.conf
2019/05/18 11:35:58 [notice] 45#45: signal process started

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://www.jasonneurohr.com and
https://jasonneurohr.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=www.jasonneurohr.com
https://www.ssllabs.com/ssltest/analyze.html?d=jasonneurohr.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/www.jasonneurohr.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/www.jasonneurohr.com/privkey.pem
Your cert will expire on 2019-08-16. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the “certonly” option. To non-interactively renew *all* of
your certificates, run “certbot renew”
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.
- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

There are some critical parts of the output above.

• Certbot updated the NGINX configuration - Deploying Certificate to VirtualHost /etc/nginx/conf.d/default.conf etc. in the output
• Certbot has saved the Let’s Encrypt data at /etc/letsencrypt - Your account credentials have been saved in your Certbot configuration directory at /etc/letsencrypt. etc. in the output

Now NGINX is using Let’s Encrypt to provide the TLS encryption for the website. Awesome!. But in my case and I assume yours is similar if your reading this, we have some additional clean up/maintenance to do.

First, we need to take the configuration changes made to NGINX and update the CI/CD pipeline to include the updated configuration, or our next deployment will remove the configuration, and we don’t want that headache!

Second, we need to make sure that certbot is run at regular intervals to do the certificate renewal process for us. Let’s Encrypt certificates expire every 90 days, and I don’t know about you, but I don’t want to have to remember to perform the renewals manually! Again we need to update the CI/CD pipeline to a) include the install of certbot-nginx as part of the container image, and b) add a cron job to the NGINX container which will run the renewal command below.

/usr/bin/certbot renew -n

In my pipeline, I have a shell script with the renew command above copied into the Docker image in the /etc/periodic/hourly path.