Using Let's Encrypt in Development with NGINX and AWS Route 53
Read in 9 minutes
I frequently see people struggling to set up HTTPS in development. If you’re a long time developer, you may have done this in the past with self-signed certificates, buying your own certificates and tweaking your hosts file, or using tools like puma-dev. While these approaches work to an extent, Let’s Encrypt changed the game, at least for me.
With Let’s Encrypt and a DNS provider like AWS Route 53, you’ll be able to run HTTPS with wildcard subdomains without having to mess with your /etc/hosts file, or having to install tools that create a custom DNS resolver.
I’m going to focus on macOS, my development environment, but you can pretty much follow the same instructions everywhere. Just install the software dependencies as needed.
Configuring AWS Route 53
On the dashboard, select “Route 53” under “Networking & Content Delivery”. You can also type “route 53” on the search field. You’ll be redirected to AWS Route 53’s dashboard.
Now, on the sidebar, click on “Hosted Zones”.
You’ll need a domain for this, so you have a few options:
- Use a domain you already have and it’s not being used (e.g.
fnando.com) - Use a subdomain on a existing domain that’s being used (e.g.
dev.fnando.com). - Buy a new domain, maybe one of those fancy
.dev, which convey exactly what you’re doing (e.g.fnando.dev).
I decided to buy yet another domain and went with option #3, just because it’s shorter, specially when doing wildcard domains (something.dev.fnando.com vs something.fnando.dev). It looks nicer too! 🤓
Once you create your hosted zone, you have to configure your domain and point its DNS to AWS Route 53. The hosts you’ll need are defined under the record type NS. Go to your domain provider and set this up. I use Namecheap, so this is how you do it:
Back to AWS Route 53. Let’s create two A records that point your DNS to your development machine, in this case the loopback address 127.0.0.1.
The first record will handle fnando.dev. Click on “Create Record Set”, choose “A - IPv4 address” under the record type and set the value to 127.0.0.1. Make sure you don’t type anything under the name; otherwise, you’d be pointing a subdomain instead.
The second record will handle wildcard subdomains. Click on “Create Record Set” once again, choose “A - IPv4 address”, but this time use * as the record name. The value should be 127.0.0.1, just like before.
And the waiting game starts. You now have to wait until your DNS is propagated completely, but that shouldn’t take long. You can check it using dig on the command-line.
$ dig +short fnando.dev A
127.0.0.1
$ dig +short '*.fnando.dev' A
127.0.0.1
While you wait, you can set up a new AWS credential restricted to this domain. Before we move on, look at your browser’s url: you’ll need the zone id, so copy this value or write it down somewhere.
Now, let’s create the user and a policy. This can be done on AWS IAM, so search for this option under the services menu.
On the sidebar, click on “Policies”, then “Create Policy”.
Use the JSON below as your policy. Remember to replace YOUR_ZONE_ID with your zone id.
{
"Version": "2012-10-17",
"Id": "letsencrypt-mac policy",
"Statement": [
{
"Effect": "Allow",
"Action": ["route53:ListHostedZones", "route53:GetChange"],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["route53:ChangeResourceRecordSets"],
"Resource": ["arn:aws:route53:::hostedzone/YOUR_ZONE_ID"]
}
]
}
Now, click on “Review policy”. Give it a recognizable name and click on “Create policy”.
It’s time to create a new user. On the sidebar, click on “Users” and then “Add User”. Give it a name like letsencrypt-mac, or something that describes your machine. You’ll also have to select “Programmatic Access” under “Access type”.
Click “Next”. Now we’re going to select the policy we’ve created a few steps before. Click on “Attach existing policies directly” and search for your policy, in this case letsencrypt-mac.
Click “Next: Tags”, then “Next: Review”. Finally, click “Create User”.
This step is important: you’ll be presented with your access and secret keys. Save both of them somewhere safe, like your password manager.
As far as AWS goes, you’re all set up. Now, let’s configure certbot, the command-line interface that’ll interact with Let’s Encrypt.
Configuring certbot
Since I’m a macOS user, I use homebrew. To install certbot, run brew install certbot. You can also find instruction for other system on certbot’s website.
$ brew install certbot
==> Downloading https://homebrew.bintray.com/bottles/certbot-1.4.0.catalina.bottle.tar.gz
Already downloaded: /Users/fnando/Library/Caches/Homebrew/downloads/f25750b88db0e526ac7fce38587eade58ef847892744acd31091a7d8fa4cdda4--certbot-1.4.0.catalina.bottle.tar.gz
==> Pouring certbot-1.4.0.catalina.bottle.tar.gz
🍺 /usr/local/Cellar/certbot/1.4.0: 1,451 files, 12.5MB
To automatically issue certificates that are validated against AWS Route 53’s DNS, we need to install a certbot plugin called certbot-dns-route53. We can Python’s pip.
$ pip3 install certbot-dns-route53
...
Installing collected packages: setuptools, zope.interface, pycparser, cffi, six, cryptography, PyOpenSSL, josepy, pytz, pyrfc3339, urllib3, certifi, chardet, idna, requests, requests-toolbelt, acme, docutils, python-dateutil, jmespath, botocore, s3transfer, boto3, parsedatetime, configobj, ConfigArgParse, distro, zope.deprecation, zope.event, zope.hookable, zope.proxy, zope.deferredimport, zope.component, certbot, certbot-dns-route53
Successfully installed ConfigArgParse-1.2.3 PyOpenSSL-19.1.0 acme-1.4.0 boto3-1.13.16 botocore-1.16.16 certbot-1.4.0 certbot-dns-route53-1.4.0 certifi-2020.4.5.1 cffi-1.14.0 chardet-3.0.4 configobj-5.0.6 cryptography-2.9.2 distro-1.5.0 docutils-0.15.2 idna-2.9 jmespath-0.10.0 josepy-1.3.0 parsedatetime-2.5 pycparser-2.20 pyrfc3339-1.1 python-dateutil-2.8.1 pytz-2020.1 requests-2.23.0 requests-toolbelt-0.9.1 s3transfer-0.3.3 setuptools-46.4.0 six-1.15.0 urllib3-1.25.9 zope.component-4.6.1 zope.deferredimport-4.3.1 zope.deprecation-4.4.0 zope.event-4.4 zope.hookable-5.0.1 zope.interface-5.1.0 zope.proxy-4.3.5
You can verify that certbot can see the plugin by running certbot plugins.
$ certbot plugins
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* dns-route53
Description: Obtain certificates using a DNS TXT record (if you are using AWS
Route53 for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-route53 =
certbot_dns_route53._internal.dns_route53:Authenticator
* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Now, let’s generate the certificates. The first thing to know is that you need to export your AWS credentials as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. It’s up to you how you want to manage these variables. Personally, I like adding them to ~/.zsh/user.sh, which is then loaded by my ~/.zshrc file. For this article, I’ll just export them before using them.
$ export AWS_ACCESS_KEY_ID=AKIAY7V6RRKVYA3Z4MGA
$ export AWS_SECRET_ACCESS_KEY='REDACTED_SECRET'
To generate a certificate, use the command certbot certonly. Notice that I’m specifying local directories; this is required if you don’t want to use sudo. After the process is complete, the certificate will be saved to ~/local/letsencrypt/live/fnando.dev. If you’re not sure if everything is set up accordingly, use the switch --dry-run; this will run certbot on their staging environment, which has a higher limit for failures. In production, you will be blocked from generating new certificates for a hour after a certain number of failures.
certbot certonly \
-n \
--agree-tos \
--email user@example.com \
-d fnando.dev \
-d '*.fnando.dev' \
--dns-route53 \
--preferred-challenges=dns \
--logs-dir /tmp/letsencrypt \
--config-dir ~/local/letsencrypt \
--work-dir /tmp/letsencrypt
Once the command finishes running, you’ll see something like this:
Saving debug log to /tmp/letsencrypt/letsencrypt.log
Found credentials in environment variables.
Plugins selected: Authenticator dns-route53, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for fnando.dev
dns-01 challenge for fnando.dev
Waiting for verification...
Cleaning up challenges
Non-standard path(s), might not work with crontab installed by your operating system package manager
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/Users/fnando/local/letsencrypt/live/fnando.dev/fullchain.pem
Your key file has been saved at:
/Users/fnando/local/letsencrypt/live/fnando.dev/privkey.pem
Your cert will expire on 2020-08-23. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
This is all we need to do. When your certificates are about to expire, you’ll receive an email from Let’s Encrypt.
You’ll receive an reminder to renew your certificates via email. To renew your certificates, just run the same command above (i.e. certbot certonly).
certbot certonly \
-d fnando.dev \
-d '*.fnando.dev' \
--dns-route53 \
--preferred-challenges=dns \
--logs-dir /tmp/letsencrypt \
--config-dir ~/local/letsencrypt \
--work-dir /tmp/letsencrypt
Saving debug log to /tmp/letsencrypt.log
Plugins selected: Authenticator dns-route53, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for fnando.dev
dns-01 challenge for fnando.dev
Waiting 30 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/Users/fnando/local/letsencrypt/live/fnando.dev/fullchain.pem
Your key file has been saved at:
/Users/fnando/local/letsencrypt/live/fnando.dev/privkey.pem
Your cert will expire on 2021-01-29. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Once it’s done, remember to restart the webserver.
Configuring NGINX
Once again, we can install NGINX using homebrew. Just run the command brew install nginx.
$ brew install nginx
==> Downloading https://homebrew.bintray.com/bottles/nginx-1.17.10.catalina.bottle.tar.gz
Already downloaded: /Users/fnando/Library/Caches/Homebrew/downloads/d7118d9cc53ef3be545ac049f7e50aa30f2378f673aa925702deaa6117fb403c--nginx-1.17.10.catalina.bottle.tar.gz
==> Pouring nginx-1.17.10.catalina.bottle.tar.gz
==> Caveats
Docroot is: /usr/local/var/www
The default port has been set in /usr/local/etc/nginx/nginx.conf to 8080 so that
nginx can run without sudo.
nginx will load all files in /usr/local/etc/nginx/servers/.
To have launchd start nginx now and restart at login:
brew services start nginx
Or, if you don't want/need a background service you can just run:
nginx
==> Summary
🍺 /usr/local/Cellar/nginx/1.17.10: 25 files, 2.1MB
Given that I develop web applications all the time, I like starting NGINX on boot. To do it so, run the command sudo brew services start nginx, and homebrew will take care of copying the launch file to /Library/LaunchDaemons/homebrew.mxcl.nginx.plist.
$ sudo brew services start nginx
Warning: Taking root:admin ownership of some nginx paths:
/usr/local/Cellar/nginx/1.17.10/bin
/usr/local/Cellar/nginx/1.17.10/bin/nginx
/usr/local/opt/nginx
/usr/local/opt/nginx/bin
/usr/local/var/homebrew/linked/nginx
This will require manual removal of these paths using `sudo rm` on
brew upgrade/reinstall/uninstall.
Warning: nginx must be run as non-root to start at user login!
==> Successfully started `nginx` (label: homebrew.mxcl.nginx)
Yeah, I know… sudo. But that’s how you can hit https://fnando.dev instead of having to specify a non-privileged port. Another thing you could do is setting up /usr/local permission to the group admin, so it’s up to you.
Your NGINX configuration must be added to /usr/local/etc/nginx/servers/. I like to use the apex domain name (i.e. the domain name without subdomain) as the file name, so I’m going to create a /usr/local/etc/nginx/servers/fnando.dev.conf file with the content below.
upstream fnando_dev {
server 127.0.0.1:3000 max_fails=0;
server 127.0.0.1:4567 max_fails=0;
server 127.0.0.1:5000 max_fails=0;
server 127.0.0.1:5001 max_fails=0;
server 127.0.0.1:9292 max_fails=0;
server 127.0.0.1:9393 max_fails=0;
}
server {
listen 80;
listen 443 ssl;
server_name fnando.dev;
ssl_certificate /Users/fnando/local/letsencrypt/live/fnando.dev/fullchain.pem;
ssl_certificate_key /Users/fnando/local/letsencrypt/live/fnando.dev/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://fnando_dev;
}
}
server {
listen 80;
listen 443 ssl;
server_name *.fnando.dev;
ssl_certificate /Users/fnando/local/letsencrypt/live/fnando.dev/fullchain.pem;
ssl_certificate_key /Users/fnando/local/letsencrypt/live/fnando.dev/privkey.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://fnando_dev;
}
}
Notice that you cannot use ~ to indicate your home directory, so use the full path instead. Another thing is that you can specify any number of servers on the upstream statement, so make sure your framework’s port is listed there.
Now, restart the server with sudo brew services restart nginx.
$ brew services restart nginx
Stopping `nginx`... (might take a while)
==> Successfully stopped `nginx` (label: homebrew.mxcl.nginx)
==> Successfully started `nginx` (label: homebrew.mxcl.nginx)
If you’re all set up, you can hit your application on your custom domain, like https://fnando.dev. To quickly test it, start your web application and hit that url.
As you can see, this is a 100% valid certificate.
And subdomains work just fine. 😎
Wrapping up
You may be wondering why would you develop using HTTPS, and the answer is that many things require HTTPS, like webauthn. You can check a full list of features that require secure context on Mozilla’s website.
I tried all sort of combinations in the past, but this is the best option by far. No hacks, no crazy “add certificate roots to keychain” setups. Thanks, Let’s Encrypt.