🏡 Frank Zhao

ACME DNS challenge with private CA

9 min read

Background

Since I have set up step-ca to provide Ingress certificates with private CA, I can use step-ca as the root CA for issuing certificates on my network. Since step-ca supports using an ACME provisioner, we can use the DNS-01 challenge with an ACME client.

There are also HTTP-01 and TLS-ALPN-01 challenge types available, however the require ports to be opened which can be less practical.

The first use case for this as part of my homelab was to use update the certificate used by the OPNsense web GUI. OPNsense has a plugin os-acme-client that provides an ACME client, which uses acme.sh under the hood.

A complexity I have in my setup is that I am using Unbound DNS, which is not an authoritative DNS server and cannot provide the CNAME and TXT records I will need for the DNS-01 challenge method. Due to this limitation, I chose to use a separate lightweight DNS server, acme-dns to only serve TXT records.

Prototyping

Since I know the os-acme-client plugin uses acme.sh for handing the ACME process, I tried to get a working setup using just the acme.sh script first. Once I’ve confirmed this works, I can the move this configuration into OPNsense.

In the following examples, I have configured the following static DNS records:

  • ca.juju.net → step-ca
  • acme.juju.net → acme-dns
  • opnsense.juju.net → OPNsense

Step CA

Create ACME provisioner

To set up Step CA, we need to add an ACME provisioner.

step ca provisioner add acme --type ACME

I also edit the config/ca.json to specify only using DNS-01 as the allowed challenge.

{
	"ACME",
	"name": "acme",
	"challenges": ["dns-01"],
	...
}

Then restart step-ca.

systemctl restart step-ca.service

Issue certificate for acme-dns

Issue a certificate and key pair for acme-dns to use in the next step for its API. Since the step-ca root certificate has been added to the system CA bundle, it will be trusted. To issue this cert, we use the JWK issuer. Note that this certificate needs to have a perpetual (or very long) validity.

$ STEPPATH=/etc/step-ca step ca certificate acme.juju.net acme.crt acme.key \
  --password-file /etc/step-ca/password.txt \
  --not-after 87660h \
  --san 192.168.25.5 \
  --san acme.juju.net

The certificate acme.crt and key acme.key will be written to the working directory.

acme-dns

Setup acme-dns server

To set up acme-dns, I build the binary, create a new user and systemd service for the server.

git clone https://github.com/joohoi/acme-dns
cd acme-dns
export GOPATH=/tmp/acme-dns
go build
sudo mv acme-dns /usr/local/bin
sudo mkdir /etc/acme-dns
sudo adduser --system --gecos "acme-dns" --disabled-password --group --home /var/lib/acme-dns acme-dns
sudo cp acme-dns.service /etc/systemd/system/acme-dns.service

Copy the certificate and key generated previously in Issue certificate for acme-dns to /etc/acme-dns/ and ensure they are readable by the acme-dns user.

sudo cp /etc/step-ca/acme.key /etc/acme-dns/
sudo cp /etc/step-ca/acme.crt /etc/acme-dns/
sudo chown acme-dns /etc/acme-dns/acme*

Set up the configuration:

$ cat /etc/acme-dns/config.cfg
[general]
# DNS interface. Note that systemd-resolved may reserve port 53 on 127.0.0.53
# In this case acme-dns will error out and you will need to define the listening interface
# for example: listen = "127.0.0.1:53"
listen = "0.0.0.0:53530"
# protocol, "both", "both4", "both6", "udp", "udp4", "udp6" or "tcp", "tcp4", "tcp6"
protocol = "both"
# domain name to serve the requests off of
domain = "acme.juju.net"
# zone name server
nsname = "acme.juju.net"
# admin email address, where @ is substituted with .
nsadmin = "frank.juju.net"
# predefined records served in addition to the TXT
records = [
    # domain pointing to the public IP of your acme-dns server 
    "acme.juju.net. A 192.168.25.5",
    # specify that acme.example.org will resolve any *.acme.example.org records
    "acme.juju.net. NS juju.net.",
]
# debug messages from CORS etc
debug = false
 
[database]
# Database engine to use, sqlite3 or postgres
engine = "sqlite3"
# Connection string, filename for sqlite3 and postgres://$username:$password@$host/$db_name for postgres
# Please note that the default Docker image uses path /var/lib/acme-dns/acme-dns.db for sqlite3
connection = "/var/lib/acme-dns/acme-dns.db"
# connection = "postgres://user:password@localhost/acmedns_db"
 
[api]
# listen ip eg. 127.0.0.1
ip = "0.0.0.0"
# disable registration endpoint
disable_registration = false
# listen port, eg. 443 for default HTTPS
port = "8445"
# possible values: "letsencrypt", "letsencryptstaging", "cert", "none"
tls = "cert"
# only used if tls = "cert"
tls_cert_privkey = "/etc/acme-dns/acme.key"
tls_cert_fullchain = "/etc/acme-dns/acme.crt"
# only used if tls = "letsencrypt"
acme_cache_dir = "api-certs"
# optional e-mail address to which Let's Encrypt will send expiration notices for the API's cert
notification_email = ""
# CORS AllowOrigins, wildcards can be used
corsorigins = [
    "*"
]
# use HTTP header to get the client ip
use_header = false
# header name to pull the ip address / list of ip addresses from
header_name = "X-Forwarded-For"
 
[logconfig]
# logging level: "error", "warning", "info" or "debug"
loglevel = "debug"
# possible values: stdout, TODO file & integrations
logtype = "stdout"
# file path for logfile TODO
# logfile = "./acme-dns.log"
# format, either "json" or "text"
logformat = "text"

Some notes:

  • I use port 53530 for listening to DNS because I have Pi-hole serving on port 53.
  • The certificate and key used in tls_cert_privkey and tls_cert_fullchain are manually issued from Step CA in the previous step Issue certificate for acme-dns.

Enable and start the service.

sudo systemctl enable acme-dns.service
sudo systemctl start acme-dns.service

Register a client

$ curl -s -X POST https://acme.juju.net:8445/register | jq
{
  "username": "3b028baf-05e2-48e6-a2b7-accc5fa88e1b",
  "password": "32cCdcQ9wEOB9JfFKMeEFYXGRBZM_4Btn2zGy89U",
  "fulldomain": "89d786ab-54b8-4d58-8217-8febffcd43d6.acme.juju.net",
  "subdomain": "89d786ab-54b8-4d58-8217-8febffcd43d6",
  "allowfrom": []
}

Make note of these output values.

Suppose we want to request a certificate for opnsense.juju.net. In /etc/acme-dns/config.cfg, add the configuration for the CNAME ‘magic record’ _acme-challenge.opnsense.juju.net that points to the fulldomain from the previous step.

records = [
    # domain pointing to the public IP of your acme-dns server 
    "acme.juju.net. A 192.168.25.5",
    # specify that acme.example.org will resolve any *.acme.example.org records
    "acme.juju.net. NS juju.net.",
    "_acme-challenge.opnsense.juju.net. CNAME 89d786ab-54b8-4d58-8217-8febffcd43d6.acme.juju.net",
]

Response from dig:

$ dig @192.168.25.5 -p 53530 _acme-challenge.opnsense.juju.net
 
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @192.168.25.5 -p 53530 _acme-challenge.opnsense.juju.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48849
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.opnsense.juju.net. IN	A
 
;; ANSWER SECTION:
_acme-challenge.opnsense.juju.net. 3600	IN CNAME 89d786ab-54b8-4d58-8217-8febffcd43d6.acme.juju.net.
 
;; Query time: 0 msec
;; SERVER: 192.168.25.5#53530(192.168.25.5) (UDP)
;; WHEN: Sun Feb 02 23:43:06 AEDT 2025
;; MSG SIZE  rcvd: 159

acme.sh

Set login details

Set ACMEDNS_BASE_URL to the URL of the acme-dns server API. The remaining values are from the client registration in Register a client.

export ACMEDNS_BASE_URL="https://acme.juju.net:8445"
export ACMEDNS_USERNAME="3b028baf-05e2-48e6-a2b7-accc5fa88e1b"
export ACMEDNS_PASSWORD="32cCdcQ9wEOB9JfFKMeEFYXGRBZM_4Btn2zGy89U"
export ACMEDNS_SUBDOMAIN="89d786ab-54b8-4d58-8217-8febffcd43d6"

Set up DNS forwarding

When resolving the DNS validation TXT records, step-ca will use the system resolver. This means we need to set up Unbound to forward queries for _acme-challenge.domain and subdomain.domain to acme-dns.

There seems to be a --resolver flag in the step ca command to override which DNS server is used for query, but this does not seem to have any effect for me. (See: Github PR) If this were working, I suspect I would be able to avoid the need for query forwarding.

Request a certificate

$ acme.sh --issue --dns 'dns_acmedns' \
  -d opnsense.juju.net \
  -w opnsense \
  --server 'https://ca.juju.net:8444/acme/acme/directory' \
  --dnssleep 10 \
  --debug
...
[Sun 02 Feb 2025 23:46:05 AEDT] Your cert is in: /home/frank/.acme.sh/opnsense.juju.net_ecc/opnsense.juju.net.cer
[Sun 02 Feb 2025 23:46:05 AEDT] Your cert key is in: /home/frank/.acme.sh/opnsense.juju.net_ecc/opnsense.juju.net.key
[Sun 02 Feb 2025 23:46:05 AEDT] The intermediate CA cert is in: /home/frank/.acme.sh/opnsense.juju.net_ecc/ca.cer
[Sun 02 Feb 2025 23:46:05 AEDT] And the full-chain cert is in: /home/frank/.acme.sh/opnsense.juju.net_ecc/fullchain.cer
[Sun 02 Feb 2025 23:46:05 AEDT] _on_issue_success
[Sun 02 Feb 2025 23:46:05 AEDT] '' does not contain 'dns'

Great, a new certificate is issued. Now we can configure OPNsense to automate this.

Configuring OPNsense

Install the os-acme-client plugin for OPNsense, then we can configure it to request certificates from Step CA.

Update truststore

OPNsense system CA bundle must contain the Step CA root CAs for the ACME client to work. Under System > Trust > Authorities, import the Step CA root and intermediate certificates.

ACME client settings

Under Services > ACME Client > Settings:

  • Enable plugin - Checked (although this appears to have no impact.)
  • Auto renewal - Checked
  • Log Level - debug (this will be useful for troubleshooting)

Accounts

Register an account with Step CA under ACME Client > Accounts. Select “Custom CA URL” in the dropdown and specify the URL to the Step CA provisioner. This will be in the format:

https://<hostname>/acme/<provisoner name>/directory

Save and click register on the new row. The status should change to “OK (registered)“.

Challenge type

Under ACME Client > Challenge Types, create a new DNS-01 challenge type.

  • Enabled - Checked
  • Name / Description - Set
  • Challenge type - DNS-01
  • DNS Service - ACME DNS
  • DNS Sleep time - 10 (this needs to be set to a non zero value to avoid checking public DNS resolvers).
  • ACME DNS - Set User, password and subdomain to values obtained in Register a client.
  • ACME DNS URL - Set to the URL of the ACME DNS server.

Automations

After the certificate renewal occurs, we need to restart the OPNsense web UI so that it can start using the refreshed certificate. We can set up an automation to perform this action automatically.

Under ACME Client > Automations, create a new automation that has the “Run Command” set to “Restart OPNsense Web UI”.

Certificates

Under ACME Client > Certificates, add a new certificate request.

  • Enabled - Checked
  • Common Name - set to the hostname you want to issue this certificate to. In this case I use opnsense.juju.net.
  • ACME Account - Select the Step CA account created in the previous step.
  • Challenge Type - Select the ACME DNS challenge created in the previous step.
  • Auto Renewal - Checked
  • Renewal Interval - 1 (days)
  • Automations - Restart Web UI

Once saved, try issuing a new certificate using the “Issue or Renew Certificate” button on the row. If all has been configured correctly, a new certificate will be issued. Otherwise you will need to look at the plugin logs for errors.

You can also verify that the new certificate is listed under System > Trust > Certificates.

Use the new certificate

Once the certificate is available for use, we can configure the OPNsense web UI to use this certificate. In System > Settings > Administration, select the issued certificate in the “SSL Certificate” row. Remember to restart the web UI to see if the new certificate is presented.

Add renew job to cron

To renew certificates automatically, we can set up a cron job. In ACME Client > Settings > Update Schedule, set up a cron job to renew ACME certificates once every 24 hours.

Graph View