Don't Get Rate-Limited: Use Let's Encrypt Staging

Don't Get Rate-Limited: Use Let's Encrypt Staging

3 minute read

While getting Pomerium Core set up this week for a demo, I ran into Let's Encrypt rate limits. Here’s how using Let's Encrypt staging can save you time when setting up auto-provisioned certificates.

For context, I work at Pomerium. This was part of a local demo stack I was building with Docker Compose. Pomerium is an identity-aware proxy (think of it as an application gateway) that secures access to your apps with built-in policy enforcement, SSO, microsegmentation etc. All the Zero Trust security goodies.

I had autocert enabled for Pomerium in my config.yaml:


autocert: true

What is autocert? permalink

autocert is a Pomerium feature that automatically provisions and renews TLS certificates for your routes using Let’s Encrypt. No manual cert management, no copying files around — just tell Pomerium to handle it and it will issue valid certificates on the fly.

This works great… until you accidentally annoy Let’s Encrypt while testing.

What happened permalink

I was adding a few test routes (verify.demo.maisonlab.dev, authenticate.demo.maisonlab.dev, etc.) and restarting Pomerium repeatedly to tweak my config.

Then this showed up in the logs:


HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of domains in the last 168h0m0s

Rate limited. No more certs for verify.demo.maisonlab.dev for 7 days.

Why this happened permalink

First, I had the wrong volume path for the autocert cache:


services:
  pomerium:
    image: pomerium:mcp
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./config.yaml:/pomerium/config.yaml:ro
-      - ./pomerium-cache:/pomerium/.cache
+      - ./pomerium-cache:/data/autocert
+      # this is where Pomerium stores autocert data
    networks:
      - pom-network

# ...

Without the correct cache volume, Pomerium wasn’t persisting issued certs between Docker compose restarts. Each time I restarted the Pomerium container, it tried to re-request a new cert and Let’s Encrypt counted every attempt.

Second, even failed attempts count toward Let’s Encrypt’s certificate provisioning quota. 🫠

The fix: use staging while testing permalink

If you’re working on a new Pomerium setup, use Let’s Encrypt’s staging environment while testing certificate provisioning:


autocert: true
# autocert_ca defaults to https://acme-v02.api.letsencrypt.org/directory
autocert_ca: https://acme-staging-v02.api.letsencrypt.org/directory

See the autocert_ca docs. When you point Pomerium at the staging certificate authority (CA):

  • You can test your flow without worrying about production rate limits.
  • The only catch: staging certs aren’t trusted by browsers (you’ll get a warning), but that’s fine for local testing.

Once your config is good and your cert cache is persisting properly, flip back to production by removing the autocert_ca line.

TL;DR permalink

If you see:


HTTP 429 urn:ietf:params:acme:error:rateLimited

You’re in rate-limit jail for 7 days.

Fred Armissen as a dictator saying Straight to jail!

  • Make sure your autocert volume is correct for the Pomerium container, i.e. (/data/autocert).
  • Use Let’s Encrypt staging while testing.
  • Flip back to production once you’re happy.

One last note: This isn’t a Pomerium-specific quirk. It’s how Let’s Encrypt’s rate limits work in general. If you’re automating certificate issuance with any tool, the same caution applies when testing.

Hope this saves someone else a bit of time (and avoids a few unnecessary retries). If you’re building out your own Pomerium setup and hit this, now you know what’s going on. Also, if you're using Pomerium, hit me up! I'd love to know how you're using it or what issues you run into, if any.

If you want to stay in touch, all my socials are on nickyt.online.

Photo by Andrew Wulf on Unsplash