May 10, 2023

Docker in Production: What We Learned the Hard Way

Docker is great until something goes wrong and you need to debug a container that's been deleted.

Here's what we learned running Docker in production.

Log Everything

Use a proper logging driver. JSON-file is fine for dev. In production, ship logs to Cloud Logging or similar.

Health Checks Matter

Docker's health checks saved us more times than I can count. If a container is running but not responding, kill it and restart.

Volume Management

Data in containers is ephemeral. Important data goes in volumes. Backups go somewhere else entirely.

Keep Images Small

We went from 1.2GB images to 200MB by using multi-stage builds and Alpine base images. Faster deployments, lower costs.

The Gotcha

Container restarts lose in-memory state. Obvious in hindsight, painful to discover at 3 AM.