DevOps#

Day 2–3 · The bits between code and customer

Code in your editor helps no one. DevOps is the practice of getting code onto a machine, keeping it running, knowing when it isn't, and being able to fix it without rebuilding from scratch. For a forward deployed engineer, this often happens on a customer's laptop, in their network, against their constraints. You need to be self-sufficient.

This chapter is the longest in the bootcamp on purpose. It includes the Welzin Homelab Runbook - the actual notes from our internal home server build - because everything in there will appear, in some form, in your work.

Sub-chapter deep-dives#

This primer is the overview and the homelab runbook. Four sub-chapters go deeper on the pieces you'll reach for most - work through them as part of this chapter:

Outline#

1. Mental model - what "DevOps" actually means
2. Virtualisation - VMs, containers, when to use which
3. Docker fluency - images, containers, Compose, volumes, networks
4. Reverse proxies - Nginx, Caddy, Traefik, SSL/Let's Encrypt
5. Process management - systemd, supervisord, PM2
6. Monitoring - Prometheus + Grafana + Loki; alerting; SLOs in one paragraph
7. Backups and restores - restic, rsync; the only backup that exists is the one you've restored
8. Secrets management - Vault, AWS Secrets Manager, sealed-secrets, anything but plaintext
9. Networking on a server - firewalls, VPNs (Tailscale, WireGuard), DNS
10. Welzin Homelab Runbook - the full real-world example

101 Primer#

What DevOps actually is#

DevOps is the loop:

  plan → build → test → release → deploy → operate → observe → plan

A "DevOps engineer" title is overloaded - sometimes infra, sometimes platform, sometimes SRE. At Welzin, every engineer owns the loop for their code. There is no separate ops team to throw it to. If you wrote it, you can:

1. Build it into a container.
2. Run it on a server.
3. Observe it in Grafana.
4. Wake up when it breaks (during business hours, for the bootcamp's purposes).

VMs vs containers vs bare metal#

A container is not a tiny VM. It shares the host kernel, has near-zero startup cost, and is built from a declarative Dockerfile. This is what you'll use 95% of the time.

Docker fluency in one screen#

# Build an image from a Dockerfile in current dir
docker build -t welzin/api:dev .

# Run it, expose port, set env, name the container
docker run -d --name api -p 8000:8000 -e DATABASE_URL=... welzin/api:dev

# Inspect
docker ps                 # running containers
docker logs -f api        # tail logs
docker exec -it api sh    # shell into a running container
docker stats              # live CPU/mem

# Clean up
docker stop api && docker rm api
docker image prune -af    # free disk

A minimal Dockerfile for a Node service:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build
EXPOSE 8000
CMD ["node", "dist/server.js"]

Rules:

• Multi-stage builds for any compiled language. Keep the final image small.
• One process per container. Not one application, one process. If you need multiple, use Compose.
• Pin versions. node:20-alpine, not node:latest. Reproducibility beats laziness.
• Don't bake secrets in. Pass via env at runtime.

docker-compose for local dev#

# docker-compose.yml
services:
  api:
    build: .
    ports: ["8000:8000"]
    env_file: .env
    depends_on: [db]

  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: dev
    volumes: ["pgdata:/var/lib/postgresql/data"]
    ports: ["5432:5432"]

volumes:
  pgdata:

docker compose up -d and you have a reproducible local stack. Every Welzin repo should have one. New engineer onboarding = git clone && docker compose up.

Reverse proxies and TLS#

A server exposes one app. A real server exposes many - your API, a dashboard, a metrics endpoint - all behind one IP and HTTPS. That's what a reverse proxy does.

Caddy is the easy default. Single binary, auto-HTTPS via Let's Encrypt out of the box:

# Caddyfile
api.welzin.ai {
  reverse_proxy localhost:8000
}

dash.welzin.ai {
  reverse_proxy localhost:3000
}

Run caddy run and you have HTTPS on both subdomains. Done.

Nginx is what you'll see in customer environments. More config, more flexible. Traefik is K8s-native. Same concept, different ergonomics.

Monitoring - the four numbers that matter#

You don't need a hundred dashboards. You need the Four Golden Signals for every service:

1. Latency - how long a request takes (p50, p95, p99).
2. Traffic - requests per second.
3. Errors - error rate, broken down by status code.
4. Saturation - how full the system is (CPU, memory, queue depth).

Stack:

• Prometheus scrapes metrics from your app's /metrics endpoint.
• Grafana visualises them.
• Loki does the same for logs (push, query like PromQL).
• Alertmanager routes alerts to Slack/PagerDuty.

Wire a few alerts you'd want at 3 AM:

• p99 latency > 2× normal for 5 min.
• Error rate > 1% for 5 min.
• Disk > 85% on any volume.
• No requests for 10 min on a service that should always have traffic.

SLOs in one paragraph: Pick a target ("99.9% of API requests under 500ms over 30 days"). The gap between target and 100% is your error budget. If you burn through it, you stop shipping features and fix reliability. That's the whole game.

Backups#

The only backup that exists is the one you've restored. Setup steps:

1. Automate - daily snapshot of databases (logical, pg_dump or managed snapshot) and stateful volumes.
2. Encrypt - restic is good. Encrypted, deduplicated, S3-compatible.
3. Off-site - different region, different account/credentials.
4. Restore drill - once a month, restore the latest snapshot to a scratch machine. Time it. Verify.

If you've never restored, you have no backups.

Secrets#

Plaintext secrets in a Google Doc / Slack message / .env checked into Git is how breaches happen. (See the warning in the homelab runbook below - the original doc had this problem.)

Hierarchy of right:

1. Cloud secrets manager (AWS Secrets Manager, GCP Secret Manager) for production.
2. A KMS-encrypted file for infra-as-code (SOPS + age).
3. A team password manager (1Password, Bitwarden) for human-held credentials.
4. Env vars at runtime for the app to read. Never bake into images.
5. Rotation policy - every 90 days minimum, immediately on team-member departure.

VPNs and remote access#

Don't expose admin services (SSH, dashboards, databases) to the public internet. Put them behind a VPN.

• Tailscale - easiest. Zero-config mesh VPN, free for small teams. This is what Welzin uses.
• WireGuard - what Tailscale is built on. Use directly if you want full control.
• OpenVPN - legacy, painful, sometimes required by customer policy.

After Tailscale setup, SSH to a machine via its private name (ssh aman@cygwin) instead of an IP. No public port 22, no Fail2Ban arms race.

Welzin Homelab Runbook#

This is the operational document for the Welzin home server (codenamed CYGWIN). Read it as a working example of everything above. Everywhere you see a password or IP in this document, in real life that goes in a secrets manager - not in source.

Security note (2026): The original version of this doc had plaintext passwords for AWS RDS, WordPress admin, and Cockpit. Those have been rotated and removed. Replace any <SECRET> placeholder by reading from the team password manager.

Hardware#

Virtualisation layer#

• Hypervisor: Harvester (HCI based on KVM)
• Guest OS: Ubuntu Server LTS in VMs
• Container runtime: Docker Engine + Docker Compose
• Orchestration: k3s (lightweight Kubernetes) for the multi-service workloads

# k3s install (single-node)
curl -sfL https://get.k3s.io | sh -
sudo k3s kubectl get nodes

Use kubectl, k9s, or Lens for management.

Mail (internal only)#

Postfix as MTA + Dovecot for IMAP. Roundcube webmail. Rspamd for spam filtering. SMTP relay through SendGrid to avoid IP-reputation issues for outbound. Not exposed to the public internet without DKIM/SPF/DMARC properly configured and an opted-in sending domain.

Networking & security#

• Static IPs via Netplan on each VM.
• SSH: key-based only, port 22 closed at the perimeter; access via Tailscale mesh.
• Firewall: ufw on each VM + pfSense at the perimeter.
• TLS: Caddy (auto Let's Encrypt) for any HTTP-fronted service.
• DNS: pfSense handles internal DNS; Cloudflare for public records.
• DDoS: Cloudflare proxy for any externally exposed service.

Tailscale setup (the path we use)#

# On every VM that should join the mesh:
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up                # opens browser to authenticate
tailscale ip -4                  # confirm the 100.x.y.z address

# From any other node on the tailnet:
ssh cygwin@100.94.85.72          # or by MagicDNS name

VPN & privacy stack (optional)#

• WireGuard for any non-Tailscale clients.
• Pi-hole as a DNS-level ad/tracker blocker for the LAN.

Web hosting workloads#

• Reverse proxy: Caddy (preferred) or Nginx where Caddy can't be installed.
• App runtimes: Node.js, Go, Python (under systemd or in Docker).
• CMS: Ghost for blog, WordPress only where a customer requires it.
• Cockpit on each VM for web-based admin (port 9090, behind Tailscale only).

Cockpit install on Ubuntu#

sudo apt update && sudo apt upgrade -y
sudo apt install -y cockpit
sudo systemctl enable --now cockpit.socket
sudo ufw allow 9090/tcp        # only meaningful if 9090 is reachable
sudo ufw reload

Then browse https://<server>:9090 and log in with a Linux user. Never expose 9090 to the public internet. Tailscale-only.

File sharing / NAS#

• Samba for Windows-friendly LAN shares.
• Nextcloud (in Docker) for browser-based file access and sync.
• TrueNAS Core or OpenMediaVault as the underlying NAS OS, when we add dedicated storage hardware.

Databases on the homelab#

• PostgreSQL (primary)
• MongoDB (only where a service was built on it)
• Qdrant (for vector workloads)
• Redis (cache, sessions)

All run as Docker containers with named volumes for state, with restic taking nightly encrypted snapshots to S3.

AI / ML stack#

• Ollama for local LLMs (Llama 3.x family).
• LlamaIndex / LangChain when a project genuinely benefits (avoid by default).
• Whisper for transcription.
• Coqui TTS for text-to-speech.
• MLflow for any classical ML experiment tracking (owner: Vikram).
• Kubeflow evaluated, not in default stack.

# Ollama quick-start
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama run llama3.2

Monitoring stack#

• Prometheus (scraping /metrics on every service).
• Node Exporter on every VM.
• Grafana dashboards: Four Golden Signals per service + Node board for hardware.
• Loki + Promtail for log aggregation.
• Alertmanager → Slack #alerts.

Backup & self-repair#

• Hourly snapshots of databases via cron.
• restic nightly to S3 (encrypted, deduplicated).
• Systemd units with Restart=on-failure for app services.
• Health-check + auto-restart scripts behind a watchdog timer.

Reference: standard Ubuntu setup commands#

# Base
sudo apt-get update && sudo apt-get upgrade -y
hostname -I

# SSH
sudo apt install -y openssh-server
sudo systemctl enable --now ssh

# UFW
sudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw enable
sudo ufw status verbose

# Web
sudo apt install -y caddy           # preferred
# or: sudo apt install -y nginx

# Postgres (if not in Docker)
sudo apt install -y postgresql
sudo -u postgres psql

# Ollama
curl -fsSL https://ollama.com/install.sh | sh

Cloud counterpart (AWS, e.g. credzin.com box)#

For a public-facing PHP/WordPress site backed by RDS:

1. Security group for EC2 - inbound: 80, 443, 22 (from your IP only).
2. Launch EC2 (t3.small or larger; t2.micro for dev only) - Amazon Linux 2023 or Ubuntu LTS.
3. RDS (MySQL) - same VPC/subnet as EC2; public access No; SG allows 3306 only from EC2's SG.
4. Install LAMP on the EC2 box (Apache + PHP + MySQL client).
5. Deploy WordPress to /var/www/html, set permissions to www-data:www-data (Ubuntu) or apache:apache (AL2023).
6. Configure wp-config.php with the RDS endpoint and secrets pulled from AWS Secrets Manager at deploy time, not hardcoded.
7. Cockpit on port 9090, accessible only over Tailscale.

# Updating WordPress domain after a move
# 1) wp-config.php
define('WP_HOME','https://newdomain.com');
define('WP_SITEURL','https://newdomain.com');

# 2) DB rewrites (run in phpMyAdmin or psql equivalent)
UPDATE wp_options
   SET option_value = REPLACE(option_value, 'oldurl.com', 'newurl.com')
 WHERE option_name IN ('home', 'siteurl');

UPDATE wp_posts    SET guid         = REPLACE(guid,         'oldurl.com', 'newurl.com');
UPDATE wp_posts    SET post_content = REPLACE(post_content, 'oldurl.com', 'newurl.com');
UPDATE wp_postmeta SET meta_value   = REPLACE(meta_value,   'oldurl.com', 'newurl.com');

Hands-on Checkpoints#

• Write a Dockerfile for a Hello-World Node or Python app. Build it. Run it. curl it.
• Write a docker-compose.yml with the app + a Postgres + a Redis. Bring it up.
• Put Caddy in front of the app and get HTTPS via Let's Encrypt on a real domain (use a *.welzin-bootcamp.dev subdomain if available).
• Set up Prometheus + Grafana scraping /metrics from the app. Build a 4-panel dashboard.
• Configure a Slack alert when the app's error rate goes above 0.
• Join the Tailscale tailnet and SSH to a homelab VM.
• Take a restic backup of /var/lib/postgresql to S3, delete the data, restore it.

Further reading#

• Site Reliability Engineering - Google (free) sre.google/books
• Docker's official tutorial
• Caddy docs - short, complete
• Tailscale Learn - for VPN mental model
• How Linux Works - Brian Ward

Welzin opinion: Boring infra is good infra. Caddy + Docker Compose + Postgres + Tailscale will serve you for years. K8s is a tax - only pay it when you have the team size and ops practice to make it net-positive.