Kubernetes#

Sub-chapter 2 of DevOps · You'll deploy on it at a customer long before you ever need to run your own

The Docker sub-chapter got you a container. Kubernetes (K8s) is the layer that runs thousands of them across a fleet of machines, keeps them alive when nodes die, rolls out new versions without downtime, and load-balances traffic between them. You will almost never write Kubernetes (controllers, operators) as a forward-deployed engineer - but you will deploy on it constantly, because half our enterprise customers have already standardised on EKS, GKE, or AKS and they expect you to speak it.

The single mental model that unlocks everything: Kubernetes is a desired-state reconciler, not a deploy script. You don't tell it "start this container." You hand it a YAML file that says "I want 3 replicas of this image, reachable on port 80," and a control loop works forever to make reality match that wish. A node dies, a pod is reaped, you kubectl apply a new image - the reconciler notices the drift and fixes it. Once you internalise that, kubectl stops feeling like magic. At Welzin we treat the YAML as the source of truth and the cluster as a cache of it.

Outline#

1. Desired-state reconciliation - you declare what you want; the control loop continuously makes reality match.
2. Pod - the smallest deployable unit; one or more containers sharing a network/IP. You rarely create them directly.
3. ReplicaSet - keeps N identical pods running. You almost never touch it directly either.
4. Deployment - the object you actually use; manages ReplicaSets to give you rolling updates and rollbacks.
5. Service - a stable virtual IP + DNS name that load-balances across a set of pods (pods are cattle, Services are the address).
6. Ingress - HTTP(S) routing from outside the cluster to Services, by hostname/path; the L7 front door.
7. ConfigMap & Secret - inject non-secret config and (base64'd) secrets into pods as env vars or files.
8. Namespace & context - logical partitions inside a cluster; kubectl context picks which cluster + namespace you're aiming at.
9. Resources & probes - requests/limits size the pod; liveness/readiness probes decide if it's healthy and ready for traffic.
10. When NOT to use it - a 5-person team on Vercel almost certainly shouldn't run a cluster. K8s is overhead you adopt deliberately.

101 Primer#

The reconciliation loop, concretely#

kubectl apply -f deploy.yaml   # "here is my desired state"
# control plane stores it in etcd → controllers reconcile → kubelet runs pods
kubectl delete pod my-app-7f8-abc   # delete a pod...
kubectl get pods                    # ...and a new one is already coming up

You deleted a pod and Kubernetes immediately replaced it. You didn't ask it to. That's the whole game: the Deployment said "3 replicas," reality dropped to 2, the reconciler restored 3. Stop thinking in imperative run/stop verbs.

The object hierarchy you'll meet#

Deployment → manages a ReplicaSet → manages Pods. A Service selects pods by label and gives them one stable address. An Ingress routes outside HTTP traffic to Services. You write Deployments, Services, Ingresses, ConfigMaps, and Secrets. You almost never write Pods or ReplicaSets by hand - let the Deployment own them.

A real Deployment + Service + Ingress#

This is roughly what we ship for a stateless web app. Save as app.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/web:1.4.2  # pin a tag, NEVER :latest
          ports:
            - containerPort: 8080
          env:
            - name: LOG_LEVEL
              valueFrom:
                configMapKeyRef: { name: web-config, key: log_level }
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef: { name: web-secrets, key: db_password }
          resources:
            requests: { cpu: "100m", memory: "128Mi" }   # what the scheduler reserves
            limits:   { cpu: "500m", memory: "256Mi" }   # the hard ceiling (OOMKill past memory)
          readinessProbe:                                  # "should I get traffic yet?"
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:                                   # "am I wedged? restart me"
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 15
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: prod
spec:
  selector: { app: web }      # picks every pod labelled app=web
  ports:
    - port: 80                 # the Service's port
      targetPort: 8080         # the container's port
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: prod
  annotations:
    kubernetes.io/ingress.class: alb   # EKS ALB ingress; nginx on GKE/self-managed
spec:
  rules:
    - host: app.welzin.ai
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 80 } }

kubectl apply -f app.yaml
kubectl get deploy,svc,ingress -n prod

kubectl - the five commands you'll run all day#

kubectl get pods -n prod                 # list pods (add -o wide for node/IP)
kubectl describe pod web-7f8-abc -n prod # events + why it won't start (READ THE EVENTS)
kubectl logs web-7f8-abc -n prod         # stdout/stderr; add -f to follow, --previous for last crash
kubectl exec -it web-7f8-abc -n prod -- sh   # shell into a running container
kubectl apply -f app.yaml                # converge cluster to the YAML

Add kubectl get events -n prod --sort-by=.lastTimestamp when something is weird - events are where Kubernetes tells you the truth.

Rolling updates and the undo button#

Bump the image tag in app.yaml and apply. The Deployment spins up new pods, waits for their readiness probes, then tears down old ones - zero downtime if probes are honest.

kubectl set image deployment/web web=...ecr.../web:1.4.3 -n prod  # or just edit YAML + apply
kubectl rollout status deployment/web -n prod    # watch it converge
kubectl rollout history deployment/web -n prod   # see revisions
kubectl rollout undo deployment/web -n prod      # PANIC BUTTON: revert to the previous ReplicaSet

rollout undo is the most important command on this page. Bad deploy at 2am? One command, traffic is back on the last-good revision in seconds. This works because the old ReplicaSet is still recorded - it's the reconciler's memory, not a re-deploy.

Requests, limits, and why they matter#

requests is what the scheduler reserves to place your pod - set it too low and you'll get co-tenants starving each other; too high and you waste money on an oversized cluster. limits is the hard ceiling: exceed the memory limit and the kernel OOMKills your container (you'll see OOMKilled in describe); exceed the CPU limit and you're throttled, not killed. At Welzin: always set both. A pod with no limits is a noisy neighbour waiting to take down a node.

Liveness vs readiness - the distinction people get wrong#

• Readiness probe failing → pod is removed from the Service's load-balancer pool but kept running. Use it for "still warming up / lost its DB connection / shedding load."
• Liveness probe failing → pod is killed and restarted. Use it only for "wedged beyond recovery."

Get them backwards and you'll restart-loop a perfectly healthy app that's merely slow to boot, or serve 500s from a pod that should've been pulled out of rotation. Rule of thumb: readiness gates traffic, liveness gates the kill switch.

Config and secrets#

kubectl create configmap web-config --from-literal=log_level=info -n prod
kubectl create secret generic web-secrets --from-literal=db_password='...' -n prod

Reality check: a Kubernetes Secret is only base64-encoded, not encrypted - anyone with read access to the namespace can decode it. For real secrets at customers we use External Secrets Operator backed by AWS Secrets Manager (ap-south-1), or sealed-secrets, and we enable encryption-at-rest on etcd. Never commit a Secret manifest with a real value to git.

Namespaces and context switching#

kubectl config get-contexts                 # which clusters do I know about?
kubectl config use-context prod-eks-aps1     # switch to the prod EKS cluster
kubectl config set-context --current --namespace=prod   # stop typing -n every time
kubectl get ns                               # list namespaces

Install kubectx/kubens (or kubie) on day one. The most expensive Kubernetes mistake juniors make is running a delete against prod while they thought they were in staging. Make your shell prompt show the current context (use kube-ps1). Context confusion is a production incident waiting to happen.

Debugging a CrashLoopBackOff#

CrashLoopBackOff means: the container starts, exits/crashes, and K8s keeps restarting it with exponential backoff. The status is a symptom, not the cause. Work it like this:

kubectl get pods -n prod                          # spot the CrashLoopBackOff
kubectl describe pod web-7f8-abc -n prod          # check Events + Last State + exit code
kubectl logs web-7f8-abc -n prod --previous       # logs from the crashed instance (KEY FLAG)

The top causes, in order of how often we see them: bad env/config (missing ConfigMap or Secret key → app exits on boot), wrong image or a missing migration, an aggressive liveness probe killing a slow-starting app, or hitting the memory limit and getting OOMKilled. The --previous logs plus the exit code in describe almost always tell you which.

Hands-on Checkpoints#

• Install kind (or minikube), spin up a local cluster, and confirm kubectl get nodes shows it Ready.
• Deploy the app.yaml above (swap the image for nginx or a tiny app of yours) and reach it via kubectl port-forward.
• kubectl scale deployment/web --replicas=5, watch new pods appear, then kubectl delete one pod and watch the reconciler replace it.
• Do a rolling update by changing the image tag and apply; then kubectl rollout undo and confirm you're back on the old revision.
• Deliberately break the image tag to a non-existent one, observe the resulting ImagePullBackOff/CrashLoopBackOff, and diagnose it using describe + logs --previous.
• Create a ConfigMap and a Secret, wire both into the pod as env vars, and kubectl exec in to print them.
• Set a memory limit lower than the app needs, trigger an OOMKilled, and confirm you can see it in kubectl describe pod.

Further reading#

• Kubernetes Concepts - official docs
• kubectl Cheat Sheet - official
• Configure Liveness, Readiness and Startup Probes
• Managing Resources for Containers (requests & limits)
• kind - Kubernetes in Docker (local clusters)
• "Let me tell you a story about K8s" - Kelsey Hightower's takes

Welzin opinion: Don't reach for Kubernetes until you actually need it. For most early-stage products, Vercel/Render/Fly/ECS Fargate ships faster, breaks less, and frees up the brain-cycles a cluster would eat. Learn K8s because customers run it - not because your weekend project does. When a 5-person team adopts Kubernetes "to be ready to scale," they usually trade a scaling problem they don't have for an operations problem they do.