MLOps#

Sub-chapter 3 of DevOps · DevOps for models - the ops discipline around ML

A model in a notebook is a liability, not an asset. It runs once, on your machine, against a CSV you'll never find again, and the moment you close the laptop the magic is gone. MLOps is the boring, unglamorous work of turning that one-off result into something that runs in production, stays correct as the world changes, and can be rolled back when it doesn't. It is DevOps for models - same loop (build → test → release → deploy → observe), but with one extra, hostile dependency that classic DevOps never has to think about: data.

The AI/ML chapter teaches you to build models. This chapter is about everything that happens after the model is good enough - and a blunt truth for Welzin: most of our products don't train anything. They call an LLM API. For those, "MLOps" is not GPUs and training clusters. It's prompt versioning, evals, and cost monitoring. We'll cover both, but be honest about which one you'll actually do.

Outline#

1. Why notebooks fail in prod - non-determinism, hidden state, "works on my data"
2. Experiment tracking - MLflow: params, metrics, artifacts, reproducibility
3. Data & model versioning - DVC for data, a model registry for weights
4. The lifecycle - train → eval → package → deploy → monitor, as a loop
5. Serving patterns - batch vs online, REST endpoints, model-as-a-container
6. CI/CD for models - eval gates before promote, not just unit tests
7. Monitoring in prod - data drift, concept drift, performance decay
8. The feedback loop - capturing ground truth and retraining
9. Reproducibility & rollback - pin everything, promote/demote a version
10. The LLM special case - prompts and evals are the artifact; cost & latency

101 Primer#

Why "it works in the notebook" fails in production#

A notebook is a lie of convenience. Cells run out of order, variables linger from three experiments ago, the random seed is unset, and the training data is a file path on your disk. Reproduce that on a server and you get a different model - or no model. The job of MLOps is to kill every source of non-determinism:

• Pin the code - a git SHA, not "the version I had open".
• Pin the data - a hash of the exact dataset, not latest.csv.
• Pin the environment - a locked requirements.txt / container, not your global Python.
• Pin the randomness - set seeds; log them.

If you can't say "this model came from commit X, data Y, env Z", you don't have a model. You have a coincidence.

Experiment tracking with MLflow#

The first habit: log every run. Params in, metrics out, the model artifact attached. MLflow is the Welzin default (it's already in the homelab stack, owner: Vikram).

import mlflow
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score

mlflow.set_tracking_uri("http://mlflow.welzin.internal:5000")
mlflow.set_experiment("churn-prediction")

with mlflow.start_run(run_name="rf-baseline"):
    params = {"n_estimators": 300, "max_depth": 12, "random_state": 42}
    mlflow.log_params(params)

    model = RandomForestClassifier(**params).fit(X_train, y_train)
    f1 = f1_score(y_val, model.predict(X_val))

    mlflow.log_metric("val_f1", f1)
    mlflow.log_artifact("data/feature_list.json")   # what features it expects
    mlflow.sklearn.log_model(model, "model")

Now every experiment is a row in the MLflow UI: who ran it, with what params, what it scored, and the exact artifact. You stop emailing each other model_final_v3_REAL.pkl.

Data and model versioning#

Git is for code. Git is terrible for a 4 GB parquet file. Use DVC: it stores a tiny pointer file in git and pushes the real bytes to S3.

dvc init
dvc remote add -d s3store s3://welzin-ml/dvc
dvc add data/train.parquet          # creates data/train.parquet.dvc (the pointer)
git add data/train.parquet.dvc data/.gitignore
git commit -m "data: churn training set v2"
dvc push                            # bytes go to S3, pointer stays in git

Anyone who clones the repo runs dvc pull and gets the exact dataset that commit was trained on. That's reproducibility for data.

For trained weights, promote the best run into a model registry with named stages:

result = mlflow.register_model(
    model_uri="runs:/<run_id>/model",
    name="churn-classifier",
)
client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
    name="churn-classifier", version=result.version, stage="Staging",
)
# ...after eval gates pass...
client.transition_model_version_stage(
    name="churn-classifier", version=result.version, stage="Production",
)

Serving code loads models:/churn-classifier/Production - it never hardcodes a version. Rollback is one API call: point Production back at the previous version.

Serving patterns: batch vs online#

Two ways to get predictions out of a model:

Batch is cheaper, simpler, and the right default. Don't stand up a 24/7 GPU endpoint to serve a number that only changes once a day.

When you do need online, the model is just a container - exactly the Docker discipline from sub-chapter 2. Wrap it in FastAPI, bake it into an image, run it like any other service:

# serve.py
from fastapi import FastAPI
from pydantic import BaseModel
import mlflow.pyfunc

app = FastAPI()
model = mlflow.pyfunc.load_model("models:/churn-classifier/Production")

class Features(BaseModel):
    tenure_months: int
    monthly_charges: float
    support_tickets: int

@app.post("/predict")
def predict(f: Features):
    pred = model.predict([f.model_dump()])
    return {"churn_probability": float(pred[0])}

@app.get("/healthz")
def health():
    return {"ok": True}

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY serve.py .
EXPOSE 8000
CMD ["uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"]

Same docker build, same Caddy reverse proxy, same Prometheus /metrics. The model is now a normal citizen of your infra.

CI/CD for models: the eval gate#

Classic CI asks "does the code run?" Model CI also asks "is the model good enough to ship?" You add an eval gate - a threshold the model must clear against a held-out test set before it can be promoted.

# .github/workflows/model-ci.yml
name: model-ci
on: { push: { paths: ["models/**", "training/**"] } }
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt && dvc pull
      - name: Run evaluation
        run: python eval.py --min-f1 0.78 --max-latency-ms 50
      # eval.py exits non-zero if metrics fall below threshold → no promote

# eval.py (the gate)
import sys, json, mlflow
metrics = run_evaluation(model, test_set)   # your eval logic
print(json.dumps(metrics))
if metrics["f1"] < args.min_f1:
    sys.exit(f"BLOCKED: f1 {metrics['f1']:.3f} < {args.min_f1}")

The point: a model that's worse than what's live should never reach Production, even if the code is perfect. Tests pass; the gate still fails it. Promote on merit, not on green checkmarks.

Monitoring in production#

A deployed model rots silently. It doesn't throw a 500 - it just gets quietly, increasingly wrong while latency and uptime look perfect. You monitor three distinct failures:

• Data drift - the inputs change. Your fraud model was trained on pre-2024 spend patterns; now everyone uses a new payment method. The feature distribution moved. The model's assumptions are stale.
• Concept drift - the relationship between inputs and the target changes. Same inputs, different correct answer. "What makes a customer churn" shifted because a competitor launched. This is more dangerous and harder to spot.
• Performance decay - the metric you actually care about (accuracy, F1) drops once ground truth arrives.

Evidently makes drift detection a few lines:

from evidently import Report
from evidently.presets import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
result = report.run(reference_data=train_ref, current_data=last_24h)
result.save_html("drift.html")   # ship a summary metric to Prometheus too

Wire a drift score and a rolling-accuracy metric into Grafana next to the Four Golden Signals. Alert when drift crosses a threshold - that's your signal to investigate or retrain.

The feedback loop#

Monitoring tells you the model decayed. The feedback loop fixes it: capture the model's predictions and the eventual ground truth (did the customer actually churn?), feed that back as fresh training data, retrain, eval-gate, promote. A model in production is never "done" - it's the start of the loop, not the end.

The LLM special case - and what Welzin actually does#

For products that call an LLM API (most of ours), there's no training, no GPUs, no weights to version. The artifact you ship is the prompt. So MLOps collapses to three things:

1. Prompt versioning - prompts live in git, reviewed in PRs, with a version tag. A prompt change is a deploy. Treat system_prompt_v7 like you'd treat a model version.
2. Evals as the test suite - you can't unit-test a generation, so you build an eval set: inputs + expected behavior, scored by assertions, regex, or an LLM-as-judge. This is your eval gate. A prompt change that drops the eval score doesn't ship.
3. Cost & latency monitoring - tokens are money and p99 latency is UX. Log tokens-in/out, cost-per-request, and latency per call, and alert on regressions. A prompt tweak that doubles output tokens is a 2× bill nobody approved.

# minimal LLM eval gate
cases = load_jsonl("evals/extraction.jsonl")   # {input, must_contain, must_not}
passed = 0
for c in cases:
    out = call_llm(PROMPT_V7, c["input"])
    ok = all(s in out for s in c["must_contain"]) and \
         all(s not in out for s in c["must_not"])
    passed += ok
score = passed / len(cases)
assert score >= 0.95, f"eval regressed: {score:.2%}"   # blocks promote

How this differs from classic DevOps: data is a dependency you don't control. Code is deterministic - same input, same output, forever. A model (or an LLM) depends on data and a world that keep moving, so "it passed CI" is necessary but not sufficient. You ship, then you watch, because correctness can decay without a single line of code changing.

Hands-on Checkpoints#

• Train any small model and log it to MLflow - params, one metric, the model artifact. Open the UI and compare two runs.
• Put a dataset under DVC, push to S3, delete it locally, dvc pull it back, confirm the hash matches.
• Register your best run to the model registry, transition it Staging → Production, then roll back to the previous version.
• Wrap the model in a FastAPI /predict endpoint, containerise it, and curl it through a Caddy reverse proxy.
• Write an eval.py gate and a CI workflow that blocks promotion when F1 falls below a threshold.
• Run Evidently against a "reference" set and a deliberately drifted "current" set; export a drift score and alert on it in Grafana.
• For an LLM feature: version a prompt in git, build a 20-case eval set, and add a CI step that fails on regression plus logs tokens and cost per call.

Further reading#

• MLflow docs - tracking, registry, serving
• Made With ML - Goku Mohandas, the best end-to-end MLOps course
• Google's MLOps whitepaper - the maturity-level mental model
• Evidently docs - drift and performance monitoring
• DVC docs - data and pipeline versioning
• Hidden Technical Debt in ML Systems - Google; why ML systems rot

Welzin opinion: For most Welzin products, MLOps is not a training pipeline - it's prompt versioning, an eval set, and a cost dashboard. Build those three before you ever utter the word "GPU." And whatever you serve, batch beats online until proven otherwise: don't run a 24/7 endpoint to deliver a number that changes once a day.