AIOps#

Sub-chapter 4 of DevOps · ML for ops, not ops for ML

Don't confuse this with the previous sub-chapter. MLOps is ops for ML models - how you train, version, deploy, and monitor a model in production. AIOps is ML/AI for ops - pointing statistics and language models at your own infrastructure to find anomalies, cut alert noise, suggest root causes, and help a human resolve an incident faster. One keeps your models healthy; the other uses models to keep everything else healthy.

AIOps is also the single most over-marketed term in this whole chapter. Every observability vendor has bolted "AI-powered" onto a dashboard and tripled the price. Most of what they sell you is good defaults plus a regex. The honest version is small and useful: a few statistical baselines instead of static thresholds, deduplicated alerts so you only get paged once, and an LLM that reads your logs and writes the first draft of the postmortem. For a small team, AIOps is mostly good alerting plus an LLM on your logs - not a platform purchase.

Outline#

1. The three pillars - metrics, logs, and traces are the data AIOps consumes
2. What AIOps actually is - the Gartner framing vs. the marketing reality
3. Anomaly detection - statistical baselines, seasonality, and when a static threshold wins
4. Alert fatigue - noise reduction, deduplication, and event correlation
5. Root-cause assistance - topology, dependency graphs, and "what changed"
6. Automated remediation - runbooks → self-healing, and the human-in-the-loop gate
7. LLMs for ops - incident summaries, natural-language log queries, postmortem drafts
8. Failure modes - hallucination, confident-but-wrong RCA, and over-automation
9. The small-team stack - what you actually build at Welzin
10. Tie-back - how this sits on top of the Four Golden Signals from the primer

101 Primer#

The three pillars are the input, not the product#

AIOps doesn't invent data - it eats the telemetry you already produce. From the DevOps primer you have metrics (Prometheus, the Four Golden Signals), and the runbook added logs (Loki) and, if you instrument it, traces (OpenTelemetry spans showing one request hopping across services).

No telemetry, no AIOps. If your /metrics endpoint is empty, no amount of AI fixes that. Get the primer's stack working first.

What AIOps actually is#

Gartner coined the term: platforms that combine big data and machine learning to automate and enhance IT operations - ingesting metrics/logs/traces/events, then doing anomaly detection, event correlation, and causality analysis to support faster detection and response.

Translated to honest engineering, AIOps is four jobs:

1. Detect - notice something is off without a human pre-defining every threshold.
2. Correlate - group the 200 alerts from one outage into one incident.
3. Diagnose - suggest what changed and where the blast originated.
4. Act - trigger a runbook, with or (carefully) without a human.

That's it. Everything else is a dashboard.

Anomaly detection - and when a threshold beats ML#

The pitch: "ML learns normal and alerts on abnormal." Sometimes true. Often a static threshold is better, cheaper, and easier to debug.

A static threshold is great when you know the limit: disk > 85%, error rate > 1%, p99 > 2× normal. Use them. They're in the primer for a reason. ML earns its keep when normal is a moving target with seasonality - traffic that's high every weekday at 10am and near-zero at 3am. A flat threshold either screams every night or misses a real daytime dip.

A pragmatic baseline (no platform required):

# Seasonal anomaly check: compare today's value to the same
# slot over the last N weeks. Flag if it's a robust-z outlier.
import numpy as np

def is_anomalous(history_same_slot, current, z=3.5):
    # history_same_slot = e.g. requests at Tue 10:00 for last 8 weeks
    med = np.median(history_same_slot)
    mad = np.median(np.abs(history_same_slot - med)) or 1e-9
    robust_z = 0.6745 * (current - med) / mad   # MAD-based z-score
    return abs(robust_z) > z, robust_z

This catches "Tuesday 10am is 4× quieter than every other Tuesday" - a real outage - without paging you for the normal nightly lull. MAD (median absolute deviation) is used instead of standard deviation because it doesn't get wrecked by the one huge spike from last week's incident. For most teams, a handful of seasonal baselines like this is your "AI." Reach for Prophet, ARIMA, or a vendor model only when the simple version provably misses things.

Alert fatigue is the real enemy#

The actual problem at 3am isn't too few alerts - it's too many. One database stall fans out into 200 pages from every downstream service. Humans start ignoring the channel. AI cannot fix a team that has already learned to mute #alerts.

Three cheap techniques, none of which need a neural network:

• Deduplication - collapse identical alerts within a window into one.
• Grouping - Alertmanager already does this; group by alertname, cluster, service.
• Correlation - when DB latency spikes and 12 services error within 60s, that's one incident, not 13. Correlate on time + topology.

# Alertmanager: stop the fan-out before it reaches a human
route:
  group_by: ['alertname', 'service']
  group_wait: 30s        # let related alerts arrive before firing
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      group_wait: 0s

This is "AIOps" with zero ML and it removes 80% of the noise. Do it first.

Root-cause assistance: "what changed?"#

When an incident fires, the single most useful question is "what changed in the last 30 minutes?" - a deploy, a config push, a feature flag, a traffic shift. AIOps tools build a topology / dependency graph (often from trace data) so they can say "the errors originate at payments-db, and everything red downstream is a symptom, not a cause."

You can approximate this without a product: correlate your alert timestamps against your deploy log.

# Did anything ship right before the incident started?
git log --since="40 minutes ago" --pretty='%h %an %s'
kubectl rollout history deploy/payments | tail -5

90% of incidents trace to a recent change. A tool that just surfaces the deploy diff next to the alert outperforms most "AI root cause" features.

Automated remediation - and why the gate matters#

The dream is self-healing: detect → diagnose → fix, no human. The reality is a spectrum, and you should climb it slowly:

1. Runbook - a human follows written steps.
2. Scripted runbook - one command runs the steps.
3. Auto-trigger with approval - system proposes the fix, a human clicks yes.
4. Full self-healing - system acts alone (e.g. Restart=on-failure, HPA autoscaling).

Full automation is only safe for narrow, idempotent, well-understood actions: restart a crashed process, scale out under load, fail over to a replica. For anything that mutates data, deletes, or reroutes traffic - keep a human-in-the-loop gate. An auto-remediation that's wrong doesn't just fail to fix the incident; it can amplify it (restart-loop a thundering herd, drain the wrong node, mask a data-corruption bug by hiding the symptom). The gate exists because a confident automated action at 3am, based on a misdiagnosis, turns a one-service blip into a company-wide outage.

LLMs for ops - the genuinely new part#

This is where 2024-onwards AIOps got actually interesting, and where Welzin gets real leverage. An LLM is good at the language parts of an incident:

Summarize an incident from raw logs:

# On-call copilot: turn 5,000 log lines into a human briefing.
from anthropic import Anthropic

logs = open("incident.log").read()[-50_000:]   # last chunk only
msg = Anthropic().messages.create(
    model="claude-opus-4-8",
    max_tokens=600,
    messages=[{"role": "user", "content":
        "You are an SRE copilot. From these logs, give: (1) a one-line "
        "summary, (2) the likely failing component, (3) the first error "
        "timestamp, (4) 3 things to check. Do NOT invent log lines; quote "
        "exact text for any claim. Say 'insufficient evidence' if unsure.\n\n"
        + logs}],
)
print(msg.content[0].text)

Other high-value uses: natural-language log queries ("show me 500s on checkout in the last hour" → LogQL/PromQL), drafting postmortems (timeline + impact from the incident channel - a draft a human edits), and an on-call copilot in Slack that proposes next steps.

The failure modes are real and you must design around them:

• Hallucination - an LLM will happily invent a plausible-sounding root cause and a log line that was never written. Force it to quote exact evidence and allow "insufficient evidence."
• Confident-but-wrong RCA - fluent prose reads as authoritative. Treat its conclusion as a hypothesis to verify, never as the verdict.
• Stale context - it only knows what you fed it; the real cause may be outside the window.

So: LLMs draft and assist; they do not decide and act. A copilot that proposes a kubectl command is great. A copilot that runs it unsupervised is how you delete prod.

The small-team Welzin stack#

You don't buy an AIOps platform. You assemble it:

• Detection - Prometheus + Alertmanager with a few MAD/seasonal baselines for the metrics that are spiky by nature; static thresholds for everything else.
• Noise reduction - Alertmanager grouping + dedup. This alone is most of the win.
• Correlation - deploy log and trace topology next to the alert. "What changed" beats "what is broken."
• LLM layer - an #oncall Slack bot (Claude) that summarizes the firing alerts, tails Loki, and drafts the postmortem. Human approves every action.
• Remediation - Restart=on-failure and autoscaling for the safe stuff; everything else stays a human-gated runbook.

This sits directly on top of the primer's monitoring stack. The Four Golden Signals are still the source of truth; AIOps just makes them louder when they should be and quieter when they shouldn't.

Hands-on Checkpoints#

• Take one spiky metric (e.g. requests/sec) and write the MAD seasonal baseline above; replay last week's data and confirm it catches a real dip without firing on the nightly lull.
• Configure Alertmanager group_by + group_wait so a simulated multi-service outage produces one notification, not twenty.
• Write a script that, on any alert, prints the last 5 deploys and the diff that shipped just before it.
• Build a Slack /incident command that feeds the last N Loki lines to an LLM and posts a 4-bullet summary - with a rule that it must quote exact log text.
• Have the LLM draft a postmortem from a real (or staged) incident timeline; edit it and note every place it got the facts wrong.
• Add Restart=on-failure to one service and prove auto-restart works; then write the human-gated runbook for an action you would never automate (e.g. failover) and explain why.
• Compare: pick an alert and try a static threshold vs. a learned baseline. Decide which you'd actually ship, and justify it.

Further reading#

• Site Reliability Engineering - Google (free), esp. the chapters on monitoring and effective alerting sre.google/books
• Gartner: AIOps definition & market guidance - the canonical framing
• OpenTelemetry docs - metrics, logs, and traces as standard telemetry
• Prometheus alerting best practices - how to alert on symptoms, not causes
• Facebook Prophet - when you genuinely need seasonal time-series forecasting
• Google: Building Secure & Reliable Systems (free) - on safe automation and human-in-the-loop

Welzin opinion: AI can't fix an org that ignores its own alerts. Fix the noise and the alerting culture first; then add a model. An LLM on top of a channel everyone has muted just hallucinates into the void.