GenAI 101#

Sub-chapter 2 of AI/ML · The mental model of modern LLMs

This is the "what's actually going on" chapter. By the end you should be able to look at any LLM-flavoured feature request and immediately answer: what model class? what prompting pattern? structured output or free text? how will I evaluate it?

We're not training models from scratch. We're operators of pre-trained ones - and that's a discipline with its own rigour.

Outline#

1. The model hierarchy - classical ML, deep learning, foundation models, agents
2. What an LLM actually is - autoregressive transformers in one screen
3. Prompting as software engineering - system role, structure, examples, prefills
4. Structured outputs - JSON mode, tools-as-schemas, when to validate
5. Embeddings - what they are, what to use them for
6. Model selection - Haiku / Sonnet / Opus, when to climb the ladder
7. Cost, latency, streaming - the production levers
8. Hallucinations and how to design around them

101 Primer#

The hierarchy in your head#

            classical ML
                │   (logistic regression, gradient boosting, random forest)
                │   tabular, supervised, fast, auditable
                ▼
          deep learning
                │   (CNNs for vision, transformers for text)
                │   raw inputs, learned features, GPU
                ▼
        foundation models / LLMs
                │   (Claude, GPT, Llama, Gemini)
                │   pre-trained at scale, prompt-as-program
                ▼
              agents
                    LLM + tools + loop, multi-step actions

Most product work at Welzin happens at the LLM and agent layers. Sub-chapter 1 covered the classical layer. This sub-chapter is about the middle two.

What an LLM actually is (without the magic)#

An LLM is a function:

LLM(prompt) → probability distribution over the next token

You sample from that distribution to get the next token, append it to the prompt, and call again. That's autoregressive generation. The "transformer" architecture is how the function is computed - self-attention over the whole context window, learned during pre-training on the open web + curated data + RLHF.

This sounds reductive on purpose. The implications are:

• The model can only attend to what's in its context window. No context = no answer. Stuff the context with the right things and you get the right answer.
• The model is stateless between calls. "Memory" is just including prior turns in the next prompt.
• The model doesn't know what it doesn't know. It will confidently complete plausible-sounding nonsense (a hallucination). Design accordingly.
• Quality scales with parameters, training data, and inference compute (extended thinking) - not linearly, but reliably enough that "go up a tier" is a real lever.

Prompting is software engineering#

A prompt is a function specification written in English. Treat it like code: version it, test it, review it.

The structure that works:

1. System role. Who the model is and the one thing it must never do.
2. Task. The single action being performed.
3. Inputs. Delimited (XML tags or fenced blocks).
4. Output schema. Explicit, ideally JSON.
5. Examples (few-shot). One or two if the format is non-obvious.

A working example (Anthropic-style):

SYSTEM = """You are an extraction agent for customer-support emails.
You output ONLY a JSON object matching this schema:
  { intent: "question" | "complaint" | "request" | "other",
    urgency: "low" | "medium" | "high",
    summary: string ≤ 200 chars }
Never include markdown, prose, or explanation. If a field is uncertain,
choose the most conservative value, never null."""

USER = f"""<email>
{email_body}
</email>"""

Things that move quality:

• XML tags around inputs - Claude attends to structure.
• Explicit output schema in the system prompt - far more reliable than a hint mid-prompt.
• Two well-chosen examples > a paragraph of instruction.
• An explicit <thinking> step before the answer, for non-trivial reasoning. (Or use extended thinking - see Chapter VII.)

Structured outputs#

In production, you almost always want structured outputs. The options, best-first:

1. Tools as schemas. Define your output as a "tool" the model must call. The SDK guarantees the call's arguments match the schema.
2. JSON mode / response_format. Force JSON, validate with Zod / Pydantic.
3. Prefill. Start the assistant's reply with {. Cheap, hacky, works for prototypes.
4. Free text + parse. Last resort. You will spend a week on regexes.

Validation rule: whatever you get back is untrusted. A schema-conforming JSON object can still have semantically wrong values. Layer validation:

class Intent(BaseModel):
    intent: Literal["question", "complaint", "request", "other"]
    urgency: Literal["low", "medium", "high"]
    summary: str = Field(max_length=200)

parsed = Intent.model_validate_json(model_response)  # schema check
# Now your domain rules:
if parsed.urgency == "high" and len(parsed.summary) < 20:
    raise ValueError("high urgency requires a substantive summary")

Embeddings - what they are, when to use them#

An embedding is a fixed-dim vector (typically 768–3072 dims) representing the meaning of a chunk of text such that similar meanings → nearby vectors under cosine distance.

What they're good for:

• Semantic search. Find passages "about" a topic, not just containing the exact word.
• Clustering and topic discovery. UMAP-project them, eyeball.
• Deduplication of near-duplicate content.
• Recommender systems - "users who read this also read…".

What they're not good for:

• Reasoning. They encode similarity, not truth. Two opposite statements can be near each other in vector space.
• Exact lookups. BM25 or a keyword index is still better for exact-token search.
• Stable IDs. Two different embedding models produce incompatible spaces. Don't mix.

Practical defaults:

• For most product work: text-embedding-3-small (1536d, OpenAI) - cheap and good.
• Self-hosted: BGE-large-en-v1.5 or Jina embeddings v3.
• Always store the embedding model version with the vector - you will rotate models and need to know what to re-index.

Embeddings show up properly in Sub-chapter 3 (RAG). This is the primer.

Model selection - Haiku, Sonnet, Opus#

For Claude (and analogously for other providers):

Decision pattern:

1. Prototype on Sonnet. Get the prompt right.
2. Try Haiku. If quality holds, ship Haiku. Cost drops 10×.
3. Lift to Opus only when Sonnet visibly struggles and quality is worth the cost premium.

Cost, latency, streaming#

The three knobs you have:

• Model tier - biggest single lever.
• Prompt caching - covered in Chapter VII. Free latency + cost win for repeating prefixes.
• Streaming - perceived latency. Your UI must stream tokens to the user the moment they arrive.

Three habits:

• Truncate retrieval. More context is not better past a point. Both quality and cost get worse.
• Async-ify everything non-interactive. Background queue, not request-path.
• Set max_tokens. A runaway response is your cost overrun.

Hallucinations and how to design around them#

Hallucinations are not a bug to be eliminated - they're a property to be designed around.

Three patterns that work:

1. Cite, don't claim. If you're answering from sources, show which sources. RAG with citations (Sub-chapter 3).
2. "I don't know" is a feature. Train your prompt to say "I don't have this information in my sources" when it doesn't. Make that the easiest path for the model.
3. Verify before acting. If the model produces a decision (delete a record, send an email), require a tool-use confirmation step or a deterministic verifier in code.

A specific UI pattern: a "confidence" badge next to LLM-generated content, derived from your eval data, so users know whether to trust it at a glance.

Hands-on Checkpoints#

• Write a 3-shot prompt that classifies an email into {question, complaint, request, other}. Use the Anthropic API. Validate output with a Pydantic model.
• Add a <thinking> step. Compare quality on 10 examples by eye.
• Embed 50 short documents with text-embedding-3-small (or BGE). Plot them with UMAP. Eyeball the clusters.
• Run the same task on Haiku and Sonnet. Compare quality + cost.
• Build a small UI that streams the response token-by-token to the user.
• Pick one feature in your prototype where a hallucination would be costly. Add a fallback path the model can take ("I don't have this") and a UI signal for it.

Further reading#

• Anthropic - Prompt engineering guide
• Lilian Weng - Prompt engineering
• Simon Willison's blog - best running commentary
• Sebastian Raschka - Understanding LLMs - depth without the hype
• Hamel Husain - Field notes from operating LLMs - practical and opinionated

Welzin opinion: Prompt-engineering posts on Twitter are not engineering. A prompt with no eval set is wishful thinking. Build evals before you tune.