The LLM Evaluation & Observability Playbook: How to Know Your AI Actually Works

Most teams ship LLM features the way they'd never ship normal software: no tests, no monitoring, and a "looks good to me" from whoever wrote the prompt. Then a model update, a prompt tweak, or a new edge case silently breaks production — and nobody finds out until a customer does.

This playbook fixes that. LLM evaluation is how you measure quality before you ship; LLM observability is how you watch quality after you ship. Together they form the feedback loop that turns AI from a demo into a dependable product. This is the discipline the RAG playbook and the Multi-Agent Orchestration playbook both point at but don't teach — so here it is, end to end.

The core principle: You cannot improve what you cannot measure, and you cannot trust what you cannot see. Every serious LLM application needs an offline eval suite (a test set you score on every change) and online observability (traces and metrics from real traffic). One without the other is half a system.

Why Evals Come Before Scale

The instinct is to perfect the prompt, ship it, and add tests "later." In LLM development that order is backwards, for one reason: LLM outputs are non-deterministic and the inputs are infinite. A change that fixes one case routinely breaks three others you didn't think to check. Without an eval suite, every prompt edit is a coin flip you can't see land.

Evals give you three things you otherwise lack:

1. A regression net — proof that today's change didn't break yesterday's wins.
2. A north star — a single score that tells you if you're getting better or worse.
3. A migration tool — the only safe way to swap models (e.g. to a newer, cheaper one) is to run both against the same eval set and compare.

Rule of thumb: Write your first eval the day you write your first prompt — even if it's ten hand-picked examples in a spreadsheet. Ten real test cases beat zero, and you'll add the next ninety as production surfaces them.

The Two Halves: Evaluation vs. Observability

Evaluation and observability are complementary, not interchangeable. Know which problem each solves:

The loop closes when production traces feed your eval set: real failures captured by observability become new test cases in evaluation. That flywheel is the whole game.

Part 1: Building Your Evaluation System

Step 1.1 — Build a golden dataset

An eval is only as good as its dataset, and the best datasets come from reality, not imagination. Your golden set is a curated collection of inputs paired with either a known-good output or a set of criteria the output must satisfy. Build it from three sources:

• Real production logs — the highest-value examples, especially failures and edge cases.
• Hand-written hard cases — the tricky inputs you know break things.
• Synthetic generation — use a model to expand coverage, then have a human vet it.

Start with 20-50 examples; quality and diversity matter far more than volume. Tools like Argilla help teams label and curate datasets collaboratively, and Cleanlab surfaces mislabeled or low-quality examples so your "golden" set is actually golden.

Step 1.2 — Choose a scoring method

How you score an output depends on what kind of correctness you need. There are three families, and mature systems use all three:

Always reach for the cheapest method that works. Use assertions for anything you can check with code (does it return valid JSON? does it avoid the forbidden phrase?), reserve LLM-as-a-judge for subjective quality, and use human review to calibrate the judge and spot-check high-stakes flows.

Step 1.3 — Write an assertion eval

Code-based checks are your first line of defense and should cover everything mechanical:

text
For each test case, assert ALL of the following:
- Output parses as valid JSON matching the expected schema.
- The "summary" field is between 20 and 80 words.
- The output contains NONE of the forbidden phrases: [list].
- If the input is out-of-scope, the output is exactly the refusal string.

Any failed assertion fails the test case. Report pass rate per assertion type.

Promptfoo is purpose-built for exactly this — a config-driven test runner that evaluates prompts across models with assertions and side-by-side diffs. DeepEval brings the same idea to a pytest-style developer workflow.

Part 2: LLM-as-a-Judge, Done Right

For subjective quality, LLM-as-a-judge uses a strong model to grade outputs against an explicit rubric. It's the only scalable way to measure things like helpfulness, faithfulness, and tone — but it's also where most teams get evals wrong. Done carelessly, the judge is biased, inconsistent, and falsely reassuring.

The judge prompt

text
You are a strict evaluator. Score the RESPONSE against the rubric below.

QUESTION: {input}
RESPONSE: {output}
REFERENCE (if available): {expected}

Rubric — score each 1-5 and justify in one sentence:
1. Faithfulness: Is every claim supported by the reference / not fabricated?
2. Relevance: Does it actually answer the question asked?
3. Completeness: Are key points missing?

Then give a PASS/FAIL: PASS only if Faithfulness >= 4 AND Relevance >= 4.
Output JSON: { "faithfulness": n, "relevance": n, "completeness": n,
"verdict": "PASS"|"FAIL", "reasoning": "..." }

The five judge pitfalls (and fixes)

Calibrate the judge against humans. An LLM judge you've never compared to human labels is a number generator, not an evaluator. Maintain a small human-graded set and check that your judge agrees with it before you trust its verdicts. Platforms like Braintrust, Galileo, and Maxim AI provide managed judge frameworks and human-in-the-loop review to keep judges honest.

Part 3: Domain-Specific Evals

Generic evals catch generic problems. RAG systems and agent systems each need their own scorecards.

Evaluating RAG

RAG quality splits into two independently-measurable halves: retrieval (did we fetch the right context?) and generation (did we use it faithfully?). Ragas is the standard framework here, scoring metrics like context precision/recall, faithfulness, and answer relevancy. The key insight: a wrong RAG answer is either a retrieval failure or a generation failure, and you must measure them separately to fix the right one. The full retrieval-and-generation architecture lives in the Production RAG Knowledge Assistant playbook.

Evaluating agents

Agents add a dimension single-shot LLMs don't have: the trajectory. It's not enough to score the final answer — you have to evaluate the path. Did the agent pick the right tools? In a sensible order? Without looping? Agent evals therefore score three things:

1. Final outcome — did it accomplish the goal?
2. Trajectory — was the sequence of tool calls correct and efficient?
3. Tool-call accuracy — were the right tools called with the right arguments?

AgentOps and Langfuse capture the full trajectory so you can replay and score it. This is the observability layer the Multi-Agent Orchestration playbook calls non-negotiable.

Part 4: Evals in CI — The Regression Gate

An eval suite that only runs when you remember to run it isn't a safety net. Wire it into CI so no prompt or model change merges without passing:

text
On every pull request that touches a prompt, chain, or model config:
1. Run the full eval suite against the golden dataset.
2. Compare the aggregate score to the main branch baseline.
3. FAIL the build if the score regresses beyond the threshold (e.g. -2%).
4. Post a diff comment: which test cases newly passed and newly failed.

This turns subjective "I think it's better" debates into objective, reviewable numbers. Promptfoo, Braintrust, and LangSmith all integrate with CI and store historical scores so you can see quality trend over time.

Treat a score regression like a failing unit test. It blocks the merge. The moment evals become advisory, they get ignored under deadline pressure — and your quality quietly erodes one "ship it anyway" at a time.

Part 5: Observability in Production

Once you ship, observability is how you see what your evals couldn't anticipate. Tracing is the foundation — capturing the full execution of every request so you can debug any single interaction.

What a trace must capture

A useful LLM trace records the entire chain, not just the final call:

• The full prompt sent (after template rendering) and the raw completion.
• Every retrieval, tool call, and agent handoff, with inputs and outputs.
• Token counts, latency, and cost per step.
• Model, version, and all parameters.
• A session/user ID to group multi-turn conversations.

Langfuse (open-source) and LangSmith are the leading tracing platforms; Arize Phoenix and Literal AI cover the same ground with strong eval integration. For a lighter-weight start, a gateway like Helicone or Lunary gives you request logging, cost analytics, and caching with a one-line proxy change.

The four signals to monitor

The highest-leverage habit: sample production traces, score them with your judge, and feed the failures back into the golden dataset. That's how your eval suite stays representative of reality instead of slowly going stale.

Part 6: Guardrails and Safety

Evals measure quality; guardrails enforce safety in real time. They are the runtime checks that sit between your model and your user, blocking or correcting bad outputs before they land. Guardrails catch what evals can only warn about:

• Input guardrails — block prompt injection, off-topic requests, and PII before they reach the model.
• Output guardrails — validate structure, strip leaked secrets, and check for toxicity or hallucination before returning a response.

Giskard specializes in scanning LLM apps for vulnerabilities and quality issues, and WhyLabs brings monitoring and guardrails together for governance. The reliable pattern is the same one from the agent playbook: gate anything irreversible behind a check, and never let an unvalidated output trigger a real-world action.

The Tool Map

Map each job to a tool — most teams run one tracing platform plus one eval runner, and add specialists as needed:

• Tracing / observability: Langfuse, LangSmith, Arize Phoenix, Literal AI, LangWatch.
• Eval runners / testing: Promptfoo, DeepEval, Braintrust.
• Managed eval + judge platforms: Galileo, Maxim AI.
• RAG evals: Ragas.
• Agent observability: AgentOps.
• Gateways (logging + cost + caching): Helicone, Lunary.
• Quality, safety & governance: Giskard, WhyLabs, Cleanlab, Argilla.
• Cost optimization: Nebuly.

Common Failure Modes (and How to Avoid Them)

Reference Workflow

text
        DEVELOPMENT                         PRODUCTION
   ┌───────────────────┐             ┌─────────────────────┐
   │  Golden dataset    │            │   Real user traffic  │
   └─────────┬─────────┘             └──────────┬──────────┘
             ▼                                   ▼
   ┌───────────────────┐              ┌─────────────────────┐
   │  Eval suite        │             │   Tracing (every     │
   │  assertions +      │             │   prompt, tool, call)│
   │  LLM-as-judge      │             └──────────┬──────────┘
   └─────────┬─────────┘                         ▼
             ▼                          ┌─────────────────────┐
   ┌───────────────────┐               │  Online evals +      │
   │  CI regression gate │ ◀── blocks  │  user feedback +     │
   │  (no merge if worse)│    merge    │  cost/latency/drift  │
   └───────────────────┘               └──────────┬──────────┘
             ▲                                     │
             └──────── failures become new test cases ◀┘

The arrow that matters most is the bottom one: production failures flow back into the eval set. That loop is what keeps a system trustworthy as the world, the models, and the users change.

Your First Week with LLM Evals & Observability

1. Day 1: Add a tracing layer. Drop in Helicone (one-line proxy) or Langfuse so you can see every production request.
2. Day 2: Hand-pick 20 real examples into a golden dataset — half normal, half hard edge cases.
3. Day 3: Write code-based assertions for everything mechanical (schema, format, refusals) with Promptfoo or DeepEval.
4. Day 4: Add an LLM-as-judge for one subjective dimension (faithfulness or helpfulness) and calibrate it against your own labels on those 20 cases.
5. Day 5: Wire the suite into CI as a regression gate. Now no change ships without a score.
6. Week 2+: Sample production traces, score them, and feed failures back into the golden set. Add cost, latency, and drift dashboards.

The goal isn't a perfect score — it's a trustworthy feedback loop. When every change is measured before it ships and every production failure becomes a test case, your AI stops being a black box you hope works and becomes a system you know works. That's the difference between a demo and a product in 2026.

Frequently Asked Questions

What is the difference between LLM evaluation and LLM observability? LLM evaluation measures quality offline against a fixed dataset before you ship — it answers "is this version good?" LLM observability monitors real production traffic after you ship through traces and metrics — it answers "what's actually happening?" Production systems need both, and the failures observability surfaces should become new evaluation test cases.

What is LLM-as-a-judge? LLM-as-a-judge is an evaluation technique where a strong language model grades another model's output against an explicit rubric. It's the most scalable way to measure subjective qualities like helpfulness, tone, and faithfulness, but it must be calibrated against human labels and protected against biases like position bias and self-preference to be reliable.

How do I evaluate a RAG system? Evaluate retrieval and generation separately. Score retrieval with metrics like context precision and recall (did you fetch the right chunks?), and score generation with faithfulness and answer relevancy (did the model use that context correctly?). Ragas is the standard framework for both halves.

Which LLM observability tool should I use? For tracing and evals, Langfuse (open-source) and LangSmith are the leading platforms. For lightweight request logging, cost tracking, and caching via a one-line proxy, use a gateway like Helicone. For agent-specific trajectory tracing, use AgentOps.