Services

LLMOps evaluation and rollout playbook

Evaluate prompts, models, retrieval, and tools before AI changes reach users.

LLMOps is the operating system around LLM applications: versioned prompts, controlled model changes, evaluation suites, deployment workflow, observability, and cost governance. Use this playbook when AI tooling has moved beyond ad hoc experimentation.

Operating model#

Treat every AI change as a release artifact.

ArtifactVersion with
Prompt templatesApplication code and release notes.
Model configurationProvider, model, temperature, max tokens, tools, and safety settings.
Retrieval contextIndex version, source snapshot, chunking strategy, and access filters.
Tool contractsJSON schemas, permissions, idempotency behavior, and timeout policy.
Evaluation dataGolden examples, adversarial cases, expected formats, and acceptance thresholds.

Evaluation layers#

Fast checks in pull requests#

Run deterministic checks before reviewers approve a change:

  • Prompt template renders for every required variable.
  • Structured output schemas parse successfully.
  • Tool names and arguments match registered contracts.
  • Token budget stays below the workflow limit.
  • Safety fixtures do not trigger disallowed behavior.

Offline quality gates#

Use a stable test set for release decisions:

  • Golden tasks for expected user workflows.
  • Edge cases gathered from production traces and support tickets.
  • Retrieval questions with known source documents.
  • Regression examples from previous incidents.
  • Representative high-cost or high-latency prompts.

Track pass rate by prompt/model/index version. Do not compare only aggregate scores; review failures by category so teams know whether the change broke retrieval, reasoning, formatting, safety, or tool use.

Online monitoring#

Production feedback loops should measure:

  • User-rated success and escalation rate.
  • Task completion and human-approval outcomes.
  • Cost per successful task.
  • Latency by provider/model/region.
  • Tool error rate and retry rate.
  • Retrieval empty-context and low-relevance rates.

Rollout sequence#

  1. Shadow test against historical traces with no user impact.
  2. Internal dogfood with trace sampling and manual review.
  3. Limited cohort with strict budget and rollback thresholds.
  4. Progressive rollout by tenant, team, route, or workflow.
  5. Steady-state review with scheduled eval refresh and cost review.

Use feature flags for user exposure, but keep prompt/model/index configuration versioned in the deployment pipeline so releases are reproducible.

Risk controls#

  • Keep provider API keys scoped per environment and rotate them through the same secret-management process as other production credentials.
  • Cap tokens, tool calls, retries, and concurrent runs.
  • Require approval for production-impacting tools.
  • Record model/provider changes in release notes.
  • Validate generated API payloads outside the model before execution.
  • Store trace data under the same retention and access policies as other sensitive telemetry.

Cost governance#

Cost issues usually come from prompt growth, retrieval bloat, failed retries, or provider/model changes.

Add dashboards for:

  • Spend by model, workflow, tenant, and environment.
  • Tokens per successful task.
  • Top prompts by cost and error rate.
  • Retry spend and timeout spend.
  • Forecasted month-end cost from current usage.

Set alert thresholds before launch. A useful starting point is warning at 50 percent of monthly budget, urgent at 80 percent, and automatic exposure reduction or approval-only mode at 100 percent.

Example release record#

yaml
1
change: support-assistant-rag-v4
2
owner: support-platform
3
prompt_version: 2026-06-26.4
4
model: gpt-4.1-mini
5
retrieval_index: support-kb-2026-06-25
6
risk_level: medium
7
evals:
8
  golden_tasks: 94.2%
9
  structured_output: 100%
10
  safety_regression: 100%
11
  retrieval_relevance: 91.8%
12
rollout:
13
  phase: limited-cohort
14
  cohort: internal-support-team
15
  rollback_if:
16
    error_rate: '> 3% for 15m'
17
    cost_per_task: '> €0.08 p95'
18
    escalation_rate: '> baseline + 20%'

Handoff checklist#

  • Prompt, model, retrieval, and tool versions are captured.
  • Evaluation suites run in CI or a release workflow.
  • Rollout and rollback thresholds are explicit.
  • Trace sampling and sensitive-data redaction are configured.
  • Cost dashboards and alerts are live.
  • Owners know how to refresh evals from production traces.

References#