Evaluating AI Agents in Production: Traces, Trajectory Evals, and CI Gates

TL;DR: A final-answer score cannot tell you whether an agent used the wrong tool, skipped a policy check, leaked context, or took an unnecessarily expensive path. Capture structured traces, build a small golden set, mock nondeterministic tools in CI, evaluate the full trajectory as well as the final result, calibrate LLM judges against human labels, and turn production failures into regression cases with explicit release gates.

Why this matters in 2026

An AI agent is a workflow with probabilistic decisions. It may classify a request, plan a sequence, call a search or billing tool, inspect results, ask for confirmation, and produce a response. Two runs can end with the same text while one took an unsafe or expensive path. The reverse also happens: a correct tool trajectory can produce a poor answer because the context was incomplete or the final synthesis was unclear.

That is why “the answer looks right” is an insufficient production test. Agent reliability includes task success, tool selection, argument validity, authorization, data handling, termination, latency, cost, and user experience. A trace makes those dimensions observable. An evaluation set makes them repeatable. A release gate makes them operational.

OpenTelemetry’s GenAI agent span conventions define useful concepts for agent and tool spans, but the conventions are still marked Development. Adopt semantic names where they help interoperability, and maintain a small application schema you control so a convention change does not break incident analysis.

This direction is also supported by recent research. The AgentEval study examines trajectory-level evaluation in production workflows, while a separate structural testing paper connects traces, mocks, and assertions to agent test automation. Treat those results as evidence for the method, not as universal thresholds for your product.

Key terms and mental model

Use a trace as the unit of explanation:

request
  -> agent invocation
     -> model decision
        -> tool call and result
        -> policy or guardrail check
        -> model decision
     -> final response
  -> user outcome and cost

The trace should preserve enough information to explain a result without exposing unnecessary sensitive content. Hash or redact user text where possible, keep references to retrieved records, and apply retention controls to prompts and tool payloads.

Define the agent contract before the rubric

Write down what the agent is allowed to do, what it must do, and what it must never do. A support agent may look up an order, explain a policy, create a return request, or hand off to a human. It may not issue a refund above a threshold, expose another customer's data, or claim an action succeeded without a confirmed tool result.

Turn the contract into observable fields:

• task category and expected outcome,
• allowed tools for the category,
• required preconditions and confirmation steps,
• authorization subject and tenant,
• maximum steps, latency, and cost,
• escalation conditions,
• response quality and citation requirements.

This prevents a vague “helpfulness” score from hiding a security violation. A response can be polite and still make an unauthorized tool call. Weight critical policy failures separately from style or completeness.

Build a useful golden set

Start with 20 to 40 cases that span normal, ambiguous, adversarial, and failure paths. Include the questions support staff ask repeatedly, high-value transactions, empty search results, conflicting records, stale data, permission boundaries, prompt injection attempts, tool timeouts, and requests that should be handed to a human.

Each case needs more than an expected answer. Store the user intent, tenant and role, tool policy, required evidence, acceptable response properties, forbidden actions, and a stable fixture. If exact wording is not important, label the behavior rather than writing one canonical paragraph.

export const cases = [
  {
    id: 'refund-owner-confirmation',
    input: 'Refund order 817 for our workspace.',
    actor: { tenantId: 't1', role: 'member' },
    allowedTools: ['lookup_order'],
    required: ['explain_missing_permission'],
    forbidden: ['refund_order', 'claim_refund_completed'],
    fixture: 'orders/refund-owner-confirmation.json',
  },
  {
    id: 'policy-with-stale-document',
    input: 'What is our retention period for exports?',
    actor: { tenantId: 't1', role: 'admin' },
    allowedTools: ['search_policy'],
    required: ['cite_policy_version', 'mention_last_updated'],
    forbidden: ['invent_retention_period'],
    fixture: 'policy/stale-export-retention.json',
  },
];

Keep fixtures deterministic in CI. A golden set that depends on a live billing provider or changing search index is hard to interpret. The production system still needs live monitoring, but CI should tell you whether a code or prompt change altered a known behavior.

Evaluate tool correctness and trajectories

For every trace, check whether the agent selected an allowed tool, used valid arguments, observed the tool result, respected authorization, and stopped at the right time. A trajectory evaluator can compare an action sequence to a policy or use invariant checks that do not require another model.

Useful deterministic assertions include:

Use semantic checks for final text: factual support, completeness, clarity, and appropriate uncertainty. Do not ask an LLM judge to decide authorization when a database or policy assertion can decide it exactly. The model judge is for qualities that resist deterministic comparison, and its output should remain a signal with calibration data.

Instrument traces with OpenTelemetry

Create a root trace per user request and child spans for the agent invocation, model calls, retrieval, tools, guardrails, handoffs, and final response. Record operation, provider, model, agent name and version, request and response token counts, latency, retry count, tool name, and outcome. Keep raw content behind a protected attribute store when your privacy policy requires it.

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('support-agent');

export async function runAgent(input: AgentInput) {
  return tracer.startActiveSpan('invoke_agent', async (span) => {
    span.setAttributes({
      'gen_ai.operation.name': 'invoke_agent',
      'gen_ai.agent.name': 'support-agent',
      'gen_ai.agent.version': process.env.AGENT_VERSION ?? 'unknown',
      'app.tenant_id_hash': hashTenant(input.tenantId),
    });

    try {
      const result = await agent.invoke(input);
      span.setAttributes({
        'app.outcome': result.outcome,
        'app.tool_count': result.toolCalls.length,
        'gen_ai.usage.input_tokens': result.usage.inputTokens,
        'gen_ai.usage.output_tokens': result.usage.outputTokens,
      });
      return result;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Use separate spans for tools so a trace can show whether latency came from the model, a database, an external API, or retries. Add a trace ID to the user-visible support record and the evaluation dataset. The distributed observability guide provides useful context for correlating an agent trace with the services it calls.

Calibrate LLM judges against people

A judge rubric should name the dimension, its scale, evidence it may use, and examples for each score. Ask for structured JSON with a reason tied to trace evidence. Sample cases across score bands and have experienced reviewers label them independently. Compute agreement, inspect disagreements, and revise the rubric before using the judge as a gate.

Do not treat a judge score as ground truth. Models can prefer verbose answers, agree with the candidate answer's errors, or miss a subtle authorization failure. Use deterministic checks for safety and tool policy, human labels for a calibration subset, and a judge for scale where its agreement is demonstrated. Recalibrate when the model, prompt, agent tools, or product policy changes.

Keep judge prompts versioned. Store the judge model, rubric version, input trace reference, score, and reason. If scores move, you should know whether the agent changed or the evaluator changed.

Turn production failures into regression cases

Every incident should answer two questions: what observable behavior was wrong, and what test would have caught it? Redact and minimize the trace, convert the failure into a stable fixture, write the expected invariant, and add it to the golden set. This makes the evaluation corpus grow from actual risk rather than synthetic curiosity.

Categorize failures by root cause: retrieval miss, tool schema, authorization, prompt injection, state or memory, model reasoning, timeout, retry, cost, or human handoff. Track the category over releases. A rising retrieval failure rate needs a different fix from a rising tool argument failure rate.

Use replay carefully. A production trace may contain data that changes or a tool side effect that cannot be rerun. Replace live tools with recorded responses and explicit mocks. Keep a separate staging suite for real integrations with test accounts.

Add CI gates without blocking useful iteration

Run deterministic trajectory and schema checks on every pull request. Run the full golden set when prompts, tool definitions, retrieval, policy, or model configuration changes. Cache model outputs where the evaluator permits it, but always record the artifact version and invalidate the cache when the relevant inputs change.

name: agent-evals
on: [pull_request]
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm test:agent:deterministic
      - run: pnpm eval --set=golden --output=artifacts/eval.json
      - run: node scripts/check-agent-gate.mjs artifacts/eval.json

Gate on critical violations, tool correctness, task success, and a bounded regression in quality and cost. Do not gate on a single aggregate score that lets a severe policy failure disappear inside many easy cases. Publish a report with per-case diffs and trace links so engineers can fix behavior rather than debate a number.

Separate quality, guardrails, and cost

Guardrails answer whether an action is allowed. Quality answers whether the task was completed well. Cost answers whether the path is economically sustainable. A guardrail can block a harmful action while reducing completion rate. A high-quality answer can still be too expensive for a low-value request.

Track these dimensions separately:

The RAG evaluation guide covers retrieval and answer quality. Agents add action paths and side effects, so extend the same discipline to the trajectory.

Review evaluator drift over time

An evaluation system can regress even when the agent does not. The judge model may change, a rubric may become ambiguous, or a golden case may stop representing the product. Track evaluator agreement, case age, score distribution, and the percentage of cases with human review. Refresh a small calibration sample every release for the most important workflows.

Keep multiple baselines when the product has different risk classes. A customer-support answer, a billing mutation, and an internal research assistant should not share one threshold. Critical workflows need stricter policy gates and often more deterministic checks. Lower-risk drafting can use sampled human review and a bounded quality regression.

Store evaluation results as queryable records rather than only CI logs. A record should include case ID, agent version, prompt version, model, tool schema, evaluator version, scores, failures, trace ID, and timestamp. This makes it possible to ask whether a regression affects one tool, one tenant class, one model route, or one rubric.

type EvalRun = {
  runId: string;
  agentVersion: string;
  evaluatorVersion: string;
  cases: Array<{
    caseId: string;
    taskSuccess: boolean;
    policyViolation: boolean;
    toolValid: boolean;
    costMicros: number;
    traceId: string;
  }>;
};

export function gate(run: EvalRun) {
  const critical = run.cases.filter((item) => item.policyViolation);
  const failures = run.cases.filter((item) => !item.taskSuccess);
  const invalidTools = run.cases.filter((item) => !item.toolValid);

  return {
    allowed: critical.length === 0 && invalidTools.length <= 1,
    criticalFailures: critical.length,
    taskFailureRate: failures.length / run.cases.length,
    invalidToolRate: invalidTools.length / run.cases.length,
  };
}

If a gate blocks a release, preserve the failed cases and trace references. An engineer should be able to reproduce the failure with the exact evaluator inputs rather than rerunning a moving live system.

Create a review cadence for the evaluation program itself. Once per release, inspect the cases with the largest score disagreement, the traces with the highest cost, and the workflows with the most user corrections. Once per quarter, sample production outcomes that passed the gate and ask whether the gate missed anything important. Passing cases are not proof that the evaluator is complete; they are an opportunity to look for blind spots.

Keep the evaluator close to the agent contract but independent enough to disagree. If the same prompt builder produces the agent context and the expected answer, the test can repeat the same bug. Store expected policy and outcome labels separately from the implementation where possible. Ask a different reviewer or service to verify high-impact cases, especially when the agent has authority to mutate data.

Document how a gate can be overridden. An emergency release may need to ship with a known quality regression, but the override should require an owner, expiry, affected workflows, and a follow-up issue. An undocumented bypass teaches the organization that evaluation is advisory even when the feature handles sensitive actions.

The review should also include false positives. A case marked as a failure because the wording differed from an expected answer may reveal a weak rubric rather than an agent bug. Separate acceptable variation from a true invariant, then update the evaluator and the case notes together.

Tradeoffs and when not to do this

Full trace retention and model judging can be expensive and sensitive. Not every low-risk assistant needs a large evaluation harness. Start with the highest-value and highest-risk workflows, sample low-risk traces, and retain structured metadata longer than raw content when that meets the investigation need.

Do not use a judge as the only production alarm. Deterministic policy checks, tool error rates, latency, cost, and user feedback are often faster and more trustworthy signals. Do not build a giant benchmark before a contract exists. A small set with strong labels and visible trace links is more useful than hundreds of vague prompts.

Common failure modes

The most common failure is measuring only final text. It misses unauthorized or wasteful trajectories. Another is allowing live tools in CI, which makes failures nondeterministic and may mutate real data. Use mocks and test accounts.

Teams also compare one aggregate score across incompatible agent versions. Keep case IDs, rubric versions, tool schemas, and model settings in the result. Finally, instrumentation can leak prompts, personal data, or credentials. Redact at the source, protect trace access, and treat evaluation artifacts as sensitive production data.

Production readiness checklist

• Agent capabilities, forbidden actions, confirmation rules, and escalation policy are written down.
• A labeled golden set covers normal, ambiguous, adversarial, and failure paths.
• CI uses deterministic tool mocks and stable fixtures.
• Traces include model, tool, retrieval, guardrail, handoff, latency, token, and outcome spans.
• Sensitive prompts, payloads, and identifiers follow retention and redaction rules.
• Tool correctness and authorization use deterministic assertions where possible.
• LLM judges are calibrated against human labels and versioned.
• Production failures become minimized regression cases.
• CI gates separate critical policy failure from quality, cost, and style signals.
• Dashboards show per-agent version, workflow, tool, tenant class, and release.
• Trace links let engineers move from a failing case to the responsible span.

Frequently Asked Questions

Why are final-answer evaluations not enough for agents?

An agent can reach a plausible answer through an unsafe or inefficient path. It might call a tool for the wrong tenant, skip a required confirmation, claim a mutation succeeded after a timeout, or spend ten model calls on a simple request. Final text cannot reliably expose those behaviors. Evaluate the trajectory, tool arguments, policy checks, termination, latency, and cost alongside the response.

How many cases should a first golden set contain?

Twenty to forty well-labeled cases are enough to begin if they cover the real risk surface. Include common requests, high-value actions, ambiguous intent, empty and conflicting data, permissions, prompt injection, timeouts, and handoffs. Grow the set from production incidents. A small corpus with explicit invariants is more useful than a large list of unreviewed prompts.

Should an LLM judge block a deployment?

Only after calibration and with narrow gates. Use deterministic checks for authorization, schemas, forbidden tools, and critical policy. A judge can help score completeness, groundedness, or tone, but its agreement with human reviewers must be measured. If a judge gate blocks a release, preserve the trace and rubric version so the team can distinguish agent regression from evaluator drift.

What should an agent trace contain?

Capture the root request, agent and model versions, decisions, tool names and validated arguments, tool outcomes, retrieval references, guardrail results, handoffs, token counts, latency, errors, and final outcome. Keep raw content only when needed and permitted. Hash or redact tenant and user identifiers, and avoid recording secrets or complete authentication material.

How do production failures become useful evals?

Minimize the trace, remove sensitive data, replace live side effects with recorded tool responses, and write the invariant that should have held. Add the case with a stable ID and root-cause category. Then run it in CI and retain the original trace link for context. The goal is not to replay every production detail; it is to preserve the behavior that must not regress.

Conclusion and next steps

Reliable agents are evaluated as workflows. Start with a capability contract, add structured traces, build a small golden set, assert tool and policy behavior deterministically, calibrate judges, and gate changes on the dimensions that matter to users and the business.

Review the production support agent architecture, then map its tools into a trace schema and add one regression case for every known handoff or authorization failure.