Cutting LLM Spend Without Cutting Quality: Routing, Caching, and Token Budgets

TL;DR: Lower LLM cost by finding the expensive user journeys first. Attribute input, cached input, output, retries, and tool work by feature and tenant. Route simple requests to cheaper models, cache stable prefixes, cap context and output, batch asynchronous work, and degrade gracefully. Keep a quality and completion metric beside every cost change so savings do not come from silently failing users.

Why this matters in 2026

LLM spend is rarely controlled by one model price. SaaS cost grows from request volume, long context, repeated instructions, retrieval payloads, retries, tool calls, background jobs, and users who can trigger unbounded work. A feature that costs a few cents in a demo can become a material gross-margin problem when every workspace runs it on every document or message.

The first mistake is applying a cheaper model before understanding the workload. The second is counting only the model call while ignoring embeddings, search, storage, queueing, and review. Cost engineering is a product and architecture practice. You need a cost per successful task, a quality floor, and a policy for what happens when a tenant reaches a budget.

Providers expose different caching, batch, and pricing semantics. OpenAI prompt caching uses stable prefixes and reports cached input tokens. Anthropic prompt caching supports explicit breakpoints and different cache durations. Gemini context caching documentation describes implicit and explicit caching behavior that varies by model. Read the current provider documentation before building a cost model around a specific feature.

Key terms and mental model

Use this flow for attribution:

user action -> feature and tenant policy -> model route
           -> prompt assembly -> cache lookup -> model and tools
           -> result quality -> usage ledger -> product margin dashboard

The unit is not “tokens per request.” It is “cost per successful business outcome.” A support reply that requires two cheap attempts can cost more than one expensive accurate attempt. A cached prompt may reduce input spend while a long generated answer still dominates the total.

Build a token and cost ledger

Record provider, model, request ID, feature, tenant, plan, user or workspace class, input tokens, cached tokens, output tokens, tool calls, retries, latency, and outcome. Use a hashed tenant identifier in shared telemetry and keep billing identity in a protected ledger. Store a pricing version with each record so a future price change does not rewrite historical economics.

type UsageRecord = {
  requestId: string;
  feature: 'draft' | 'search' | 'classify' | 'agent';
  tenantIdHash: string;
  provider: string;
  model: string;
  inputTokens: number;
  cachedInputTokens: number;
  outputTokens: number;
  toolCostMicros: number;
  retryCount: number;
  outcome: 'success' | 'fallback' | 'error';
  pricingVersion: string;
};

export function computeUsageCost(record: UsageRecord, price: PriceCard) {
  const billableInput = record.inputTokens - record.cachedInputTokens;
  return (
    billableInput * price.inputMicrosPerToken +
    record.cachedInputTokens * price.cachedInputMicrosPerToken +
    record.outputTokens * price.outputMicrosPerToken +
    record.toolCostMicros
  );
}

Do not wait for the provider invoice to discover which feature is expensive. Emit usage after every request, then reconcile totals against provider usage reports. Track unknown model and missing token fields as data-quality errors. A dashboard with 30 percent unattributed spend is a reason to fix instrumentation before tuning prompts.

Reduce context before switching models

Context is often the largest input cost and the largest quality variable. Remove duplicate system instructions, old chat turns, irrelevant retrieval chunks, unused tool descriptions, and schema fields the user cannot reach. Retrieve a small candidate set, rerank it, and pass only the evidence needed for the task. Summarization can save tokens, but it adds a model call and may erase a detail that matters.

Design the prompt in layers:

1. stable policy and role instructions,
2. stable tool schemas and output contract,
3. cached workspace or product context,
4. current retrieved evidence,
5. user request and current state.

Put stable material first when the provider uses prefix caching. Do not insert a changing request ID or timestamp at the front. Keep the cache key scoped to the tenant and policy version. A cache hit must not cross a permission boundary.

Measure quality after context reduction. A smaller prompt that increases unsupported answers is not a saving. Keep a retrieval hit and answer groundedness score with the cost record, especially for knowledge features.

Route requests with a cascade

Not every request needs the largest reasoning model. Use request classification, confidence, and escalation rules. A small model can handle formatting, extraction, classification, and short drafts. A mid-tier model can handle ordinary support or summarization. A larger model can handle ambiguous, high-value, or low-confidence tasks.

export async function runWithRouting(input: RequestInput) {
  const policy = await getTenantPolicy(input.tenantId);
  const route = classifyTask(input);

  if (policy.monthlyBudgetRemainingMicros < route.minimumCostMicros) {
    return gracefulDegrade(input, 'budget');
  }

  const first = await callModel({
    model: route.fastModel,
    input,
    maxOutputTokens: route.fastOutputLimit,
  });

  const quality = await scoreFastResult(first, input);
  if (quality.accepted || !route.allowEscalation) return first;

  return callModel({
    model: route.strongModel,
    input,
    maxOutputTokens: route.strongOutputLimit,
    metadata: { escalatedFrom: route.fastModel },
  });
}

The classifier and quality check have costs too. Set an escalation budget and measure the percentage of requests that move up. If most requests escalate, the first model may add cost and latency without value. Use a stable evaluation set to tune the confidence threshold, and keep a manual or deterministic path for simple tasks.

Use caching with correct boundaries

Prompt caching works best when a long prefix is identical across requests. Put system instructions, tool definitions, response schemas, and stable workspace context in the reusable prefix. Keep user input and fast-changing retrieval evidence later. Provider cache windows and minimum lengths differ, so report hit rate and cached tokens rather than assuming every request will hit.

Cache application results separately from prompt prefixes. A product policy answer can be cached by document version and tenant policy. A personalized account answer should use a short TTL or no shared result cache. Include model, prompt, tool schema, retrieval index, and source versions in the cache key.

Invalidation must follow data ownership. When a workspace changes a policy document, remove or version cached answers that depend on it. When a user loses access, do not serve a previously cached response without an authorization check. The cheapest cache hit is still a security bug if it returns stale private data.

Batch and queue asynchronous work

Summaries, embeddings, classifications, and nightly recommendations often do not need a synchronous response. Put them on a queue, deduplicate by source version, batch requests where the provider supports it, and expose progress or last-generated time in the product. OpenAI’s Batch API documentation describes an asynchronous completion window and batch endpoints. Other providers have different limits and semantics.

export async function enqueueSummary(document: Document) {
  const key = `summary:${document.id}:${document.version}`;
  const existing = await jobs.findByIdempotencyKey(key);
  if (existing) return existing;

  return jobs.enqueue('document-summary', {
    idempotencyKey: key,
    tenantId: document.tenantId,
    documentId: document.id,
    version: document.version,
    runAfter: new Date(),
    priority: document.plan === 'enterprise' ? 10 : 1,
  });
}

Batching changes user experience and retry behavior. Do not make a request wait on a job that is meant to be asynchronous. Make jobs resumable, record partial failures, and stop generating work for deleted or inaccessible documents.

Enforce per-tenant budgets and graceful degradation

A shared API key without tenant budgets creates an economic and abuse boundary problem. Calculate a monthly or daily allowance by plan, rate-limit bursts, and reserve capacity for interactive actions. Use a ledger that is idempotent on provider request ID so retries do not double charge your internal balance.

Graceful degradation should be a product decision. Options include a smaller model, shorter context, a queued result, a user-provided bring-your-own-key path, a deterministic template, or a clear temporary unavailability message. Do not silently return an empty or low-quality answer and call it success.

Alert on spend velocity, cost per successful task, cache miss rate, escalation rate, retry rate, and tenant outliers. The agent evaluation guide shows why quality and trajectory metrics should travel with cost metrics for agent features.

Set budget behavior at the product boundary

Budget enforcement should happen before an expensive model call and after usage is recorded. Before the call, estimate the maximum cost from model, input size, output limit, and tool plan. After the call, debit the actual usage and reconcile any provider adjustment. Use a reservation for concurrent requests so five simultaneous calls cannot all observe the same remaining balance.

export async function reserveBudget(input: { tenantId: string; estimateMicros: number }) {
  const result = await db.transaction(async (tx) => {
    const budget = await tx.budgets.lockForUpdate(input.tenantId);
    if (budget.remainingMicros < input.estimateMicros) return { allowed: false };

    await tx.budgets.update(input.tenantId, {
      remainingMicros: budget.remainingMicros - input.estimateMicros,
      reservedMicros: budget.reservedMicros + input.estimateMicros,
    });
    return { allowed: true, reservationId: crypto.randomUUID() };
  });

  if (!result.allowed) return gracefulDegrade(input, 'budget');
  return result;
}

Release unused reservation after the response and debit actual usage. If a provider call times out, the reservation should remain visible until reconciliation decides whether the request completed. This prevents a retry storm from turning an uncertain network error into unbounded spend.

Give each plan a predictable experience. A free plan might receive a queued summary, while an enterprise interactive workflow gets reserved capacity and a stronger fallback. Keep the policy visible in product copy and support documentation.

Connect cost controls to user value

Cost attribution is most useful when it follows a product event. Record whether a generated draft was accepted, edited heavily, discarded, converted into a ticket, or used in a completed workflow. For search, record whether the user found an answer, asked again, opened a source, or escalated. These outcomes let you compare model and prompt changes by value rather than by token count.

Use cohort analysis for expensive tenants. A large workspace may have legitimate document volume, while another may have a runaway integration or a retry loop. Show the tenant administrator which features consume allowance, what is queued, and how to reduce work. Keep a platform-level circuit breaker for provider incidents and a tenant-level circuit breaker for abuse, but make both recoverable through an operator workflow.

Budget policies should support migrations. When changing models or prompt versions, reserve a small experiment budget and compare control and treatment cohorts. Do not roll a new route to every tenant just because a benchmark is cheaper. A feature with a lower mean cost can have a dangerous p99 cost from oversized inputs or repeated escalations. Track distributions and outliers.

CREATE TABLE ai_usage_ledger (
  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  tenant_id uuid NOT NULL,
  request_id text NOT NULL UNIQUE,
  feature text NOT NULL,
  model text NOT NULL,
  input_tokens integer NOT NULL,
  cached_tokens integer NOT NULL DEFAULT 0,
  output_tokens integer NOT NULL,
  cost_micros bigint NOT NULL,
  outcome text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ai_usage_tenant_day_idx
  ON ai_usage_ledger (tenant_id, created_at DESC);

Keep the ledger append-only and derive balances or reports from it. If you need corrections for a provider invoice, add an adjustment record rather than rewriting the original usage. This preserves the audit trail when a customer questions a bill or an internal model route changes.

For internal planning, convert the ledger into three views. The engineering view shows token and latency drivers. The product view shows cost per successful workflow and the tenants or features that create value. The finance view shows provider invoice reconciliation, committed capacity, credits, and the price card used for each period. One raw event stream can support all three without forcing each team to calculate cost differently.

Set a review threshold for prompt changes. A small instruction edit can change input tokens, cache hit rate, output length, and escalation probability. Require a usage and quality comparison for prompts that affect a high-volume feature, even when the code diff is only a few lines. Store the prompt version in the trace and usage ledger so the change can be rolled back independently from the application release.

For internal planning, convert the ledger into three views. The engineering view shows token and latency drivers. The product view shows cost per successful workflow and the tenants or features that create value. The finance view shows provider invoice reconciliation, committed capacity, credits, and the price card used for each period. One raw event stream can support all three without forcing each team to calculate cost differently.

Set a review threshold for prompt changes. A small instruction edit can change input tokens, cache hit rate, output length, and escalation probability. Require a usage and quality comparison for prompts that affect a high-volume feature, even when the code diff is only a few lines. Store the prompt version in the trace and usage ledger so the change can be rolled back independently from the application release.

Inspect the long tail, not only the average. One tenant with an oversized history or one integration retrying a failed request can consume more capacity than hundreds of ordinary users. Add maximum input size and maximum tool depth, reject duplicate jobs, and surface an operator alert before the provider bill arrives. Limits protect both margins and the availability of interactive requests.

Model unit economics with a quality floor

Create a before and after table from an actual feature cohort. The numbers below are an illustrative method, not provider pricing:

The table is useful only when the quality floor is explicit. Define acceptable task success, refusal, groundedness, latency, and support-contact rates. If a control saves 40 percent but causes a 5 percent completion drop on a paid workflow, the business outcome may be negative.

Tradeoffs and when not to do this

The cheapest model is not always the cheapest system. Routing adds classification and evaluation calls. Caching adds invalidation and privacy design. Batching reduces price or infrastructure pressure but adds delay. Shorter context reduces cost but may reduce recall or groundedness. Budgets can protect margin while frustrating users if the product does not explain the state.

Do not optimize a low-volume feature before fixing instrumentation. Do not cache mutable or personalized answers without an authorization model. Do not use a larger model as a blanket remedy for poor retrieval, unclear tools, or bad context. Fix the largest source of unnecessary work first.

Common failure modes

Teams often count output tokens but ignore context and retries. Another failure is putting a timestamp or request ID at the beginning of every prompt, which destroys prefix reuse. A third is routing by user plan only, even though task difficulty varies widely within a plan.

Cost ledgers also fail when retries produce duplicate records or background jobs regenerate the same source version. Use idempotency keys and source versioning. Finally, a budget fallback that is not user-visible can turn a cost control into a trust problem. Tell the user whether the result is queued, abbreviated, or unavailable.

Production readiness checklist

• Usage is attributed by provider, model, feature, tenant, plan, request, and outcome.
• Input, cached input, output, retries, tools, retrieval, and storage costs are represented.
• Stable prompt prefixes are designed for provider caching and scoped by policy and tenant.
• Context reduction is measured against groundedness and task success.
• Model routing and escalation thresholds are evaluated on a representative set.
• Asynchronous work uses idempotency, source versions, retries, and deletion checks.
• Per-tenant budgets, burst limits, and interactive capacity reservations exist.
• Graceful degradation is explicit in the product experience.
• Pricing versions and provider invoice reconciliation are stored.
• Dashboards show cost per successful task beside quality and latency.

Frequently Asked Questions

Should I switch to a cheaper model first?

Usually no. First measure which tasks, tenants, prompts, and retries create spend. Remove duplicate context, cap unnecessary output, cache stable prefixes, and fix repeated failures. Then route a measured subset to a cheaper model and compare task success, groundedness, latency, escalation, and support contacts. A cheap model that needs a second attempt or creates review work may cost more per successful outcome.

What is the best place to use prompt caching?

Use it for long, stable prefixes such as system policy, tool definitions, response schemas, and versioned workspace context. Put changing user input and retrieval results after that prefix when the provider supports prefix caching. Check the provider’s minimum token, cache lifetime, scope, and usage reporting rules. Never let a cached prefix or result cross a tenant or authorization boundary.

How should SaaS pricing account for LLM usage?

Start with internal cost per successful task by feature and tenant plan. Add a usage allowance or fair-use policy, reserve expensive capabilities for plans that support their margin, and show enough status that users understand queued or limited work. Avoid promising unlimited high-cost actions until you have measured heavy users, retries, and model price changes.

When should work move to a batch job?

Move work when the user does not need the result in the current interaction. Document summaries, embeddings, classifications, and scheduled recommendations are common candidates. Use an idempotency key tied to source version, make progress visible, and handle deletion or permission changes before execution. A batch job should improve cost and capacity without making the product pretend the result is immediate.

How do I avoid cost optimizations that damage quality?

Define a quality floor before changing the model, context, or output limit. Run a representative evaluation set, use field feedback after rollout, and compare cost per successful task rather than cost per request. Keep safety, groundedness, completion, latency, and escalation metrics beside spend. Roll back a change when it crosses a business-critical threshold, even if the provider invoice is lower.

Conclusion and next steps

LLM cost engineering starts with visibility and ends with a product policy. Attribute the full workflow, shorten and cache context, route by difficulty, batch asynchronous work, enforce tenant budgets, and keep quality beside spend.

Compare the AI SaaS architecture guide with your current feature map, then use the AI integration cost guide to turn one feature’s usage ledger into a plan and margin review.