Production LLM pipelines: structured outputs, tool use, evals and guardrails

Short answer: treat the model as an untrusted part of the system. Constrain its output to a schema and validate it, give it a few narrow tools with a person approving consequential actions, and mark outside text as data. Then route work by difficulty, cache what's stable and score every change against an evaluation set built from real cases.

A demo needs a prompt. A production system needs everything around the prompt: constrained outputs, narrow tools, defences against hostile input, cost control, measurement and a plan for failure.

These are the patterns we apply when we put Claude or Gemini inside a business workflow.

Free text is hard to act on. When a model's answer has to drive software (create a ticket, fill a form, update a CRM record), ask for structured output that matches a schema, or have the model call a tool whose input schema is strict: required fields, enumerations for fixed choices, no extra properties. Both Claude (opens in a new tab) and Gemini (opens in a new tab) can hold their output to a JSON schema.

Then validate again on your side. Model output is untrusted input, exactly like a form submission. Parse it with a schema validator, and if it fails, return the specific error to the model once and let it correct itself. If it fails twice, stop and escalate rather than guessing.

Agents become dependable when their abilities are narrow and explicit:

• Few, well-named tools with clear descriptions of when to use them and when not to.
• Least privilege: read-only by default; writes limited to the records and fields the task needs.
• Approval for consequential actions: sending email to a customer, changing an order or issuing a refund goes through a person, at least until the evidence says otherwise.
• Idempotency: a retried tool call must not create two invoices.

Emails, web pages, uploaded documents and form fields can contain instructions written by someone else: "ignore your previous instructions and…". That is prompt injection, first on the OWASP Top 10 for LLM applications (opens in a new tab), and the defence is architectural:

• Pass external content to the model clearly marked as untrusted data, with an instruction that it never overrides the system's rules.
• Restrict what an agent can do with what it reads: who it can email, which links it can include, which tools it can call.
• Escalate anything that looks like social engineering (changed bank details, urgent payment requests, requests for credentials) to a human.

Not every step needs the largest model:

• Use small, fast models such as Claude Haiku or Gemini Flash-Lite for classification, routing and extraction, and reserve the larger models for reasoning and writing. Anthropic positions Haiku (opens in a new tab) for exactly these high-volume tasks.
• Keep long, stable instructions in a cached system prompt. Both Anthropic (opens in a new tab) and Google (opens in a new tab) bill cached input at a tenth of the normal input price or less, which cuts the cost and latency of every turn.
• Send offline work, such as nightly document processing, through the providers' batch APIs (Anthropic (opens in a new tab), Google (opens in a new tab)), which cost half the standard price.

"It seemed fine when I tried it" does not survive contact with real users. Build an evaluation set from real cases (typical requests, awkward edge cases, known failures) and score every change against it:

• Deterministic checks for anything with a right answer: correct tool chosen, required fields present, no prices outside policy.
• Rubric grading by a model for qualities like tone, completeness and faithfulness to sources, spot-checked by people.

Run the suite whenever you change a prompt, a model or a tool. Quality becomes a number you can track rather than a feeling.

Models refuse, time out, run out of output tokens and occasionally go down. Production code should:

• Check why a response ended (finished, refused, truncated) before using it.
• Retry transient errors with backoff, and fall back to a second provider for interactive features where that makes sense.
• Degrade gracefully: a helpful message and a human contact route beat a spinner.
• Log every model call with tokens, latency, cost and outcome, so you can see problems before your customers do.

Assistants that talk to customers should say they are AI and offer a route to a person. Anthropic's usage policy (opens in a new tab) requires that disclosure for consumer-facing chatbots and AI agents, and it is simply good practice.

Pip, Softpiper's AI on this site, follows these rules. It runs on Claude with a fixed set of strict tools, discloses that it is an AI and can hand you to the team. When you ask what a project might cost, Pip does not make up a number: it calls a deterministic estimator that applies our pricing policy for your market and scope, then explains the range.

If you are planning an AI feature or agent, see our AI agents and AI product engineering practices, or book a call.

• Anthropic Usage Policy (opens in a new tab): consumer-facing chatbots and AI agents must tell users they are talking to AI.
• Claude: structured outputs (opens in a new tab) and Gemini: structured output (opens in a new tab)
• OWASP Top 10 for LLM applications: LLM01 Prompt Injection (opens in a new tab)
• Claude models overview (opens in a new tab)
• Claude: prompt caching (opens in a new tab) and Gemini: context caching (opens in a new tab)
• Claude: batch processing (opens in a new tab) and Gemini: batch inference (opens in a new tab)