AI Engineering: A Production Guide to Building with Foundation Models

A field guide to the engineering discipline that sits between a foundation model and a production system — context, retrieval, agents, evaluation, cost, and governance, with the trade-offs made explicit.

• Introduction
• 1. The New Discipline of AI Engineering
• 2. Foundation Models — Capabilities, Limitations, and Evaluation
• 3. Model Selection and Model Routing
• 4. Prompt Engineering and Structured Outputs
• 5. Context Construction and Context Engineering
• 6. Retrieval-Augmented Generation
• 7. Embeddings, Retrieval, Reranking, and Hybrid Search
• 8. Agents, Tools, Workflows, and Planning
• 9. Memory and State Management
• 10. Evaluation Methods and Evaluation-Driven Development
• 11. Human Evaluation and Automated Evaluators
• 12. Hallucinations, Reliability, and Uncertainty
• 13. Latency, Throughput, Scalability, and Cost
• 14. Fine-Tuning vs. Prompting vs. RAG
• 15. Data Collection, Feedback Loops, and Production Monitoring
• 16. Guardrails, Security, Privacy, and Governance
• 17. Designing an End-to-End Production AI Application
• 18. What Has Changed Since the Book Was Written
• 19. Prototype-to-Production Maturity Model
• Conclusion
• Checklists
• Key Takeaways
• 30-Day Learning Plan
• Further Reading
• References

Introduction

In 2022, building a production ML system meant months of data engineering, feature pipelines, model training, hyperparameter search, and serving infrastructure. In 2025, a competent engineer can wire together a surprisingly capable document-analysis system in an afternoon. That shift is real, but it has created a misleading impression: that the hard engineering problems have been solved. They have not been moved. They have been relocated downstream, from model training to application design.

The discipline that occupies this new space is what Chip Huyen, in her 2025 book AI Engineering: Building Applications with Foundation Models, calls AI engineering. The field is distinct from traditional machine learning engineering in both its tools and its failure modes. An ML engineer tuning a gradient-boosted tree worries about feature drift, label quality, and retraining frequency. An AI engineer deploying a foundation-model application worries about prompt injection, context window limits, hallucination rate, inference cost at scale, and how to evaluate a system whose outputs can never be exhaustively enumerated.

This guide covers all of those concerns systematically. It is written for software engineers and technical leaders who understand distributed systems, know what a REST API is, and are now responsible for making an LLM-powered system work reliably in production — not just in a demo. Wherever concepts originate from Huyen's book, they are attributed. Where this guide diverges, adds independent analysis, or updates with developments that post-date the book, that is noted explicitly.

1. The New Discipline of AI Engineering

What changed and why it matters

Traditional ML engineers were primarily responsible for the model. They collected data, trained a model specific to one task (fraud detection, churn prediction, demand forecasting), evaluated it against held-out test sets, and served it through a relatively thin inference layer. The model was the hard part, and it was owned end to end by specialists.

Foundation models — large, pre-trained neural networks capable of general-purpose language, code, and reasoning tasks — inverted this. The model is now a commodity, or close to one. GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 70B, and a dozen open-weight alternatives can all solve most single-step text tasks adequately. The differentiator is no longer the model itself but the system built around it: how context is constructed, how outputs are verified, how the application degrades gracefully under failure, and how it is kept aligned with business requirements over time.

Huyen frames this as a two-axis shift: from models that are trained from scratch to models that are adapted through prompting or fine-tuning; and from single-step inference to multi-step, stateful applications (agents). Both shifts move engineering complexity from training to application design.

The AI engineering stack

A production AI application is not a model call. It is a layered system:

Each layer in this diagram represents distinct engineering decisions with real trade-offs. The LLM itself — the G node — is often the least complex component to swap. The layers surrounding it are where most production challenges live.

AI engineering vs. traditional ML engineering

The table highlights why teams that are excellent at traditional ML sometimes struggle with AI engineering: the bottlenecks are different, the tooling is different, and the feedback loops are different.

2. Foundation Models — Capabilities, Limitations, and Evaluation

What foundation models are and are not

A foundation model (a term coined by the Stanford HAI group in 2021, though the underlying concept predates that) is a large neural network trained on broad, diverse data at scale, intended to be adapted to downstream tasks through prompting, fine-tuning, or retrieval augmentation rather than trained from scratch for each task. GPT-4, Claude, Gemini, and the Llama family are the most widely-used examples today.

Huyen emphasizes that understanding what these models actually do under the hood — predicting the next token given a context — is important for building reliable systems. The model has no persistent state between calls, no ability to verify facts at inference time against an external source unless explicitly given tools, and no intrinsic sense of when it is and is not confident.

Three properties are critical for AI engineers to internalize:

Context-dependence. Every response is entirely conditioned on what is in the context window at the time of the call. The model "knows" nothing outside of its training data and the current context. This is why context construction is the dominant engineering problem.

Non-determinism. Even at temperature zero, most hosted APIs introduce non-determinism through batching, quantization differences, and prompt caching. Production systems must handle variability in outputs.

Stochastic parroting is a real risk. Models trained on the web are very good at generating plausible-sounding text, which means confident-sounding wrong answers are a fundamental property of the architecture, not a bug to be patched.

Evaluating foundation model capabilities

Benchmark-driven capability evaluation is the standard approach for selecting models, but benchmark results rarely transfer cleanly to specific use cases. Huyen's framework for evaluating model capabilities distinguishes between:

• Knowledge and reasoning: does the model know facts relevant to your domain, and can it apply them through multi-step inference?
• Instruction following: does the model do precisely what the prompt says, including format requirements, length constraints, and negative instructions?
• Long-context performance: does accuracy degrade when relevant information appears early in a long context (the "lost in the middle" problem)?
• Tool use and structured output: can the model reliably call external tools and produce output in specified formats?
• Refusal and safety behavior: does the model appropriately refuse harmful requests without over-refusing legitimate ones?

For any production use case, the right evaluation approach is to build a task-specific benchmark from real representative inputs before committing to a model. Pre-built benchmarks like MMLU, HELM, and BigBench measure population-level academic capabilities. They tell you almost nothing about whether a particular model will reliably extract specific fields from your insurance claims forms.

The capability frontier vs. the deployment frontier

There is a persistent gap between what the leading models can do and what organizations actually deploy. This gap exists for three reasons:

1. Cost. The most capable models are the most expensive per token. At production scale, a 10× capability improvement rarely justifies a 10× cost increase.
2. Latency. Larger models are slower. User-facing applications often have hard latency budgets (under 500ms for interactive UIs) that the largest models cannot meet.
3. Trust and governance. Regulated industries frequently cannot send data to cloud-hosted foundation models at all, requiring local deployment of open-weight models that lag the frontier by months.

Understanding this gap is prerequisite to sensible architecture decisions.

3. Model Selection and Model Routing

The model selection decision

Choosing the right model is one of the highest-leverage decisions in AI engineering because it determines both the capability ceiling and the cost floor of your application. Huyen presents model selection as a multi-dimensional optimization problem rather than a single ranking exercise.

The relevant axes are:

Quality vs. cost. The frontier models (GPT-4o, Claude Sonnet 3.5, Gemini 1.5 Pro) are roughly comparable in general capability but differ in domain-specific strengths, pricing per million tokens, and available context windows. Smaller models (GPT-4o-mini, Claude Haiku, Gemini Flash) are 10–50× cheaper and 3–5× faster, with meaningfully lower quality on complex tasks.

Context window. As of mid-2026, most frontier models offer 128K–1M token context windows. For many use cases, context length is the deciding factor — a model that fits your entire knowledge base in one call eliminates the need for a retrieval layer entirely, at the cost of increased per-call latency and expense.

Open vs. closed. Open-weight models (Llama 3, Mistral, Qwen, Phi) can be self-hosted, which is the only option in many regulated environments. The capability gap between open and closed models has narrowed significantly since 2024, particularly for code and structured extraction tasks.

Modality. If your use case requires vision (document images, charts, medical scans) or audio (call center transcripts, voice assistants), model selection must account for multimodal capabilities.

A model-selection framework

The four axes above are not independent — a decision on one constrains the others. The following framework sequences them in the order that eliminates the most options fastest, which keeps the evaluation workload manageable instead of requiring a full bake-off across every available model on every axis:

This framework front-loads the constraints that are binary and organizational (data residency, modality) before the constraints that are continuous and tunable (latency, cost), because the binary constraints usually eliminate 80% of candidate models before any benchmarking work begins.

Model routing

Model routing is the practice of dynamically selecting which model handles a given request based on the estimated complexity, cost budget, or user tier. Huyen frames this as a practical cost-reduction strategy: the majority of production requests in most applications are simple enough for cheap, fast, small models. Only a minority require frontier-model capability.

A simple routing heuristic:

A more sophisticated approach uses a cascade: route to the cheapest model first, and if the response fails a quality gate (confidence threshold, format check, or a secondary classifier), escalate to a better model. This pattern can reduce cost by 60–80% on real production distributions while maintaining quality on the cases that matter.

# Illustrative model routing with cascade
import anthropic

client = anthropic.Anthropic()

SIMPLE_MODEL = "claude-haiku-4-5-20251001"
COMPLEX_MODEL = "claude-sonnet-5"

def route_and_complete(prompt: str, max_attempts: int = 2) -> str:
    """Route a prompt to the cheapest capable model."""
    for attempt, model in enumerate([SIMPLE_MODEL, COMPLEX_MODEL]):
        response = client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        result = response.content[0].text

        if passes_quality_gate(result) or attempt == max_attempts - 1:
            return result

    return result

def passes_quality_gate(text: str) -> bool:
    # Replace with your actual quality checks: schema validation,
    # confidence parsing, length checks, etc.
    return len(text.strip()) > 20 and not text.startswith("I cannot")

Note: This is a simplified illustration. Production routing should include structured output validation, cost tracking, and fallback logging.

4. Prompt Engineering and Structured Outputs

What prompt engineering actually is

Prompt engineering is widely misunderstood as an art form — a collection of magic phrases. Huyen's treatment is more useful: it is systematic communication design for non-deterministic text generators. The goal is to specify intent unambiguously enough that the model reliably produces the desired output across the full distribution of plausible inputs, not just the examples you tested.

This reframing has practical implications. A prompt is not a one-time message — it is a contract between your system and the model. It needs to be:

• Specific about format: what the output should look like, including length, structure, and delimiters
• Specific about scope: what the model should and should not do
• Robust to adversarial inputs: valid even when users try to override it
• Versioned and tested: like code, not like a Post-it note

The anatomy of an effective system prompt

A production system prompt typically contains:

1. Role and persona: who or what the model is acting as
2. Task specification: what it must do, with examples if needed
3. Output format: exactly how the response should be structured
4. Constraints and guardrails: what topics, tones, or actions are off-limits
5. Escalation instructions: what the model should do when it does not know the answer

CLAIMS_EXTRACTION_PROMPT = """You are a claims extraction assistant for a property insurance company.

Your task is to extract structured information from claims descriptions submitted by policyholders.

Output ONLY a valid JSON object with these exact keys:
{
  "claim_type": "one of: property_damage | theft | liability | weather",
  "incident_date": "ISO 8601 date or null if not stated",
  "location": "address or null",
  "estimated_value": "integer in USD or null",
  "injuries_reported": true | false,
  "confidence": "high | medium | low"
}

Rules:
- Do not invent information not present in the text.
- If a field cannot be determined, use null.
- Never include explanatory text outside the JSON object.
- If the description does not relate to an insurance claim, set all fields to null and confidence to "low".
"""

def extract_claim(description: str) -> dict:
    import json
    response = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=512,
        system=CLAIMS_EXTRACTION_PROMPT,
        messages=[{"role": "user", "content": description}],
    )
    return json.loads(response.content[0].text)

Structured outputs

Unstructured text generation is fine for conversational applications. For any application that needs to integrate model outputs into downstream systems — databases, APIs, business logic — you need reliable structured outputs.

The three main approaches in order of reliability:

1. Prompt-only (least reliable): instruct the model to output JSON. Works 80–95% of the time depending on model and prompt quality. Fails on edge cases.

2. JSON mode / constrained generation: most providers expose a mode that enforces valid JSON syntax. This is a significant improvement but does not enforce your specific schema.

3. Function calling / tool use (most reliable): define the output schema as a tool definition. The model is trained to fill this schema correctly and modern frontier models do so with very high reliability.

// TypeScript: structured output via Anthropic tool use
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface ClaimExtraction {
  claim_type: "property_damage" | "theft" | "liability" | "weather" | null;
  incident_date: string | null;
  estimated_value: number | null;
  injuries_reported: boolean;
  confidence: "high" | "medium" | "low";
}

async function extractClaim(description: string): Promise<ClaimExtraction> {
  const response = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 512,
    tools: [
      {
        name: "extract_claim",
        description: "Extract structured claim information from a description",
        input_schema: {
          type: "object" as const,
          properties: {
            claim_type: {
              type: ["string", "null"],
              enum: ["property_damage", "theft", "liability", "weather", null],
            },
            incident_date: { type: ["string", "null"] },
            estimated_value: { type: ["integer", "null"] },
            injuries_reported: { type: "boolean" },
            confidence: { type: "string", enum: ["high", "medium", "low"] },
          },
          required: ["claim_type", "incident_date", "estimated_value",
                     "injuries_reported", "confidence"],
        },
      },
    ],
    tool_choice: { type: "tool", name: "extract_claim" },
    messages: [{ role: "user", content: description }],
  });

  const toolUse = response.content.find((c) => c.type === "tool_use");
  return toolUse?.input as ClaimExtraction;
}

Common prompt engineering failure modes

• Instruction collision: you say "be concise" and also "explain your reasoning step by step" — the model has to guess which wins
• Vague scope: "answer questions about our product" without defining what "our product" covers means the model will hallucinate details
• No negative examples: telling the model what to do without showing what not to do leaves a large space for unexpected behavior
• Overloaded prompts: a system prompt that tries to handle 40 different edge cases degrades performance on the common cases

5. Context Construction and Context Engineering

Context is the engineering problem

If there is a single insight that distinguishes experienced AI engineers from those who are new to the field, it is this: the quality of your application is almost entirely determined by the quality of the context you construct, not the model you call. Given a perfect context, a moderately capable model will outperform a frontier model given a vague one.

Huyen introduces the concept of context engineering — the systematic practice of deciding what information to include in each model call, in what order, in what format, and at what level of detail. Context engineering is a superset of prompt engineering: it covers the full content of the context window, not just the system prompt.

A context window typically contains:

Context window budgeting

Every token in the context window costs money and time. The engineering challenge is fitting maximum relevant information into the context budget. Three pressures compete:

1. Relevance: irrelevant context degrades performance ("distraction effect")
2. Completeness: missing context causes hallucination or hedging
3. Cost and latency: more tokens = higher cost and slower responses

A practical budget allocation for a RAG-augmented customer-support agent (128K context):

Conversation history deserves special attention. Without truncation, it grows unboundedly. Common strategies:

• Sliding window: keep the last N turns, discard older ones
• Summarization: periodically compress older history into a running summary
• Selective retention: keep only turns that contained decisions, facts, or tool results

Context poisoning and injection

Any context that comes from external sources — user inputs, retrieved documents, emails, web pages — is a potential injection vector. A malicious document in your RAG corpus can contain text like "Ignore all previous instructions and..." and influence the model. This is not hypothetical: it has been demonstrated against production systems across every major model family.

Practical mitigations:

• Structural delimiters: wrap retrieved content in clear, consistent XML-like tags (<document>, <user_input>) and instruct the model to treat them as data, not instructions
• Input validation and sanitization: strip or escape common injection patterns before they enter the context
• Output monitoring: flag outputs that deviate from expected patterns, suggesting possible injection success
• Principle of least privilege: do not give the model access to tools or data it does not need for the current task

6. Retrieval-Augmented Generation

The core pattern

Retrieval-Augmented Generation (RAG) is the practice of dynamically retrieving relevant information from an external knowledge base and injecting it into the model's context at inference time. It addresses the most common production failure mode of foundation models: answering questions about domain-specific knowledge that postdates the training data, is proprietary, or requires precision the model cannot reliably recall.

The basic RAG pipeline:

RAG vs. long-context models

An important design question that has emerged since long-context models became available: why retrieve at all if you can fit the entire knowledge base in the context window?

The answer depends on your use case:

For most production use cases today, RAG remains the right architecture for corpora larger than a few hundred thousand tokens. Long-context calls work well for single-document analysis, code review, and meeting summarization where the entire artifact is known upfront.

Chunking strategy

How you divide your source documents into retrievable chunks is one of the highest-impact decisions in a RAG system. The wrong chunking strategy undermines everything downstream.

Common approaches:

• Fixed-size chunking: split into chunks of N tokens with M-token overlap. Simple and predictable, but may split sentences or paragraphs mid-thought.
• Recursive character splitting: split on paragraph breaks first, then sentence breaks, then character count. Better preserves semantic units.
• Semantic chunking: use an embedding model to identify natural semantic boundaries. More expensive but produces higher-quality chunks.
• Document structure-aware chunking: split on document-native boundaries (headings, sections, table rows). Works best when documents have consistent structure.

Overlap is the practice of including the last N tokens of the previous chunk at the start of the next. It reduces the likelihood that a retrieval query that spans a chunk boundary returns nothing useful. Typical overlap: 10–20% of chunk size.

# Recursive character splitting with overlap — provider-neutral illustration
def chunk_text(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    """Split text into overlapping chunks preserving sentence boundaries."""
    sentences = text.replace('\n', ' ').split('. ')
    chunks, current, current_len = [], [], 0

    for sentence in sentences:
        words = sentence.split()
        if current_len + len(words) > chunk_size and current:
            chunks.append('. '.join(current) + '.')
            # Keep last `overlap` words for next chunk
            overlap_words = ' '.join(current[-1].split()[-overlap:])
            current = [overlap_words + ' ' + sentence]
            current_len = len(current[0].split())
        else:
            current.append(sentence)
            current_len += len(words)

    if current:
        chunks.append('. '.join(current))
    return chunks

7. Embeddings, Retrieval, Reranking, and Hybrid Search

Embeddings and vector search

An embedding is a dense vector representation of a piece of text (or image, code, etc.) that captures its semantic meaning. Similar texts produce similar vectors, which enables similarity-based retrieval: "find me chunks that are semantically close to this query."

Popular embedding models include OpenAI's text-embedding-3-small/large, Cohere Embed v3, and open-weight alternatives like BGE-M3 and E5. For production use, the key dimensions to evaluate are:

• Retrieval quality (MTEB benchmark score for your domain)
• Dimensionality (higher = better quality but larger index)
• Context length (how much text one embedding can capture)
• Multilingual support if your corpus is multilingual
• Cost per million tokens if using a hosted API

Vector databases (Pinecone, Weaviate, Qdrant, pgvector for PostgreSQL, Chroma for local use) store these embeddings and support approximate nearest-neighbor search (ANN) at scale. ANN trades a small amount of recall for large gains in query speed.

The retrieval quality problem

Vector search works well when the query and the relevant document use similar vocabulary. It fails when:

• The query is short and ambiguous ("why did this fail?")
• The relevant document uses different terminology ("error analysis" vs. "failure investigation")
• The query requires exact matching (policy number, product SKU, date)

Hybrid search

Hybrid search combines vector search with traditional keyword (BM25) search. The two signals are complementary: BM25 excels at exact-match and rare-term queries; vector search excels at paraphrase and semantic similarity.

The combination significantly outperforms either approach alone on most retrieval benchmarks. The typical implementation uses a Reciprocal Rank Fusion (RRF) to merge the two ranked lists without needing to calibrate scores onto a common scale.

Reranking

Reranking is a second-pass scoring step that applies a more expensive, more accurate cross-encoder model to the top-k retrieved candidates to re-order them before passing them to the LLM. The key insight is that retrieval optimizes for speed and scale (returning 50–100 candidates in milliseconds), while reranking optimizes for precision (scoring a small set of candidates against the query using full cross-attention).

Cohere Rerank, Cross-Encoder models from HuggingFace, and open-source alternatives like ColBERT are the main options. Reranking consistently reduces the "noise" in retrieved context and improves generation quality on complex RAG tasks.

# Illustrative reranking pipeline using Cohere
import cohere
from typing import List

co = cohere.Client()  # API key from environment

def retrieve_and_rerank(
    query: str,
    documents: List[str],
    top_n: int = 5
) -> List[str]:
    """Retrieve and rerank documents for a query."""
    rerank_response = co.rerank(
        model="rerank-english-v3.0",
        query=query,
        documents=documents,
        top_n=top_n,
    )
    return [documents[r.index] for r in rerank_response.results]

8. Agents, Tools, Workflows, and Planning

The agent spectrum

The word "agent" has been stretched to cover everything from a simple RAG pipeline to fully autonomous systems. A more useful framing: AI systems exist on a spectrum of autonomy and loop complexity.

Huyen is precise about the definition: a true agent is a system where the LLM decides what actions to take (including which tools to call and in what order), observes the results, and iterates until it determines the task is complete. This loop is what distinguishes agents from pipelines.

Tool use

Tools are the mechanism by which an agent interacts with the world: calling APIs, querying databases, executing code, browsing the web, reading files. The model does not execute tools directly — it generates a structured specification of the tool to call, which the application layer executes and then returns as the next context input.

Effective tool design principles:

• Atomic, purpose-specific tools: one tool, one action. A "do_everything" tool is not callable reliably.
• Rich tool descriptions: the model selects tools based on their description. Poor descriptions lead to wrong tool selection.
• Idempotent tools where possible: agents retry on failure, so non-idempotent tools (send_email, execute_wire_transfer) can cause double-execution.
• Error outputs as first-class citizens: tools should return structured errors that the model can reason about, not raw exceptions.

# Tool definition for a treasury data lookup agent
TREASURY_TOOLS = [
    {
        "name": "get_account_balance",
        "description": (
            "Retrieve the current cash balance for a specified account. "
            "Use this when the user asks about available funds, current balance, "
            "or liquidity position for a named account."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "account_id": {
                    "type": "string",
                    "description": "The account identifier (e.g. 'USD-NOSTRO-001')"
                },
                "as_of_date": {
                    "type": "string",
                    "description": "ISO 8601 date for which to retrieve the balance. "
                                   "Use today's date if not specified."
                }
            },
            "required": ["account_id"]
        }
    }
]

Planning and multi-step reasoning

For complex tasks that require planning — decomposing a goal into subtasks, tracking progress, adapting to unexpected results — the agent needs a mechanism for structured reasoning before acting. Two dominant patterns:

ReAct (Reason + Act): the model alternates between reasoning steps (what do I know? what should I do next?) and action steps (call this tool). This is the natural pattern that emerges from chain-of-thought prompting combined with tool use, and it is what most modern agent frameworks implement.

Plan-and-execute: the model first generates a full plan (a sequence of steps), then executes each step. Useful when the task structure is known upfront and parallel execution is desirable, but less adaptive to unexpected tool results.

Multi-agent systems

Multi-agent architectures use multiple LLM instances — typically one orchestrator and one or more specialized subagents — to handle tasks that are too long for a single context window, require parallel execution, or benefit from specialist roles. Huyen discusses this pattern in the context of breaking tasks into independent subtasks that can be verified and composed.

Multi-agent architectures introduce coordination overhead and new failure modes: agents can disagree, loop on each other, or fail to hand off state correctly. They should be adopted when there is a demonstrated need, not as a default architecture.

9. Memory and State Management

The statelessness problem

Foundation models are stateless by design: each API call is a fresh computation with no memory of previous calls. For conversational applications, multi-session workflows, or agents that accumulate knowledge, this statelessness is a fundamental engineering challenge.

Huyen organizes memory into four types that correspond to different engineering solutions:

In-context memory (working memory): information in the current context window. Zero latency, zero infrastructure, but bounded by the context limit and destroyed at the end of the call.

External memory (long-term memory): information stored in a database and retrieved on demand. Persistent, scalable, and queryable, but introduces retrieval latency and retrieval quality risk.

In-weights memory (parametric memory): knowledge baked into the model through training or fine-tuning. Zero retrieval cost but requires a retraining cycle to update.

In-cache memory (KV cache): intermediate computation states cached by the serving infrastructure. Used transparently to avoid recomputing common prompt prefixes — reduces cost and latency significantly for applications with shared, stable system prompts.

Practical memory patterns

For a customer-support agent that needs to remember past interactions:

# Simplified memory manager for a support agent
from datetime import datetime
from typing import TypedDict

class MemoryEntry(TypedDict):
    timestamp: str
    type: str       # "fact" | "decision" | "preference"
    content: str
    importance: str # "high" | "medium" | "low"

class AgentMemoryManager:
    def __init__(self, db_client, user_id: str):
        self.db = db_client
        self.user_id = user_id

    def save(self, entry: MemoryEntry) -> None:
        self.db.insert("agent_memory", {
            "user_id": self.user_id,
            "timestamp": datetime.utcnow().isoformat(),
            **entry
        })

    def retrieve_relevant(self, query: str, limit: int = 10) -> list[MemoryEntry]:
        # Semantic search over stored memories
        return self.db.vector_search(
            table="agent_memory",
            query_embedding=embed(query),
            filter={"user_id": self.user_id},
            limit=limit
        )

    def build_memory_context(self, query: str) -> str:
        memories = self.retrieve_relevant(query)
        if not memories:
            return ""
        lines = [f"- [{m['type']}] {m['content']}" for m in memories]
        return "Relevant context from previous sessions:\n" + "\n".join(lines)

State machines for agent control

One underused but highly reliable pattern for agentic applications is to encode the agent's workflow as an explicit state machine. Instead of letting the model navigate freely, constrain it to predefined states and transitions. This dramatically reduces the space of possible behaviors and makes the system auditable.

Frameworks like LangGraph, CrewAI, and Microsoft AutoGen have converged on graph-based agent workflow representations precisely because they give engineers explicit control over which transitions are valid.

10. Evaluation Methods and Evaluation-Driven Development

Why evaluation is the hardest problem in AI engineering

In traditional software, you test by checking whether the output matches the expected output. In AI applications, the output is never a single correct value — it is a distribution over plausible responses. "Evaluate an AI application" means measuring whether the distribution of outputs is sufficiently good, across the full distribution of plausible inputs.

This is hard for three reasons:

1. Output space is infinite: you cannot enumerate all possible responses to check them
2. Quality is multidimensional: accuracy, helpfulness, tone, safety, format, and citation quality are all separate dimensions that may trade off against each other
3. Ground truth requires expertise: for most domain-specific applications, only a human expert can determine whether a response is correct

Huyen's central thesis on evaluation is that it should drive development: build your evaluation pipeline first, then use it to guide every prompt change, model upgrade, and architecture decision. The term evaluation-driven development describes this approach.

The evaluation pyramid

Code-based evals are the foundation. They check things that can be verified programmatically: schema compliance, string presence, format correctness, unit tests for tool implementations. You should have hundreds of these running in CI.

Golden datasets are small curated sets (50–500 examples) where a human expert has verified the correct output. They are the ground truth that everything else is calibrated against. You should build your golden dataset before you build your product.

LLM-as-a-Judge scales golden dataset quality to thousands of examples automatically. A judge model applies a human-written rubric and scores each response. Its validity depends entirely on calibration against the golden dataset — an uncalibrated judge is misleading, not helpful.

User evals are the ultimate measure: does the product work for real users? Thumbs-up rates, task completion, escalation frequency, and retention all tell you things that offline evals cannot.

Evaluation methods comparison matrix

The four layers trade off along the same three axes as everything else in this guide — speed, cost, and fidelity to what a human expert would actually judge:

The practical implication: code-based evals and LLM judges should run on every change because they are cheap, while the human golden dataset and user evals are the periodic checkpoints that keep the cheap layers honest. A team that only has an LLM judge with no human-labeled reference is optimizing against a target it has never verified.

Building an evaluation pipeline

A minimal but production-grade evaluation pipeline:

# Evaluation harness — provider-neutral, illustrative
from dataclasses import dataclass
from typing import Callable
import json

@dataclass
class EvalCase:
    id: str
    input: str
    expected_output: dict  # For structured tasks
    rubric: str            # For LLM judge

@dataclass
class EvalResult:
    case_id: str
    passed: bool
    score: float           # 0.0 to 1.0
    explanation: str

def run_eval_suite(
    cases: list[EvalCase],
    model_fn: Callable[[str], str],
    judge_fn: Callable[[str, str, str], EvalResult],
) -> dict:
    results = []
    for case in cases:
        actual = model_fn(case.input)

        # Code-based check first (fast)
        try:
            parsed = json.loads(actual)
            format_pass = all(k in parsed for k in case.expected_output)
        except json.JSONDecodeError:
            format_pass = False

        # LLM judge for quality (slower)
        judge_result = judge_fn(case.input, actual, case.rubric)

        results.append({
            "case_id": case.id,
            "format_pass": format_pass,
            "quality_score": judge_result.score,
            "explanation": judge_result.explanation,
        })

    passing = [r for r in results if r["format_pass"] and r["quality_score"] >= 0.7]
    return {
        "pass_rate": len(passing) / len(results),
        "avg_quality": sum(r["quality_score"] for r in results) / len(results),
        "results": results,
    }

11. Human Evaluation and Automated Evaluators

Human evaluation: when it is necessary

Human evaluation is slow, expensive, and non-scalable. It is also the only mechanism with genuine validity for evaluating whether an AI system's outputs are actually correct. Every automated evaluation method is ultimately an approximation of what a human expert would say.

When is human evaluation required?

• Domain accuracy: only a domain expert (physician, lawyer, financial analyst) can verify whether clinical notes, legal briefs, or risk assessments are correct
• Tone and appropriateness: whether a customer-facing message strikes the right emotional tone is a human judgment
• Safety and policy compliance: whether an output violates policy is ultimately a human interpretation
• Calibrating automated evaluators: you need human labels to verify that your LLM judge is trustworthy

LLM-as-a-Judge: the right way

When implemented correctly, LLM-as-a-Judge can achieve human-expert-level agreement on many quality dimensions. The implementation details that determine whether it works:

Provide a detailed, unambiguous rubric. Vague rubrics ("is this response good?") produce garbage judgments. Specific rubrics ("does the response correctly identify all three risk factors mentioned in the source document? Score 0, 1, or 2 based on how many are correctly named.") produce reliable judgments.

Use chain-of-thought reasoning before the score. Requiring the judge to explain its reasoning before giving a score improves calibration and provides interpretable output.

Validate against your human golden dataset. Run the judge against the golden dataset and measure its agreement rate. Below 80% agreement with humans, the judge is not reliable enough to trust.

Avoid self-evaluation bias. An LLM judge prefers outputs from models similar to itself and outputs that match its own stylistic preferences. Cross-judge with a different model family when possible.

# LLM-as-a-Judge implementation (illustrative)
JUDGE_PROMPT = """You are an expert evaluator for AI-generated insurance claim analyses.

Evaluate the response on the criterion below. Reason step by step, then output your score.

Criterion: {criterion}

Source text: {source}
AI response: {response}

First, write your reasoning (2-4 sentences).
Then output: SCORE: <number from 0 to 3>
0 = Completely fails the criterion
1 = Partially meets the criterion with significant gaps
2 = Meets the criterion with minor gaps
3 = Fully meets the criterion
"""

def judge_response(source: str, response: str, criterion: str) -> tuple[int, str]:
    prompt = JUDGE_PROMPT.format(
        criterion=criterion, source=source, response=response
    )
    raw = call_llm(prompt, model="claude-sonnet-5", max_tokens=256)
    lines = raw.strip().split('\n')
    score_line = [l for l in lines if l.startswith("SCORE:")]
    score = int(score_line[-1].split(":")[1].strip()) if score_line else 0
    reasoning = "\n".join(l for l in lines if not l.startswith("SCORE:"))
    return score, reasoning

12. Hallucinations, Reliability, and Uncertainty

Understanding hallucination

Hallucination is the phenomenon where a language model generates output that is factually incorrect, unsupported by the provided context, or internally inconsistent, but presented confidently as if it were true. The term is imprecise but widely used.

A more useful taxonomy distinguishes:

• Intrinsic hallucination: the model's output contradicts information in the provided context (most dangerous in RAG applications)
• Extrinsic hallucination: the model invents information not supported or refuted by the context (the source cannot be verified)
• Factual recall errors: the model states a known fact incorrectly, typically because the training data was incorrect, the model has poor recall, or the fact has changed since training

For most production applications, the most dangerous form is intrinsic hallucination in RAG systems, where the model synthesizes an answer that contradicts the retrieved documents it was given.

Measuring and reducing hallucination

Practical strategies:

Citation and grounding: require the model to cite specific passages from retrieved documents. A model that must cite cannot hallucinate as easily, and you can programmatically verify that the cited passage actually exists.

Faithfulness evaluation: measure whether the model's response is fully entailed by the provided context. This can be automated using an NLI (natural language inference) model or an LLM judge with a faithfulness rubric.

Confidence calibration: explicitly ask the model to rate its confidence. Well-calibrated models can be useful for routing uncertain cases to humans or to RAG retrieval.

Dual-pass verification: generate a response, then pass it back to the model with the original sources and ask "does this response accurately reflect the source documents?" This simple check catches a significant fraction of intrinsic hallucinations.

FAITHFULNESS_JUDGE = """
You are evaluating whether an AI response is fully supported by the provided source documents.

Source documents:
{documents}

AI response:
{response}

For each factual claim in the response, determine if it is:
- SUPPORTED: directly stated or clearly implied by the source documents
- UNSUPPORTED: not mentioned in the sources
- CONTRADICTED: directly contradicted by the sources

Output JSON: {{"faithfulness_score": 0.0-1.0, "unsupported_claims": [...], "contradicted_claims": [...]}}
"""

Uncertainty and calibration

A well-calibrated model is one where its expressed confidence matches its actual accuracy. If a model says "I'm 90% confident" 100 times, it should be correct about 90 of those times.

Most foundation models are poorly calibrated out of the box — they are trained to be helpful and definitive, which creates a systematic bias toward overconfidence. Engineering mitigations include:

• Prompting for uncertainty expression: explicitly instruct the model to say "I don't know" or "I'm not certain" when appropriate, and provide examples
• Ensemble-based uncertainty: run the same prompt multiple times at temperature > 0 and measure response variance as a proxy for model uncertainty
• Retrieval-backed grounding: if the model cannot cite a source for a claim, treat the claim as uncertain

13. Latency, Throughput, Scalability, and Cost

The performance triangle

Production AI systems face a three-way tension between quality, cost, and latency. You can usually improve any two at the expense of the third.

The shape of this trade-off holds consistently across model tiers, even though exact prices and latencies shift every few months as providers ship new models. The two charts below illustrate the pattern using generic model tiers rather than specific benchmark numbers:

Both charts are illustrative: they show the well-documented directional relationship (bigger, more capable models cost more and respond slower) using generic tier labels, not measured benchmark scores or vendor pricing for any specific model. The actual numbers for any given provider change frequently enough that hard-coding them into an architecture decision is a mistake — what should inform the decision is the shape of the curve and where your latency and cost budgets sit on it.

Latency: where time goes

In a typical LLM API call, latency has two components:

Time to first token (TTFT): the time between the API request and the first token in the response. This is dominated by prompt processing (prefill). Long contexts significantly increase TTFT.

Tokens per second (TPS): the rate at which output tokens are generated after the first token. This is where user-perceived responsiveness mainly lives for long responses.

Optimization strategies:

• Streaming: return tokens as they are generated. Users perceive streaming responses as faster even when total time is similar.
• Prompt caching: many providers cache the KV state of common prompt prefixes. A stable system prompt that is cached reduces TTFT by 50–80% on repeat calls.
• Response length control: explicitly limit output length to what you actually need. Generating 2,000 tokens when 200 would suffice is both slower and more expensive.
• Parallel calls: for tasks that can be decomposed (e.g., analyze 10 documents), fire all 10 API calls simultaneously rather than sequentially.

Cost modeling

LLM inference is priced by token (input + output), with output tokens typically 3–5× more expensive than input. A simple cost model:

def estimate_monthly_cost(
    daily_requests: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    input_price_per_1m: float,   # e.g., 3.0 for Claude Sonnet
    output_price_per_1m: float,  # e.g., 15.0 for Claude Sonnet
) -> float:
    monthly_requests = daily_requests * 30
    input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * input_price_per_1m
    output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * output_price_per_1m
    return input_cost + output_cost

# Example: claims extraction at scale
monthly = estimate_monthly_cost(
    daily_requests=10_000,
    avg_input_tokens=800,
    avg_output_tokens=150,
    input_price_per_1m=3.0,
    output_price_per_1m=15.0,
)
print(f"Estimated monthly cost: ${monthly:,.2f}")
# → Estimated monthly cost: $945.00

For high-volume use cases, three levers materially reduce cost:

1. Model downsizing via cascades (described in Section 3): route simple requests to cheap models
2. Prompt caching: cache stable system prompts — often reduces effective input cost by 50%+
3. Batch processing: many providers offer async batch APIs at 50% discount for latency-tolerant workflows

14. Fine-Tuning vs. Prompting vs. RAG

Choosing the right adaptation strategy

This is one of the most consequential decisions in building a production AI application. Huyen provides a useful framework: start with prompting, add RAG when prompting is insufficient, and consider fine-tuning only when both are insufficient or when deployment constraints require it.

When fine-tuning is appropriate

Fine-tuning bakes a specific behavior into the model weights. It is appropriate when:

• Consistent tone or format: you need the model to consistently follow a very specific output format that prompting cannot reliably enforce
• Domain-specific vocabulary: the model frequently gets domain-specific terminology wrong and you have labeled examples of correct usage
• Latency/cost at scale: you need to reduce the system prompt to near-zero for a very high-volume, well-defined task
• On-premises or air-gapped deployment: you need to run a model that you fully control

Fine-tuning is not appropriate when:

• The knowledge changes frequently (fine-tuning cannot be updated in real time)
• You do not have sufficient labeled examples (typically need 500–10,000 well-curated pairs)
• The quality improvement does not justify the engineering cost (fine-tuning pipelines are complex to build and maintain)
• Prompting or RAG has not been fully optimized first

Practical fine-tuning workflow

1. Establish a baseline: measure your current system performance on a curated eval set
2. Prepare a training set: curate high-quality (input, output) pairs — quality matters far more than quantity
3. Fine-tune on a small model: start with the smallest model that might work; iterate fast
4. Evaluate against baseline: fine-tuning can improve some dimensions while regressing others — always compare on your full eval suite
5. Monitor in production: fine-tuned models can drift or behave unexpectedly on distribution-shifted inputs

RLHF, RLAIF, and preference tuning

Beyond supervised fine-tuning, there are reinforcement-learning-based methods (RLHF, DPO, RLAIF) that align the model's output distribution with human or AI preferences. These are primarily used by model providers to produce instruction-following models from base models. For application-level fine-tuning, supervised fine-tuning on curated examples is the right starting point for almost all teams.

15. Data Collection, Feedback Loops, and Production Monitoring

The data flywheel

The best AI applications become better over time because they collect better data over time. This is the data flywheel: production usage generates signals (correct outputs, user corrections, thumbs-up/down), which feed back into training and evaluation data, which improve the model, which generates better outputs and more engagement.

Building a flywheel requires explicit instrumentation from day one. Teams that defer data collection to "once we have scale" typically find that by the time they have scale, their logging schema is incompatible with what they need.

The full loop connects the evaluation pyramid from Section 10 to production monitoring and back into context and prompt engineering — this is the mechanism that makes an AI application improve after launch rather than only before it:

Minimum required instrumentation:

# Structured trace log — illustrative schema
from dataclasses import dataclass, asdict
from datetime import datetime
import uuid

@dataclass
class LLMCallTrace:
    trace_id: str
    session_id: str
    user_id: str
    timestamp: str
    model: str
    prompt_tokens: int
    completion_tokens: int
    latency_ms: int
    system_prompt_hash: str    # Track prompt versions
    input_text: str
    output_text: str
    tool_calls: list           # If agent
    user_feedback: str | None  # thumbs_up | thumbs_down | None

def log_trace(trace: LLMCallTrace) -> None:
    # Write to your observability system (Datadog, Langfuse, custom)
    print(asdict(trace))  # Replace with actual logging

Production monitoring

AI applications need monitoring at two levels: infrastructure and model behavior.

Infrastructure metrics are standard: request rate, error rate, latency percentiles, cost per day. These tell you whether the system is up and affordable.

Model behavior metrics are new and often neglected:

• Output format compliance rate: what fraction of responses pass your format validator?
• Refusal rate: what fraction of responses are refusals? A sudden spike signals a prompt change or a shift in user inputs.
• Retrieval quality proxy: for RAG systems, do the retrieved chunks contain the answer? (This can be measured with a simple heuristic judge.)
• User satisfaction proxy: thumbs-up rate, follow-up clarification rate, escalation rate
• Latency by model and prompt version: helps identify regressions after prompt changes

Shadow evaluation

Shadow evaluation runs your new prompt or model version alongside the production version and compares outputs side-by-side without exposing the new version to users. This is the AI engineering equivalent of a canary deployment and is essential before any production promotion.

16. Guardrails, Security, Privacy, and Governance

The three threat surfaces

Huyen identifies three distinct threat surfaces for AI applications that each require different mitigations:

Input threats: malicious or harmful content submitted by users — prompt injection, jailbreaks, requests for prohibited content, PII submission.

Process threats: the model taking unintended actions — calling tools it should not have called, accessing data outside its authorized scope, being manipulated by injected content in retrieved documents.

Output threats: the model producing harmful, incorrect, or policy-violating outputs — discriminatory text, legally problematic advice, confidential data leakage, misinformation.

Input/output guardrails

A guardrail is any component that intercepts and validates inputs or outputs before they flow to the model or to the user. The two dominant patterns:

Classifier-based guardrails: a secondary model (often a fine-tuned classifier) evaluates whether the input or output violates policy. Fast and scalable. Providers offer this as a managed service (Llama Guard, Azure Content Safety, OpenAI moderation endpoint, Anthropic's moderation API).

LLM-based guardrails: use a second LLM call to evaluate whether the content is safe. More flexible and capable than classifiers, but adds latency and cost.

For regulated industries, a layered guardrail architecture is appropriate:

Privacy and data governance

In regulated industries (finance, healthcare, insurance), the data governance requirements are often stricter than the AI-specific concerns. Key considerations:

• Data residency: where are inference requests and logs stored? Cross-border data transfer restrictions apply.
• PII in prompts: production logs frequently contain PII extracted from user inputs. Log scrubbing or pseudonymization at ingestion is required.
• Vendor agreements: what does your model provider's terms of service say about data retention, training on customer inputs, and audit access?
• Access controls: who within your organization can see the LLM call logs? These logs contain everything the model was told and everything it said — treat them as sensitive data.
• Right to deletion: GDPR and similar regulations create rights to delete personal data. Logs containing PII must support deletion.

Responsible deployment checklist

• Is every AI-generated output clearly labeled as AI-generated where disclosure is required?
• Are human-in-the-loop checkpoints in place for high-stakes or irreversible decisions?
• Is there a documented incident response plan for model misbehavior?
• Are bias and fairness evaluations included in the eval suite for any use case that makes decisions affecting people?
• Is there a model card or transparency report for any externally facing AI system?

17. Designing an End-to-End Production AI Application

A worked example: insurance claims triage

To make the preceding sections concrete, consider a production use case: an AI-powered claims triage system for an insurance company. A claims handler submits a new claim, and the system:

1. Extracts structured data from the free-text description
2. Retrieves relevant policy clauses and historical claim precedents
3. Classifies the claim type and estimates severity
4. Generates a recommended action for the claims handler
5. Produces an audit trail of the reasoning

This is a multi-step, RAG-augmented, tool-using application with strict accuracy and auditability requirements. Here is the architecture:

Each step is a distinct engineering component with its own evaluation criteria:

The prototype-to-production journey

Most teams move through four phases:

Phase 1 — Prototype (days to weeks): single LLM call, no retrieval, fixed prompt, tested on 10-20 examples. Goal: prove the concept is viable.

Phase 2 — Alpha (weeks): add retrieval, multi-step pipeline, basic error handling. Eval suite of 50-100 golden examples. Goal: identify the main failure modes.

Phase 3 — Beta (months): guardrails, observability, cost controls, shadow evaluation, human review loop. Eval suite of 500+ examples with automated regression testing. Goal: understand production behavior before scale.

Phase 4 — Production (ongoing): automated retraining feedback loop, active monitoring, incident response, model upgrade management. Goal: improve over time without regressing.

18. What Has Changed Since the Book Was Written

Chip Huyen's AI Engineering was published in early 2025. Several developments in the 18 months since have materially changed the landscape:

Extended thinking models

Models with explicit chain-of-thought reasoning (o-series from OpenAI, Claude Sonnet and Opus with extended thinking) have become mainstream. These models allocate variable compute to thinking steps before producing output. For complex reasoning tasks, they significantly outperform standard models. For simple tasks, they add latency and cost without benefit. Application developers now need to choose not just which model but which reasoning mode.

Multimodal by default

Vision, audio, and document understanding capabilities are now standard across all frontier models. Applications that once required specialized OCR pipelines or vision models can now process document images directly. This changes RAG architecture: chunking a PDF often means extracting images of tables and charts, not just text.

Long-context economics

In 2024, using a 1M token context window was prohibitively expensive. By mid-2026, Gemini 1.5 Pro's 1M token window costs roughly $1.25 per call for 500K tokens of input. This is still expensive at volume but economically viable for many document-heavy use cases that previously required complex retrieval pipelines.

Memory and persistence APIs

Provider-managed memory APIs (Anthropic's upcoming memory feature, OpenAI's memory system) are emerging. These shift some state management responsibility to the provider, reducing engineering complexity but introducing vendor dependency for a core feature.

Agentic frameworks maturation

LangGraph, CrewAI, Autogen, and similar frameworks have matured significantly. The ecosystem is still fragmented but stabilizing around a few patterns: graph-based state machines for deterministic agent control, and agent-as-a-service patterns for managed long-running agents.

Model cost convergence

The cost gap between frontier models and capable mid-tier models has narrowed. Claude Haiku and GPT-4o-mini now perform at levels that were GPT-4-class 18 months ago, at 10× lower cost. The economic case for expensive frontier models is increasingly limited to genuinely complex reasoning tasks.

19. Prototype-to-Production Maturity Model

A practical maturity model for teams building AI applications:

Most teams in 2026 are operating at L1 or L2. The gap between L2 and L4 is primarily engineering discipline, not AI capability — nothing in this model requires a more capable model than what already exists.

Conclusion

The central argument of this guide, and of Huyen's book before it, is that foundation models compressed the modeling problem and expanded the systems problem. A single API call to a frontier model can now do what used to require a dedicated ML team and a multi-month training cycle. But that same API call, dropped into a production system without context engineering, retrieval, evaluation, guardrails, and monitoring, will fail in ways that are hard to detect and expensive to fix. The skill that separates teams that ship durable AI products from teams that ship demos is not prompt cleverness. It is the unglamorous engineering discipline surveyed in the sections above: building a golden dataset before the first line of production code, treating every external input as untrusted, measuring the tail of the output distribution rather than the median, and building the feedback loop that lets a system improve after it ships rather than only before.

None of this is a reason for pessimism about the technology. It is a reason to apply the same rigor to AI systems that a mature engineering organization already applies to distributed systems, security, and data pipelines. The models will keep improving on a timeline outside any single team's control. What is inside a team's control is whether the system around the model is observable, evaluable, and safe to change. That is the actual work of AI engineering, and it is where teams should spend the majority of their effort.

If this guide was useful, the original book covers the material in more depth and with more nuance than is possible here: Chip Huyen, AI Engineering: Building Applications with Foundation Models (O'Reilly, 2025).

Checklists

Architecture checklist

• Context construction is explicit and versioned
• All external inputs are treated as untrusted
• Structural delimiters separate system instructions from retrieved content
• Model calls are wrapped in retry logic with exponential backoff
• Response parsing handles malformed outputs gracefully
• Tool implementations are idempotent or guarded with confirmation steps
• Context window usage is monitored and budgeted
• Prompt versions are tracked alongside code versions

Evaluation checklist

• Golden dataset of 100+ examples with expert-verified answers
• Code-based evals run in CI on every prompt change
• LLM-as-a-Judge calibrated against golden dataset (>80% agreement)
• Evaluation results compared across model and prompt versions
• Separate eval sets for each capability dimension (accuracy, safety, format)
• Regression tests for every known failure mode
• Shadow evaluation run before any production promotion

Security checklist

• Input sanitization and injection detection
• Output content policy enforcement
• PII detection and scrubbing in logs
• Principle of least privilege applied to all agent tools
• Tool calls require explicit confirmation for irreversible actions
• Vendor DPA and data processing agreement in place
• Penetration testing includes prompt injection scenarios

Deployment checklist

• Shadow evaluation validated against 1000+ real requests before promotion
• Rollout plan includes percentage-based traffic splitting
• Rollback plan documented and tested
• Cost alerting configured for unexpected usage spikes
• Latency SLOs defined and alerting configured
• On-call runbook for LLM API outages includes fallback behavior

Monitoring checklist

• Full request traces logged (input, output, model, latency, cost)
• Output format compliance rate tracked
• Refusal rate tracked and alerted on unexpected changes
• User satisfaction proxy tracked (thumbs, escalation rate)
• Daily cost report and budget alerting
• Weekly eval suite run with results trended over time
• PII scrubbing in logs verified on sample basis

Key Takeaways

1. Context is the primary lever. The quality of your AI application is determined more by what you put in the context than by which model you call. Invest heavily in context engineering.

2. Build evaluation before you build the product. A golden dataset and automated eval pipeline are infrastructure, not overhead. Without them, you are shipping blind.

3. Start with prompting. Most use cases can be solved well enough with a well-engineered prompt. Add RAG when prompting is insufficient due to knowledge gaps. Consider fine-tuning only after both are exhausted.

4. Treat every external input as untrusted. Prompt injection is real and well-demonstrated against production systems. Architectural defense (delimiters, scrubbing, output monitoring) is required, not optional.

5. The LLM is the easy part. Swapping models is a one-line change. Retrieval quality, evaluation methodology, guardrail coverage, and data governance are where the hard, durable engineering work lives.

6. Reliability is a distribution problem. A system that works 95% of the time on demos may work 80% of the time on production inputs. Measure the tail — that is where the business risk lives.

7. Cost compounds faster than you expect. A system that costs $0.01 per call sounds cheap until it is processing 50,000 calls per day. Model routing, prompt caching, and batch processing are not premature optimization at scale.

8. Agents require more engineering discipline, not less. Agentic applications are not easier than traditional applications. They add failure modes (loops, cascading errors, side effects) while removing the determinism that makes traditional software easy to reason about.

9. Human feedback is irreplaceable. Automated evals improve iteration speed, but ground truth always requires human judgment. Build lightweight human review into your workflow from the start.

10. Governance is an engineering problem. Guardrails, audit trails, explainability, and incident response plans are not bureaucratic overhead — they are the engineering work that makes regulated and high-stakes deployment possible.

30-Day Learning Plan

This plan assumes you have software engineering background and are new to production AI engineering.

Week 1: Foundations

• Day 1–2: Read Chapters 1–3 of Huyen's AI Engineering. Build a simple chatbot with the Anthropic or OpenAI SDK.
• Day 3–4: Build your first structured output pipeline. Add JSON schema validation to the output.
• Day 5–7: Build a minimal RAG pipeline using pgvector or Chroma. Evaluate retrieval quality manually.

Week 2: Evaluation and Quality

• Day 8–9: Build a 50-case golden dataset for your use case. Write code-based evals.
• Day 10–11: Implement an LLM-as-a-Judge. Calibrate it against your golden dataset.
• Day 12–14: Run your eval suite on three different models. Document the trade-offs.

Week 3: Agents and Scale

• Day 15–17: Build a simple tool-using agent. Add retry logic and error handling.
• Day 18–19: Implement structured logging with prompt version tracking.
• Day 20–21: Add a basic guardrail using a classifier or a secondary LLM check.

Week 4: Production Engineering

• Day 22–23: Implement a model cascade for cost reduction. Measure cost before/after.
• Day 24–25: Build a shadow evaluation pipeline.
• Day 26–28: Add production monitoring dashboards (latency, cost, refusal rate, user satisfaction).
• Day 29–30: Conduct a security review against the security checklist. Identify and fix the top three gaps.

Further Reading

• Chip Huyen, AI Engineering: Building Applications with Foundation Models (O'Reilly, 2025) — the primary source for this guide. Purchase here.
• Anthropic, Claude API documentation — reference for model capabilities, tool use, and prompt caching
• OpenAI, Platform documentation — reference for structured outputs, function calling, and batch APIs
• MTEB Leaderboard — the standard benchmark for embedding model evaluation
• LangGraph documentation — reference for graph-based agent workflows
• Hugging Face PEFT library — reference for parameter-efficient fine-tuning
• Langfuse — open-source LLM observability and evaluation platform

References

1. Huyen, C. (2025). AI Engineering: Building Applications with Foundation Models. O'Reilly Media. ISBN 978-1-098-16629-8.

2. Bommasani, R., et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford HAI. https://arxiv.org/abs/2108.07258

3. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401

4. Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. https://arxiv.org/abs/2210.03629

5. Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. https://arxiv.org/abs/2307.03172

6. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. https://arxiv.org/abs/2306.05685

7. Muennighoff, N., et al. (2023). MTEB: Massive Text Embedding Benchmark. https://arxiv.org/abs/2210.07316

8. Anthropic. (2024). Claude's Constitution. anthropic.com/research/claude-s-constitution

9. Gao, L., et al. (2023). Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). https://arxiv.org/abs/2212.10496

10. Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.

11. Anthropic Developer Documentation. (2025). Prompt Caching. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching

12. Perez, F., & Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques for Language Models. https://arxiv.org/abs/2211.09527