Numbers and architecture decisions in this post reflect MemX's production pipeline as of May 2026.
MemX ran a benchmark in March 2026 with an unexpected result. A cheaper model (Gemini 2.5 Flash-Lite at $0.10/M input) routed through two sequential prompts beat the more expensive model (Gemini 2.5 Flash at $0.30/M input) routed through one big prompt. The cheap model won on every axis. Higher accuracy (99.4% vs 96.1%). Lower cost ($0.000569/case vs $0.00155/case, 2.7x cheaper). Same latency budget.
The reason was not the model. The reason was the architecture. Splitting one prompt into two specialist prompts (Stage 1 classifies the intent, Stage 2 executes the action) outperforms cramming everything into a monolith. This is not novel. Anthropic published it as a workflow pattern in December 2024. Khot et al. proved it at ICLR 2023. Almost nobody actually does it.
Quick takeaways: two-stage prompting splits one big prompt into two focused calls. Stage 1 classifies intent. Stage 2 runs the action-specific prompt. The classic academic lineage is Wei et al. (chain-of-thought, NeurIPS 2022), Zhou et al. (least-to-most, ICLR 2023), Khot et al. (decomposed prompting, ICLR 2023). The production lineage is Perplexity, Cursor, Cody, and Anthropic's own agent playbook. The cost math is counterintuitive: two calls can be cheaper than one because each call's prompt is shorter and more cacheable.
What two-stage actually means
Stage 1 is a classifier. Its job is to look at the user input and decide which action category this belongs to (search, save, modify, chat, multi-action). Output is a tiny structured JSON: `{intent: "search", confidence: 0.94}`. Stage 2 is a specialist. Given Stage 1's output, it loads the action-specific prompt (the one written specifically for search, with only search-relevant instructions, examples, and tool definitions) and executes.
Distinguish from chain-of-thought (one call, model expands its own reasoning trace inside the prompt). Distinguish from multi-agent (multiple autonomous loops running in parallel). Two-stage is deterministic, sequential, and Stage 2 reads Stage 1's full structured output. That last detail is important; it neutralises the most common objection.
The academic lineage
Wei et al.'s 2022 chain-of-thought paper got the attention. The architectural work that matters more for two-stage is Zhou et al.'s least-to-most prompting (Google, ICLR 2023) and Khot et al.'s decomposed prompting (DecomP, ICLR 2023). Both papers showed empirically that splitting a hard task into focused sub-tasks delegated separately outperforms monolithic prompting, often by margins large enough to bridge whole model generations.
Schick et al.'s Toolformer (NeurIPS 2023 oral) extended the same logic to tool selection: deciding which API to call is a separate decision from formatting the call. DSPy from Stanford NLP made the whole pattern declarative: write modules and signatures, compile multi-stage programs via optimisers like MIPROv2 or BootstrapFewShot. The throughline across 4 years of academic work: small, focused prompts compose better than monoliths.
The production lineage
Anthropic's December 2024 "Building Effective Agents" post lists five composable workflow patterns. Routing is one of them. Anthropic defines it precisely: "Routing classifies an input and directs it to a specialized followup task." The post recommends routing for "complex tasks where there are distinct categories that are better handled separately, and where classification can be handled accurately," noting that "without this workflow, optimizing for one kind of input can hurt performance on other inputs." That is a direct restatement of the two-stage thesis with corporate framing.
Look at any production LLM system in 2026 and you will find the same pattern. Perplexity runs a six-stage answer pipeline documented by Perplexity Research: query intent parsing, real-time web retrieval with hybrid BM25-plus-dense embeddings, multi-layer ML reranking, prefiltering, structured prompt assembly with pre-embedded citations, and LLM synthesis constrained by retrieved evidence. Cursor runs tree-sitter code parsing, Merkle-tree file sync, vector retrieval over Turbopuffer, `.cursorrules` system-prompt wrapping, then inference and streaming diff. Sourcegraph Cody runs a hybrid graph-plus-embedding retrieval before generation. The systems that scaled are all multi-stage.
Why two API calls can cost less than one
Two API calls intuitively sound more expensive than one. They are not. The mechanism is three-part.
- Stage 1 is highly cacheable. Anthropic prompt caching bills cached input at 10% of standard price (90% off). OpenAI bills cached input at 50% (50% off). Gemini 2.5 implicit caching is 90% off on 2.5-series models (cached input at 10% of standard) as of May 2026, with the older 75% rate now reserved for Gemini 2.0. The Stage 1 classifier prompt is mostly stable across requests, so the prefix lands in cache and stays there. Stage 1 effective cost approaches zero.
- Stage 2 prompts are narrower. The monolithic prompt enumerates every possible action's instructions ("if save, do X; if search, do Y; if chat, do Z"). Stage 2 loads only the prompt for the chosen action, which is shorter, more focused, and more accurate.
- Output tokens shrink. Output tokens cost 3 to 6 times more than input tokens. A focused Stage 2 prompt generates a focused output instead of meandering through irrelevant options.
Net effect for the MemX pipeline: Stage 1 plus Stage 2 with Flash-Lite ($0.000569 per case) is 2.7x cheaper than a single-prompt call with Flash ($0.00155 per case). The cheaper model in two stages beat the more expensive model in one. The win was not the model. The win was the architecture.
The decision matrix: when to use two-stage
| Scenario | Single-prompt | Two-stage |
|---|---|---|
| Intent classification + action execution (3+ mutually exclusive actions) | weak | wins |
| Free-form creative writing or brainstorming | wins | weak |
| Multi-turn conversation with shared state | wins | weak |
| Document extraction with known schema, short input | tie | mild win |
| RAG over heterogeneous corpora (route then retrieve then answer) | weak | wins |
| Sub-500ms hard latency budget, single network hop | wins | weak |
| Cheap model + complex task (rescue scenario) | weak | wins |
The rescue scenario is the interesting one. Two-stage rescued Gemini 3 Flash on our classification benchmark from 73.4% single-prompt to 95.0% two-stage; a +21.6 percentage point swing on the same model. If your accuracy is below your threshold, the architectural fix usually beats the model upgrade.
When NOT to use two-stage
Five scenarios where the architecture costs more than it pays.
- Task is intrinsically holistic. Creative writing, dialogue, prose. Splitting hurts.
- State cannot be cleanly serialised between stages. If Stage 1 has to pass long context to Stage 2, the cost savings evaporate.
- Latency budget is sub-500ms hard cap. Two sequential network calls plus model inference is not going to fit.
- Frontier reasoning model with thinking-by-default. Gemini 3 Pro and similar models already decompose internally; an external two-stage wrapper adds latency without lift.
- Routing accuracy is below ~95%. If Stage 1 misroutes 8% of requests, the misroute tax exceeds the specialisation gain. Improve Stage 1 first.
The Cognition counterargument, neutralised
Walden Yan at Cognition published a widely-shared essay on 12 June 2025 titled "Don't Build Multi-Agents." The two principles he names are "Share context, and share full agent traces, not just individual messages" and "Actions carry implicit decisions, and conflicting decisions carry bad results." The thrust: agent-level fan-out fragments context and produces inconsistent decisions. He is right. He is also describing a different architecture from the one this post recommends.
Multi-agent means parallel autonomous loops making independent decisions with partial context. Two-stage means a deterministic sequential pipeline where Stage 2 inherits Stage 1's full structured output. Two-stage actually satisfies both of Yan's principles. The full Stage 1 trace is shared, not summarised. The decisions do not conflict because Stage 1 routes, Stage 2 executes, and only one path runs per request. When the post crosses your feed, do not let the headline scare you off the pattern.
How to migrate a single-prompt system to two-stage
- Identify the routing dimension. What is the smallest decision you can make first that lets you load a more focused prompt second? Usually intent / action / category.
- Write the Stage 1 classifier prompt. Tiny. Output a structured JSON with one field (intent) plus a confidence score. Use structured outputs if your provider supports them.
- Split the monolithic prompt by intent. Each intent gets its own Stage 2 prompt with only the relevant instructions, examples, and tool definitions.
- Measure flap rate at Stage 1. If Stage 1 routes inconsistently, the whole pipeline is unreliable. Get to 99%+ Stage 1 accuracy before optimising Stage 2.
- Cache the Stage 1 prefix. Whatever your provider charges for cached input, that is the marginal cost of Stage 1 at steady state.
- Add a fallback path. If Stage 1 outputs `unknown` or `low_confidence`, route to a generalist Stage 2 prompt rather than failing.
Specialisation beats capacity. At the model layer (Mixture of Experts), at the prompt layer (two-stage), and at the engineering layer (small focused services). The MoE insight applied to prompt design is the architecture that quietly won production LLM systems.
Key takeaway: the model is the engine; the prompt is the chassis. Tune the chassis first. Cheap model in two stages beat expensive model in one for us, by 3.3 percentage points on accuracy and 2.7x on cost. If you are choosing between a pricier model and a sharper prompt, sharpen the prompt.
01What is two-stage LLM prompting?
A pipeline where one focused prompt classifies the intent of a user request, and a second prompt (specific to that intent) executes the action. Stage 1 is a classifier. Stage 2 is a specialist. Sequential, deterministic, with Stage 2 reading Stage 1's full structured output.
02Is two-stage prompting the same as prompt chaining?
Two-stage is a specific shape of prompt chaining where the first stage is a classifier and the second is an action specialist. Prompt chaining more generally covers any sequence of LLM calls where each call's output feeds the next. Anthropic's December 2024 "Building Effective Agents" post lists routing as one of five composable workflow patterns.
03Does two-stage prompting cost more because of two API calls?
Counterintuitively, no. Stage 1 is highly cacheable: Anthropic bills cached input at 10% of standard (90% off), OpenAI at 50% (50% off), Gemini 2.5 implicit caching at 90% off as of May 2026. Stage 2 prompts are shorter and more focused, so output tokens shrink. Our production pipeline saves 2.7x on cost vs single-prompt despite making two API calls per request.
04When should I use two-stage and when should I use chain-of-thought?
Two-stage is external decomposition (multiple API calls, deterministic routing). Chain-of-thought is internal decomposition (one API call, model expands its reasoning trace). They compose: Stage 2 can use chain-of-thought inside. Use two-stage when actions are mutually exclusive and need different prompts. Use chain-of-thought for tasks that benefit from explicit reasoning steps.
05Does Cognition's "Don't Build Multi-Agents" essay apply to two-stage prompting?
No. Walden Yan's critique targets parallel autonomous agents making independent decisions with fragmented context. Two-stage is sequential and deterministic; Stage 2 inherits Stage 1's full structured output. Two-stage satisfies both of his principles: share full traces, and avoid conflicting decisions. Different architecture, different conclusion.
06What is the easiest way to test if two-stage is right for my system?
Look at your single-prompt's failure modes. If failures cluster on specific action types (search queries misclassified as saves, for example), two-stage will likely help. If failures are spread evenly across actions, the bottleneck is model quality, not prompt structure. Measure routing accuracy first; if you can hit 99%+ at Stage 1, the rest of the pipeline will benefit from specialisation.
07What frameworks help build two-stage pipelines in 2026?
DSPy (Stanford NLP, active as of May 2026) lets you declare modules and compile multi-stage programs via optimisers like MIPROv2 and BootstrapFewShot. LangGraph handles the routing graph if you want explicit state machines. Most production teams ship two-stage as 60 lines of plain Python plus structured outputs from their provider; the framework is optional.
