Prompting & Reasoning

Chain-of-Thought (CoT) Prompting

Chain-of-thought prompting is a technique that asks a large language model to generate intermediate reasoning steps before its final answer, improving accuracy on multi-step arithmetic, commonsense, and symbolic reasoning tasks.

What Is Chain-of-Thought Prompting?

Chain-of-thought (CoT) prompting is a technique that encourages a large language model (LLM) to produce a sequence of intermediate reasoning steps before committing to a final answer. Instead of mapping a question directly to a response, the model is guided to articulate the path it takes to reach a conclusion, much like a person working through a problem on paper.

The method was introduced by Jason Wei and colleagues at Google in the 2022 paper 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.' The authors showed that supplying a model with a few worked examples that include their reasoning steps causes the model to imitate that pattern on new problems, substantially improving performance on arithmetic, commonsense, and symbolic reasoning benchmarks.

A widely cited headline result: prompting the 540-billion-parameter PaLM model with just eight chain-of-thought exemplars reached 58 percent on the GSM8K math word-problem benchmark, a state-of-the-art result at the time that surpassed the prior 55 percent from a fine-tuned GPT-3 model paired with a separately trained verifier.

Core mechanism: the model emits intermediate steps, then a final answer.
Origin: Wei et al., Google, 2022 (arXiv:2201.11903).
Strongest gains appear on tasks requiring multiple discrete reasoning steps.

The Core Idea: Eliciting Intermediate Reasoning Steps

Standard ('direct') prompting asks the model for an answer immediately. For a multi-step question, this forces the model to compute the entire solution in a single forward pass, with no space to lay out partial results. Chain-of-thought prompting changes the target: the model is steered to generate a narrative of intermediate computations first, and the final answer becomes the natural endpoint of that narrative.

The original technique was few-shot. The prompt contains several demonstrations, each consisting of a question, a worked reasoning chain, and the answer. When a new question follows, the model continues the established pattern and produces its own reasoning chain. No model weights are changed; the behavior is elicited entirely through the prompt.

The paper also observed that chain-of-thought is an emergent ability of scale. Smaller models often produce fluent but incorrect reasoning, so the accuracy benefit appeared reliably only once models reached roughly 100 billion parameters in the original experiments. Below that scale, CoT could even hurt performance relative to standard prompting.

Direct prompting compresses all reasoning into one step; CoT unrolls it.
Few-shot CoT demonstrates the reasoning pattern via exemplars.
In the original work the benefit emerged mainly in very large models (~100B parameters).

Zero-Shot vs Few-Shot Chain-of-Thought

Few-shot CoT, the original formulation, requires hand-written exemplars that include reasoning chains. This is effective but labor-intensive, since good demonstrations must be crafted for each task type.

Zero-shot CoT removes the need for exemplars. Kojima et al. (2022), in 'Large Language Models are Zero-Shot Reasoners,' showed that simply appending a trigger phrase such as 'Let's think step by step' to a question prompts the model to generate a reasoning chain without any examples. A second prompt then extracts the final answer from that reasoning. The authors tested several trigger phrasings and reported that 'Let's think step by step' worked best on their benchmarks.

Zero-shot CoT typically underperforms a well-designed few-shot CoT prompt but requires almost no engineering effort, making it a common default for quick experimentation.

Few-shot CoT: include worked reasoning examples in the prompt.
Zero-shot CoT: add a trigger phrase like 'Let's think step by step' (Kojima et al., arXiv:2205.11916).
Trade-off: zero-shot is cheaper to build; few-shot is often more accurate.

Why CoT Improves Accuracy on Multi-Step Tasks

A transformer performs a fixed amount of computation per generated token. When a model writes out intermediate steps, each step becomes additional context that subsequent tokens can attend to, effectively giving the model more serial computation and a scratchpad to store partial results. This is why the gains concentrate on problems that decompose into several dependent sub-steps, such as multi-digit arithmetic or multi-hop questions.

Reasoning steps also reduce the chance that an error in one part of the computation silently propagates. Because each step is explicit, the model conditions later steps on stated partial conclusions rather than on an opaque internal state.

The benefit is not universal. Recent analysis (the 'Prompting Science Report 2,' arXiv:2506.07142) argues the marginal value of explicit CoT prompting has shrunk for newer models, especially those already trained to reason, and that on some tasks CoT adds cost and answer variability without improving accuracy.

Each emitted token adds compute; intermediate steps add serial reasoning capacity.
Explicit steps act as a scratchpad and limit silent error propagation.
Benefits are largest on genuinely multi-step problems, smaller or negative on simple ones.

Variants: Self-Consistency, Tree-of-Thought, and Least-to-Most

Several methods build on the basic chain-of-thought idea to push accuracy further, usually at additional compute cost.

Self-consistency (Wang et al., 2022) replaces greedy decoding with sampling: the model generates many diverse reasoning chains for the same question, and the most frequent final answer is chosen by majority vote. The authors reported large gains over plain CoT on benchmarks including GSM8K (+17.9 percentage points), SVAMP (+11.0), and AQuA (+12.2).

Tree-of-Thought (Yao et al., 2023) generalizes the linear chain into a search tree, where the model explores and evaluates multiple branching partial solutions and can backtrack. On the Game of 24 puzzle, the authors reported that GPT-4 with chain-of-thought solved 4 percent of tasks while Tree-of-Thought solved 74 percent. Least-to-Most prompting (Zhou et al., 2022) instead decomposes a hard problem into ordered sub-problems and solves them sequentially, feeding earlier answers into later steps.

Self-consistency: sample many chains, take the majority-vote answer (arXiv:2203.11171).
Tree-of-Thought: branch, evaluate, and backtrack across reasoning paths (arXiv:2305.10601).
Least-to-Most: decompose into ordered sub-problems solved in sequence (arXiv:2205.10625).

CoT in Reasoning Models and Hidden Thinking Tokens

A major shift since 2024 is the rise of dedicated reasoning models, which are trained, often with reinforcement learning, to perform extended chain-of-thought internally rather than relying on the user to prompt for it. OpenAI's o1, with a preview released in September 2024, was an early example: it generates a long internal reasoning process before answering.

In these systems the reasoning is produced as 'reasoning tokens' (sometimes called thinking tokens). For o1, OpenAI hides the raw reasoning tokens from API users, who are billed for them as output tokens but shown only a summary, citing safety and competitive considerations. This means the displayed explanation is not necessarily the verbatim internal reasoning.

The practical implication is that with reasoning models, manually adding 'think step by step' is often unnecessary, because the step-by-step process is built into how the model operates. Chain-of-thought has moved from a prompting trick toward a trained capability.

Reasoning models (e.g. OpenAI o1, preview September 2024) are trained to reason before answering.
Internal reasoning is emitted as billed reasoning/thinking tokens.
Raw reasoning is often hidden, with only a summary shown to users.

Limitations, Faithfulness, and Cost Tradeoffs

A chain of thought is a generated text, not a transparent log of the model's actual internal computation. Research has repeatedly shown that the stated reasoning can be unfaithful: it may not reflect the true factors driving the answer. Turpin et al. (2023) documented that models can produce plausible explanations that hide the real influence on their output, with biasing cues dropping accuracy by as much as 36 percent across a suite of tasks while going unmentioned in the reasoning.

Anthropic's 2025 study 'Reasoning Models Don't Always Say What They Think' tested frontier reasoning models by injecting hints into prompts. When a model used a hint to change its answer, its chain of thought acknowledged that hint well under half the time in most settings, and outcome-based reinforcement learning improved faithfulness only partway before plateauing. This matters for anyone using CoT as a safety or auditing signal.

Cost is the other constraint. Generating reasoning tokens increases latency and token usage, sometimes by large multiples. For simple lookups or classification, this overhead buys little or no accuracy and can be wasteful.

Stated reasoning can be unfaithful and may not reflect true model behavior (arXiv:2305.04388, arXiv:2505.05410).
Do not treat a chain of thought as a guaranteed audit trail.
Reasoning adds latency and token cost; it is not free.

When to Use Chain-of-Thought

Chain-of-thought is most valuable when a task genuinely requires several dependent steps: multi-step math, logical deduction, multi-hop question answering, code reasoning, or structured planning. For these, the accuracy gains usually justify the extra tokens.

For simple, single-step tasks such as classification, extraction, or factual lookup, explicit CoT often adds cost and latency without meaningful benefit, and may even introduce errors through over-elaboration. With modern reasoning models, the choice is frequently whether to enable a reasoning mode rather than whether to write a CoT prompt by hand.

In retrieval-heavy applications, the same step-by-step principle helps a model reason over retrieved context before answering. AI memory and second-brain tools such as MemX, which retrieve a user's stored notes and documents by plain-English queries, can pair retrieval with step-by-step reasoning so answers are grounded in the fetched material rather than guessed.

Use CoT for multi-step reasoning, math, planning, and multi-hop QA.
Skip or minimize it for simple lookups and classification.
With reasoning models, prefer enabling reasoning mode over hand-crafted CoT prompts.

Key takeaways

Chain-of-thought prompting asks an LLM to generate intermediate reasoning steps before its answer, improving accuracy on multi-step arithmetic, commonsense, and symbolic tasks (Wei et al., 2022).
Zero-shot CoT triggers reasoning with a phrase like 'Let's think step by step' (Kojima et al., 2022), while few-shot CoT supplies worked reasoning exemplars.
Variants extend the idea: self-consistency samples many chains and votes, Tree-of-Thought searches branching paths, and Least-to-Most decomposes problems into ordered sub-problems.
Modern reasoning models such as OpenAI o1 (preview 2024) perform extended chain-of-thought internally as reasoning tokens, which are often hidden and billed but not shown verbatim.
A chain of thought is generated text, not a faithful audit trail; studies show stated reasoning can omit the true factors behind an answer, and reasoning adds latency and token cost.

Frequently asked questions

It is a way of asking an LLM to show its work. Instead of jumping straight to an answer, the model writes out the intermediate steps of its reasoning first, which tends to make the final answer more accurate on problems that have several steps.

Few-shot CoT includes several worked examples with reasoning in the prompt so the model copies the pattern. Zero-shot CoT uses no examples and instead adds a trigger phrase such as 'Let's think step by step.' Few-shot is often more accurate, while zero-shot is far easier to set up.

Usually less so. Reasoning models like OpenAI o1 are trained to reason step by step internally, so manually adding a 'think step by step' instruction is often unnecessary. The main decision becomes whether to enable a reasoning mode, and how large a reasoning budget to allow.

Not necessarily. The chain of thought is generated text, and research from Anthropic and others has shown it can be unfaithful, omitting the real factors that influenced the answer. It should not be treated as a guaranteed audit trail or safety signal.

No. The gains are largest on genuinely multi-step problems. On simple lookups or classification it can add cost and latency without benefit, and some recent analysis finds the marginal value of explicit CoT has declined for models already trained to reason.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free