Models & Evaluation

DeepSeek Architecture

DeepSeek is a Chinese AI lab whose open-weight models combine Multi-head Latent Attention (MLA) for a compressed KV cache, a fine-grained DeepSeekMoE mixture-of-experts design, and reinforcement-learning-trained reasoning, reaching frontier quality at low reported training compute.

What DeepSeek Is

DeepSeek is a Chinese artificial intelligence lab that develops large language models and releases many of them with open weights. The company spun off from High-Flyer, a quantitative hedge fund co-founded by Liang Wenfeng, and became an independent entity in July 2023. Its stated focus is efficient model architecture and training rather than raw spending, a direction visible across every model it has shipped.

The lab is best known for a sequence of mixture-of-experts (MoE) models. DeepSeek-V2 (arXiv:2405.04434, May 2024) introduced the two architectural ideas the lab is now identified with: Multi-head Latent Attention and the DeepSeekMoE expert layout. DeepSeek-V3 (arXiv:2412.19437, December 2024) scaled those ideas to 671 billion total parameters, and DeepSeek-R1 (arXiv:2501.12948, January 2025) added reinforcement-learning-trained reasoning on top of the V3 base.

What set DeepSeek apart from most frontier labs was the combination of competitive benchmark results, open weights, and unusually low reported training compute. That combination is the reason the architecture attracts technical study: it is a worked example of how careful design choices in attention, expert routing, and numerical precision can lower the cost of training and serving a frontier-scale model.

DeepSeek is an AI lab, not a single model; its model families are V2, V3, and R1, with later iterations through V3.2 and V4.
It originated from the quantitative hedge fund High-Flyer and operates as an independent company since July 2023.
Its signature techniques are Multi-head Latent Attention (MLA) and the DeepSeekMoE mixture-of-experts design.
Many DeepSeek model weights are released under permissive licenses such as MIT, which is why third parties can run and fine-tune them.

Multi-head Latent Attention (MLA)

Standard transformer attention stores a key vector and a value vector for every token and every attention head. During generation those stored vectors form the KV cache, and its size grows with sequence length and head count, which makes memory the binding constraint for long-context inference. Multi-head Latent Attention attacks this directly by compressing the keys and values into a single low-rank latent vector per token, then reconstructing the per-head keys and values on the fly.

Concretely, MLA projects the token representation down into a small latent vector and caches only that latent vector instead of the full set of keys and values. At attention time the cached latent is projected back up to recover the keys and values for each head. Because the cached object is much smaller than the full KV set, the memory footprint drops sharply. The DeepSeek-V2 report measured a reduction in KV cache of about 93.3 percent versus the prior dense DeepSeek 67B model, along with higher maximum generation throughput.

The design trades a small amount of extra arithmetic, the up-projection at inference, for a large memory saving, and the V2 paper reported that MLA matched or exceeded standard multi-head attention on quality. This memory reduction is the foundation that makes the long context windows and high serving throughput of later DeepSeek models practical.

c_t = W_DKV · h_t      (compress to latent, cache c_t)
k_t = W_UK · c_t,   v_t = W_UV · c_t      (reconstruct per-head K, V)

MLA low-rank KV compression: each token's hidden state h_t is projected down into a compact latent c_t (the only thing cached), and the per-head keys k_t and values v_t are recovered by up-projection matrices W_UK and W_UV at attention time.

MLA caches one compressed latent vector per token rather than full per-head key and value vectors.
Keys and values are reconstructed from the latent via an up-projection during attention, so the cache stays small.
DeepSeek-V2 reported roughly a 93 percent smaller KV cache than the comparable dense model.
The technique is the reason long-context inference on DeepSeek models stays memory-efficient.

DeepSeekMoE: Fine-Grained Experts

A mixture-of-experts layer replaces a single feed-forward network with many parallel expert networks plus a router that sends each token to only a few of them. This makes total parameter count large while keeping the per-token computation small, since only the selected experts run. DeepSeekMoE (arXiv:2401.06066) refines this idea with two specific strategies aimed at expert specialization.

The first strategy is fine-grained expert segmentation: each expert's feed-forward hidden dimension is split, producing a larger number of smaller experts at the same total parameter and compute budget. More, smaller experts give the router more flexible combinations to assign per token, which encourages each expert to specialize. The second strategy is shared expert isolation, where a few experts are always active for every token. These shared experts absorb common knowledge so the routed experts do not have to redundantly relearn it.

DeepSeek-V3 carries this layout to scale and pairs it with an auxiliary-loss-free load balancing strategy. Earlier MoE training typically added an auxiliary loss to keep token traffic spread evenly across experts, but that auxiliary term can degrade the main objective. DeepSeek-V3 instead adjusts a per-expert bias term to balance load without an explicit auxiliary loss, avoiding that quality penalty. V3 also adds a multi-token prediction objective during training for stronger performance.

y = Σ_i g_i(x) · E_i(x),   where g_i(x) ≠ 0 only for the top-k routed experts

MoE output: each expert E_i contributes weighted by its gate value g_i(x), but the gate is nonzero only for the top-k experts chosen by the router, so most experts do no work for a given token. DeepSeekMoE also adds always-on shared experts to this sum.

Fine-grained segmentation splits experts into many smaller units, giving the router more specialized combinations.
Shared experts run on every token to capture common knowledge and reduce redundancy among routed experts.
DeepSeek-V3 uses auxiliary-loss-free load balancing via a per-expert bias, avoiding the quality cost of an auxiliary loss.
V3 adds a multi-token prediction training objective alongside the MoE design.

DeepSeek-V3 and FP8 Training

DeepSeek-V3 is a mixture-of-experts model with 671 billion total parameters, of which about 37 billion are activated per token. It was pre-trained on 14.8 trillion tokens and combines the MLA attention and DeepSeekMoE layers validated in V2 with the auxiliary-loss-free balancing and multi-token prediction objectives described above. Its benchmark results placed it among the strongest open-weight models at release.

A central engineering choice in V3 is FP8 mixed-precision training. FP8 is an 8-bit floating-point format, and the report uses the E4M3 variant (4 exponent bits, 3 mantissa bits) with fine-grained tile-wise and block-wise scaling to keep numerical accuracy acceptable. Training in 8-bit rather than the more common 16-bit roughly halves memory and bandwidth pressure for the affected tensors, which is a large part of why the run was cheap relative to its scale.

The widely cited cost figure must be read carefully. The technical report states that DeepSeek-V3 used 2.788 million H800 GPU hours for full training, and that assuming a rental price of 2 dollars per GPU hour, total training costs amount to 5.576 million dollars. The report explicitly clarifies that this covers only the official training run and excludes the cost of prior research and ablation experiments on architectures, algorithms, or data. The figure is therefore the company's reported compute cost for the final run, not a total program cost or a fully loaded development budget.

DeepSeek-V3: 671B total parameters, about 37B activated per token, trained on 14.8T tokens.
It uses FP8 (E4M3) mixed-precision training with tile-wise and block-wise scaling to control accuracy.
Reported compute: 2.788M H800 GPU hours, or about 5.576M dollars at an assumed 2 dollars per GPU hour.
That figure is the final-run compute cost only; the report excludes prior research and ablation costs.

DeepSeek-R1 and Reasoning via RL

DeepSeek-R1, released in January 2025 (arXiv:2501.12948), is a reasoning model built on the DeepSeek-V3 base and trained largely with reinforcement learning rather than supervised demonstrations. The reinforcement-learning algorithm is Group Relative Policy Optimization (GRPO), which estimates the advantage of a response by comparing it against a group of sampled responses to the same prompt, removing the need for a separate value network.

The paper also presents DeepSeek-R1-Zero, a variant trained with pure reinforcement learning and no supervised fine-tuning stage at all. R1-Zero developed long chains of reasoning on its own, with reported AIME 2024 pass@1 rising from 15.6 percent to 71.0 percent over training, though it suffered from poor readability and language mixing. DeepSeek-R1 then adds a small amount of cold-start supervised data and a multi-stage pipeline to fix those issues while keeping the reasoning gains. The work was later published in Nature.

R1 was released with open weights under the MIT license. Its arrival in late January 2025 produced a sharp market reaction: a frontier-competitive reasoning model trained at low reported cost contradicted assumptions about how much compute leadership required, and on 27 January 2025 chip and AI stocks fell hard, with Nvidia recording one of the largest single-day market-value losses in U.S. stock market history. DeepSeek has continued to iterate since, with later releases including V3.1 (August 2025), V3.2 (December 2025), and the current DeepSeek-V4 generation (DeepSeek-V4-Pro and DeepSeek-V4-Flash, released April 24, 2026).

python

import torch
import torch.nn.functional as F

def moe_top_k(x, gate, experts, k):
    # x: [tokens, d_model]; gate: linear layer -> [tokens, n_experts]
    logits = gate(x)
    topk_val, topk_idx = logits.topk(k, dim=-1)      # pick k experts/token
    weights = F.softmax(topk_val, dim=-1)            # normalize over the k
    out = torch.zeros_like(x)
    for slot in range(k):
        idx = topk_idx[:, slot]                      # expert id per token
        w = weights[:, slot].unsqueeze(-1)
        for e in idx.unique():
            mask = idx == e
            out[mask] += w[mask] * experts[e](x[mask])
    return out

Minimal top-k MoE routing: the gate scores all experts, only the top-k run per token, and their outputs are combined with softmax-normalized weights. DeepSeekMoE uses many fine-grained experts plus always-on shared experts on top of this pattern.

DeepSeek-R1 is trained primarily with reinforcement learning (GRPO) on the V3 base to elicit reasoning.
DeepSeek-R1-Zero used pure RL with no supervised fine-tuning and self-developed long reasoning chains.
R1 shipped with open weights under the MIT license and was later published in Nature.
Its January 2025 release triggered a major market reaction, including a record single-day drop in Nvidia's value.

Why the Architecture Matters

The DeepSeek models are studied because they bundle several efficiency techniques that each address a different bottleneck. MLA targets inference memory through KV-cache compression. DeepSeekMoE targets training and inference compute by activating only a fraction of parameters per token. FP8 training targets memory bandwidth and throughput during the training run itself. Together they let a frontier-scale model train and serve at lower cost than its parameter count alone would suggest.

For practitioners, the open-weight releases matter as much as the architecture. Permissive licensing means the models can be self-hosted, fine-tuned, quantized, and evaluated without API restrictions, which is useful for teams that need data control or want to benchmark reasoning behavior directly. The R1 line in particular made open reasoning models widely available and accelerated outside research on reinforcement-learning recipes for reasoning.

The cost story should be cited with care. The often-repeated low dollar figure is the reported compute cost of a single final training run under an assumed GPU rental price, not the total cost of building the model. Treating it as a full development budget overstates the claim, which is precisely the kind of figure worth verifying before repeating.

Three techniques attack three bottlenecks: MLA for inference memory, MoE for compute, FP8 for training throughput.
Open weights enable self-hosting, fine-tuning, quantization, and independent evaluation.
The R1 release broadened access to open reasoning models and spurred outside RL-for-reasoning research.
Quote the cost figure as a single-run compute cost under an assumed GPU price, not a total program budget.

Key takeaways

DeepSeek is a Chinese AI lab whose open-weight models pair efficient architecture with low reported training compute.
Multi-head Latent Attention compresses keys and values into a small cached latent, cutting KV-cache memory by roughly 93 percent in DeepSeek-V2.
DeepSeekMoE uses fine-grained experts plus always-on shared experts, and DeepSeek-V3 adds auxiliary-loss-free load balancing.
DeepSeek-V3 has 671B total parameters with about 37B activated per token, trained on 14.8T tokens using FP8 mixed precision.
DeepSeek-R1 (January 2025) was trained largely with reinforcement learning (GRPO), including the pure-RL R1-Zero variant, and shipped with open weights.
The reported ~5.576M dollar figure is the final training run's compute cost under an assumed 2 dollars per GPU hour, not total development cost.

Frequently asked questions

DeepSeek is a Chinese AI lab that builds large language models, many released with open weights. It spun off from the quantitative hedge fund High-Flyer, co-founded by Liang Wenfeng, and became an independent company in July 2023. It is known for the V2, V3, and R1 model families.

MLA is an attention design that compresses each token's keys and values into a single low-rank latent vector, which is the only thing stored in the KV cache. The per-head keys and values are reconstructed on the fly during attention. DeepSeek-V2 reported this cut the KV cache by about 93 percent versus a comparable dense model, making long-context inference far more memory-efficient.

The DeepSeek-V3 technical report states the model used 2.788 million H800 GPU hours and, assuming a rental price of 2 dollars per GPU hour, that the total training cost was about 5.576 million dollars. The report explicitly says this covers only the official training run and excludes prior research and ablation experiments. It is therefore a final-run compute cost, not a total development budget.

DeepSeek-R1 was trained mainly with reinforcement learning on top of the DeepSeek-V3 base, using the GRPO algorithm. The R1-Zero variant used pure reinforcement learning with no supervised fine-tuning and developed long reasoning chains on its own. The full R1 model adds a small amount of cold-start supervised data and a multi-stage pipeline to improve readability while keeping the reasoning gains.

DeepSeek-R1 was a frontier-competitive reasoning model released with open weights at a low reported training cost, which challenged assumptions about how much compute a leading model required. On 27 January 2025, AI and chip stocks dropped sharply, and Nvidia recorded one of the largest single-day market-value losses in U.S. stock market history.

As of mid-2026 the lineup has moved past the original V3 and R1. DeepSeek shipped V3.1 in August 2025 and V3.2 in December 2025, the latter integrating reasoning into tool use. A DeepSeek-V4 preview, split into Pro and Flash variants, was announced in April 2026. Because model versions move quickly, verify the current lineup against DeepSeek's official documentation.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free