What is AI model distillation and how does it work in 2026?

AI model distillation (knowledge distillation) is a model compression technique that transfers capabilities from a large teacher model to a smaller student model. The student learns to mimic the teacher's behavior, inheriting most of its task performance at a fraction of the inference cost. In 2026, distillation is how 671B frontier models become 7B consumer-GPU models — with 5-50x size reduction and surprisingly small quality loss. How it works: (1) Teacher model — a large, highly capable model (e.g., DeepSeek R1 671B, GPT-5, Claude Opus). The teacher generates outputs on a set of input prompts. (2) Student model — a smaller, resource-efficient model (e.g., Qwen 7B, Llama 8B). The student is trained to reproduce the teacher's outputs. (3) Training signal — the student learns from one of three signal sources: (a) Soft labels (logits) — the teacher's probability distribution over the vocabulary at each position. Not just 'Paris is correct' but also 'the is a reasonable continuation.' This dense supervision lets a small model match a big model's behavior more efficiently than learning from hard labels alone. (b) Generated text (sequences) — the teacher's actual output tokens, used as labels for standard SFT. This is what DeepSeek used for R1-Distill — pure SFT on R1 generations. (c) Hidden states — the teacher's intermediate layer activations. Used when student and teacher share architecture family (DistilBERT, TinyBERT). (4) Loss function — typically a weighted combination of distillation loss (matching teacher) and standard cross-entropy (matching ground truth labels if available). L_KD = (1-gamma) * L_CE(y, p_s) + gamma * T^2 * KL(p_t^tau, p_s^tau), where KL is Kullback-Leibler divergence, T is temperature scaling, gamma is the distillation weight. Three main distillation approaches: (1) Logit-based distillation (Hinton, 2015) — the student learns from the teacher's output probability distribution (logits). Captures semantic correlations between vocabulary tokens. Requires white-box access (open-weights teacher). The classic approach. (2) Sequence-level distillation (CoT distillation) — the teacher generates complete outputs (including chain-of-thought reasoning), and the student trains on these via standard SFT. Works with API-only teachers (black-box). This is the approach behind DeepSeek R1-Distill. (3) Hidden-state distillation — the student mimics the teacher's intermediate layer activations. Requires shared architecture family. Used for DistilBERT, TinyBERT, MobileBERT. Less common for modern decoder-only LLMs. On-policy distillation (GKD): a newer approach where the student generates completions from its own distribution, and the teacher labels each position. The student trains on what it would actually produce at inference. Better generation quality, especially for long-form and reasoning. 2-3x compute cost (teacher pass per training step). The DeepSeek R1-Distill recipe: (1) Generate trajectories — use full R1 (671B) on ~800K math + code + science prompts. Each output includes thinking chain plus final answer. (2) Filter for correctness — for math/code, verify final answer against ground truth. For open-ended responses, use quality filtering. (3) SFT the student — train Qwen or Llama models on the filtered R1 outputs. Standard SFT, no logit access needed. (4) Result — R1-Distill-Qwen-32B captures ~85% of R1's reasoning at 1/20th the inference cost. R1-Distill beats much larger non-reasoning models on AIME math despite being 1/20th the size of full R1. The proof point: DistilBERT — HuggingFace shrank BERT by 40%, ran 60% faster, kept 97% of its language understanding on GLUE. That's the ceiling. The floor, when teams skip data quality and evaluation, is a small model that is confidently wrong on the inputs that matter. The key: 'Distillation is how 671B frontier models become 7B consumer-GPU models. By training a small student to mimic a large teacher, you can compress capabilities at 5-50x size reduction with surprisingly small quality loss.' Distillation is the quiet workhorse behind most production small models: domain QA assistants, code completion engines, and embedding rerankers are typically distilled from larger teachers rather than trained from scratch (localaimaster 2026, tanhdev 2026, redwerk 2026).

What is the difference between logit distillation and sequence-level distillation?

Logit distillation and sequence-level distillation are the two primary approaches for distilling LLMs. The difference is what signal the student learns from and whether you need access to the teacher's internal probabilities. Logit-based distillation (Hinton, 2015): (1) What it uses — the teacher's full probability distribution over the vocabulary (logits/soft labels) at each token position. (2) What the student learns — not just 'Paris is correct' but the entire probability distribution: 'Paris is 90% likely, Lyon is 3% likely, Marseille is 2% likely.' This dense supervision captures semantic correlations between tokens. (3) Access required — white-box (open-weights teacher). You need access to the teacher's logits, which means the teacher must be open-weights. API-only teachers (GPT-5, Claude) don't expose logits. (4) Loss function — L_KD = (1-gamma) * L_CE(y, p_s) + gamma * T^2 * KL(p_t^tau, p_s^tau). KL divergence between teacher and student distributions, with temperature scaling T to soften logits. (5) Temperature — controls how soft the distribution is. High temperature (T=4-8) reveals more of the teacher's distribution. Low temperature (T=1) is closer to hard labels. (6) Advantages — dense supervision (every token position carries signal), captures semantic relationships, more efficient learning per sample. (7) Disadvantages — requires open-weights teacher, more complex implementation, teacher and student must share vocabulary (or use cross-tokenizer strategies). (8) Best for — when you have an open-weights teacher and want maximum signal per sample. Sequence-level distillation: (1) What it uses — the teacher's generated text (complete output sequences). The teacher generates responses to prompts, and these responses are used as labels for standard SFT. (2) What the student learns — to reproduce the teacher's text output token by token. Standard next-token prediction (cross-entropy loss). No logit information. (3) Access required — black-box (API-only teachers work). You only need the teacher's text output, which any API provides. This is the approach used when distilling from GPT-5, Claude, or other closed models. (4) Pipeline — for each prompt: teacher.generate(prompt) → save (prompt, completion) → SFT student on all (prompt, completion) pairs. (5) Advantages — works with any teacher (API or open-weights), simple implementation (just SFT), no vocabulary matching needed, can distill chain-of-thought reasoning. (6) Disadvantages — less signal per sample (only hard tokens, no distribution), requires more samples to achieve same quality, no semantic correlation information. (7) Best for — when using API-only teachers, when distilling reasoning (CoT), when simplicity matters. This is what DeepSeek used for R1-Distill. (8) CoT distillation — a variant where the teacher generates a chain-of-thought reasoning trace alongside the answer. The student trains on both the reasoning and the answer. This is how R1-Distill transfers reasoning capability. Key differences: (1) Signal density — logit: dense (every position has a full distribution). Sequence: sparse (only the chosen token). (2) Teacher access — logit: white-box (open-weights). Sequence: black-box (any API). (3) Implementation — logit: complex (KL divergence, temperature, vocabulary matching). Sequence: simple (standard SFT). (4) Sample efficiency — logit: more efficient (more signal per sample). Sequence: less efficient (need more samples). (5) Reasoning distillation — logit: can distill reasoning but complex. Sequence: ideal for CoT distillation (just include reasoning in teacher output). (6) Legal considerations — logit: requires open-weights teacher (no legal issues with open-weights). Sequence: using closed-model APIs for distillation may violate terms of service. The key: 'Logit distillation gives more signal per sample but requires open-weights teachers. Sequence-level distillation works with any API but gives less signal per sample.' For most production distillation in 2026, sequence-level (CoT) distillation is the practical choice — it's simpler, works with any teacher, and is the method behind DeepSeek R1-Distill (localaimaster 2026, tanhdev 2026, redwerk 2026).

How does DeepSeek R1 distillation work?

DeepSeek R1 distillation is the most visible and impactful distillation project of 2025-2026. It demonstrated that reasoning capabilities from a 671B model can be transferred to 7-32B student models, which then outperform much larger non-reasoning models on reasoning benchmarks. The R1-Distill recipe: (1) Teacher — DeepSeek R1 (671B parameters, MoE, 37B active). A reasoning model trained via RL with verifiable rewards. (2) Students — Qwen 2.5 (7B, 14B, 32B) and Llama 3.1 (8B, 70B). Open-weights models with strong base capabilities. (3) Method — sequence-level distillation (CoT distillation). Pure SFT on R1-generated outputs. No logit access needed. (4) Data generation — use full R1 on ~800K math + code + science prompts. Each output includes the full chain-of-thought reasoning trace (in thinking tags) plus the final answer. (5) Filtering — for math/code problems, verify the final answer against ground truth. Only correct trajectories are kept. For open-ended responses, use quality filtering. (6) Training — standard SFT on the filtered R1 outputs. The student learns to reproduce R1's reasoning traces and answers. (7) Result — R1-Distill-Qwen-32B captures ~85% of R1's reasoning at 1/20th the inference cost. R1-Distill-Qwen-14B and R1-Distill-Llama-8B also show strong reasoning capabilities. Why it works: (1) Reasoning is transferable — R1's chain-of-thought reasoning traces (breaking problems into steps, self-reflection, verification, strategy exploration) can be learned by smaller models through SFT. The student doesn't just learn the answers — it learns the reasoning process. (2) Strong base students — Qwen 2.5 and Llama 3.1 are already capable models. Distillation refines their reasoning capability rather than creating it from scratch. (3) CoT as training signal — by including the full reasoning trace in the training data, the student learns to structure its thoughts, decompose tasks, and self-correct errors before yielding final answers. (4) Scale of data — 800K prompts with verified correct outputs provides enough signal for the student to generalize the reasoning patterns. (5) No RL needed — the student learns reasoning through SFT alone. No reinforcement learning, no reward model, no PPO. This makes the process much simpler and more accessible. Results: (1) R1-Distill-Qwen-32B — captures ~85% of R1's reasoning. Beats much larger non-reasoning models on AIME math problems despite being 1/20th the size of full R1. (2) R1-Distill-Qwen-14B — strong reasoning at 14B scale. Runs on a single RTX 4090. (3) R1-Distill-Llama-8B — reasoning in an 8B model. Runs on a laptop. (4) R1-Distill-Qwen-7B — minimal reasoning capability in a 7B model. Why Qwen 2.5 Coder is an ideal student: (1) Extensive code-centric pre-training — Qwen 2.5 Coder has exceptional coding and syntax comprehension. (2) Base models struggle with high-level logic — by distilling reasoning traces from R1, Qwen Coder combines precise code syntax with structured algorithmic planning. (3) Architecture compatibility — Qwen models are compatible with R1's output format. The legal dimension: (1) DeepSeek used their own open-weights R1 as the teacher — no legal issues. (2) However, there have been allegations of distillation from closed models. OpenAI found evidence of DeepSeek using its models to train a rival. Anthropic publicly accused DeepSeek, Moonshot, and MiniMax of distillation attacks on Claude. (3) Most closed-model providers (OpenAI, Anthropic, Mistral, xAI) include clauses prohibiting using their outputs to train competing models. (4) The safe approach: use open-weights teachers (R1, Llama, Qwen) for distillation. Avoid using closed-model APIs as teachers. The key: 'R1-Distill-Qwen-32B captures ~85% of R1's reasoning at 1/20th the inference cost. Distilled open models achieve reasoning and coding scores that surpass vanilla models multiple times their size.' DeepSeek R1 distillation proved that reasoning capability is transferable through sequence-level SFT, democratizing reasoning models (localaimaster 2026, tanhdev 2026, redwerk 2026).

Can I distill from GPT-5 or Claude to train my own model?

No — you should not distill from GPT-5, Claude, or other closed-model APIs to train your own model. Most closed-model providers explicitly prohibit this in their terms of service. The legal landscape: (1) OpenAI — terms of service prohibit using outputs to train competing models. OpenAI found evidence of DeepSeek using its models to train a rival, and the dispute became the reference case every legal team now cites. (2) Anthropic — publicly accused DeepSeek, Moonshot, and MiniMax of distillation attacks on Claude in February 2026. Made clear that they are watching for this. (3) Mistral — includes clauses prohibiting using outputs to train competing models. (4) xAI — similar restrictions on using outputs to train competing models. (5) Google — terms restrict using outputs to train competing models, though enforcement has been less aggressive. What 'competing' means: the providers interpret 'competing' broadly. It doesn't just mean building a direct competitor to GPT-5. It means using their outputs to train any model that could substitute for their services. This covers most practical distillation use cases. What you can do: (1) Use open-weights teachers — DeepSeek R1 (MIT license), Llama 3 (Llama license), Qwen 2.5 (Apache 2.0), Mistral (Apache 2.0). These models allow distillation. (2) Use your own models — if you have a fine-tuned model, you can distill from it to a smaller model. (3) Use open datasets — datasets like OpenOrca, Alpaca, Dolly are generated from open models or with explicit permission. (4) Generate synthetic data with open models — use R1, Llama, or Qwen to generate training data for your student model. (5) Check the teacher's license — always check the teacher's license before distilling. What you cannot do: (1) Use GPT-5 API outputs as training data — violates OpenAI terms. (2) Use Claude API outputs as training data — violates Anthropic terms. (3) Use any closed-model API to generate training data for a competing model — violates most providers' terms. (4) Use closed-model outputs even if you filter or modify them — the restriction is on using outputs, not on how you process them. The risks: (1) Legal action — providers can sue for breach of terms of service. (2) API access termination — providers can cut off your API access. (3) Public exposure — Anthropic and OpenAI have publicly named companies they accuse of distillation. (4) Model takedown — if your model is found to be distilled from a closed model, you may be forced to take it down. (5) Reputation damage — being publicly accused of distillation attacks damages your reputation. The safe approach: (1) Choose an open-weights teacher — DeepSeek R1, Llama 3, Qwen 2.5, Mistral. (2) Check the license — ensure the license allows distillation. Most open-weights licenses do, but some (like Llama's community license) have restrictions on commercial use. (3) Document your data lineage — keep records of which teacher generated which data. (4) Use open datasets when available — many high-quality datasets are already generated from open models. (5) When in doubt, consult a lawyer — the legal landscape around distillation is evolving. The key: 'After January 2025, the closed-model providers tightened their terms. OpenAI, Anthropic, Mistral, and xAI all include clauses that prohibit using their outputs to train competing models.' Use open-weights teachers for distillation. The open-weights ecosystem (R1, Llama, Qwen, Mistral) provides capable teachers that are legally safe to distill from (redwerk 2026, localaimaster 2026).

How much data do I need for model distillation?

The amount of data needed for model distillation depends on the task complexity, the quality of the teacher's outputs, and the capability gap between teacher and student. In 2026, the consensus is similar to fine-tuning: quality matters more than quantity. Data requirements by task type: (1) Narrow task (classification, extraction, specific format) — 1,000 to 5,000 well-curated samples. Start with real domain prompts: production logs, support tickets, redacted user queries. Diversity matters more than raw volume past a certain point. (2) Broad instruction-following — 20,000 to 100,000 samples. Need diverse prompts covering many task types. (3) Reasoning distillation (CoT) — 100,000 to 800,000 samples. DeepSeek used ~800K math + code + science prompts for R1-Distill. The reasoning traces are long (thousands of tokens each), so the total token count is much larger. (4) Domain-specific QA — 5,000 to 20,000 samples. Enough to cover the domain's question distribution. (5) Code completion — 10,000 to 50,000 samples. Need diverse coding tasks and languages. Quality over quantity: (1) The student learns whatever is in the data — including the teacher's biases, hallucinations, and stylistic tics. (2) A 1,000-sample dataset of meticulously curated samples will beat a 50,000-sample dataset of mediocre samples. (3) Time spent on filtering usually exceeds time spent on training. (4) Spot-check a random 5% slice for hallucinations. (5) For regulated outputs, get a domain expert to review the data. Data pipeline: (1) Collect prompts — use real domain prompts from production logs, support tickets, user queries. If real prompts aren't available, generate seeds with a separate model and have a human review the distribution. (2) Generate teacher outputs — for each prompt, have the teacher generate a response. For CoT distillation, the teacher generates a reasoning trace plus the answer. (3) Filter for correctness — for math/code, verify the answer against ground truth. For open-ended responses, use quality filtering (length, format, coherence). (4) Dedupe — run ROUGE-L filtering for near-duplicates. Remove exact and near-exact duplicates. (5) Drop outliers — remove length and format outliers that will confuse the student. (6) Spot-check — review a random 5% slice for hallucinations, quality, and bias. (7) Domain expert review — for regulated outputs, have a domain expert review the data. (8) Split — hold out 10-15% for evaluation. Never evaluate on training data. The 80/20 rule: the training part is the easy 20%. The data and evaluation work is the 80% that decides whether the project delivers or stalls. Most teams who go all-in on distillation end up needing help on the synthetic data and evaluation layers, not the training loop. Step-by-Step distillation: a technique worth knowing when data is limited. The teacher generates a rationale alongside the answer, and the student trains on both. Google and Snorkel researchers showed this can match a larger model with significantly less data, which matters when your seed pool is small. The key: 'For a narrow task, 1,000 to 5,000 well-curated seeds often beat 50,000 noisy ones. For a broader instruction-following student, plan for 20,000 to 100,000.' Spend more time on data quality than on training. The data pipeline is where the project is won or lost (redwerk 2026, localaimaster 2026, tanhdev 2026).

When should I use distillation vs fine-tuning vs RAG?

Distillation, fine-tuning, and RAG serve different purposes in the LLM adaptation toolkit. Understanding when to use each is essential for making the right architectural decision. Distillation: (1) What it does — transfers capabilities from a large teacher to a smaller student. The student inherits the teacher's task performance at a fraction of the inference cost. (2) Best for — when you need a smaller, faster, cheaper model that matches a larger model's performance on a specific task. When you want to deploy on consumer hardware. When inference cost is the primary constraint. When you want to transfer reasoning capabilities (CoT distillation). (3) Cost — high upfront (generate teacher outputs, filter, train) but lowest inference cost. (4) When to use — you need a 7B model that performs like a 70B model on your task. You want to self-host on consumer GPUs. You need reasoning in a small model. You're building a production system with high query volume where inference cost matters. Fine-tuning: (1) What it does — adapts a model to your domain, task, style, or format. Changes how the model behaves. (2) Best for — when you need consistent tone, style, or format. When you need domain-specific language. When you need a specific output format. When you want a smaller model to perform a specific task well. (3) Cost — moderate upfront (collect data, train with LoRA) and moderate inference cost. (4) When to use — you need behavioral adaptation (tone, style, format). You have labeled input/output pairs. You need consistent output across thousands of calls. You want to reduce reliance on expensive API models. RAG: (1) What it does — provides the model with current knowledge at query time from a knowledge base. Changes what the model knows. (3) Best for — when the model needs current information. When you need source attribution. When knowledge changes frequently. When the knowledge base is large and growing. (4) Cost — low upfront (set up retrieval) but higher per-query (retrieval + generation). (5) When to use — the model needs current knowledge. You need citations. Knowledge changes frequently. Multiple users need different knowledge. Decision framework: (1) Need current knowledge? → RAG. Distillation and fine-tuning create models with frozen knowledge. (2) Need consistent behavior/style/format? → Fine-tuning. Distillation can also help but fine-tuning is more direct. (3) Need a smaller, cheaper model that matches a larger model? → Distillation. This is distillation's unique strength. (4) Need reasoning in a small model? → CoT distillation (R1-Distill style). (5) Need all three? → Hybrid: distill a small model for capability + fine-tune for behavior + RAG for knowledge. The hybrid approach: the best production systems combine all three. (1) Distill a 7B model from a 70B teacher — for base capability and reasoning. (2) Fine-tune the distilled model — for domain-specific behavior, tone, and format. (3) Add RAG — for current knowledge and citations. Example: a customer support agent that uses a distilled 7B model (for reasoning and code understanding) + fine-tuned on support conversations (for empathetic tone and structured responses) + RAG (for current product info and policies). Distillation vs fine-tuning: (1) Distillation — transfers capability from teacher to student. The student becomes capable of tasks it couldn't do before. Focuses on capability. (2) Fine-tuning — adapts behavior to your domain. The model becomes better at your specific task. Focuses on behavior. (3) They compose — distill first (for capability), then fine-tune (for behavior). This is the R1-Distill pattern: distill reasoning from R1, then fine-tune for specific tasks. The key: 'Distillation is for capability transfer (make a small model as capable as a large model). Fine-tuning is for behavior adaptation (make a model behave the way you want). RAG is for knowledge (give a model current information).' For most production systems, the answer is a hybrid: distill for capability, fine-tune for behavior, RAG for knowledge (localaimaster 2026, redwerk 2026, ailearnings 2026).

July 13, 2026

TL;DR — AI model distillation in 2026: compress 671B models to 7B with 5-50x size reduction. DeepSeek R1-Distill-Qwen-32B captures ~85% of R1's reasoning at 1/20th cost. Three types: logit (Hinton, soft labels, KL divergence), sequence-level (SFT on teacher outputs, R1 method), hidden-state (DistilBERT: 40% smaller, 60% faster, 97% GLUE). On-policy distillation (GKD): student generates, teacher labels, 2-3x cost. CoT distillation: distill reasoning traces, R1-Distill beats larger non-reasoning models on AIME. White-box vs black-box: logit access requires open-weights. Legal: OpenAI, Anthropic prohibit distillation from APIs. Data: 1K-5K for narrow tasks, 20K-100K for broad instruction-following. Pipeline: generate → filter → dedupe → train → evaluate.

AI Model Distillation in 2026: How to Compress 671B Models to 7B with Knowledge Distillation

Distillation is how 671B frontier models become 7B consumer-GPU models. By training a small student to mimic a large teacher, you can compress capabilities at 5-50x size reduction with surprisingly small quality loss. It's the quiet workhorse behind most production small models.

Key Statistics

Metric	Value	Source
R1-Distill-Qwen-32B reasoning retention	~85% of R1	localaimaster 2026
R1-Distill size reduction	1/20th inference cost	localaimaster 2026
R1-Distill training data	~800K prompts	localaimaster 2026
DistilBERT size reduction	40% smaller	redwerk 2026
DistilBERT speed improvement	60% faster	redwerk 2026
DistilBERT quality retention	97% on GLUE	redwerk 2026
Size reduction range	5-50x	localaimaster 2026
Narrow task data	1K-5K curated samples	redwerk 2026
Broad instruction data	20K-100K samples	redwerk 2026
On-policy distillation cost	2-3x off-policy	localaimaster 2026
GKD quality gain	Better long-form generation	localaimaster 2026

Distillation Methods Comparison

Method	Signal Source	Teacher Access	Complexity	Best For
Logit (Hinton)	Soft labels (probabilities)	White-box (open-weights)	High	Maximum signal per sample
Sequence-level	Generated text	Black-box (any API)	Low	Simplicity, CoT distillation
Hidden-state	Intermediate activations	White-box (shared architecture)	Medium	BERT-family compression
On-policy (GKD)	Student-generated + teacher labels	White-box	High	Long-form, reasoning
CoT distillation	Reasoning traces + answers	Black-box or white-box	Low	Reasoning transfer (R1-Distill)

Sources: localaimaster (2026), tanhdev (2026), redwerk (2026).

Distillation Pipeline

flowchart TD Teacher["Teacher Model\n(e.g., DeepSeek R1 671B,\nGPT-5, Claude Opus)"] --> Legal{"Legal to distill?\nCheck teacher's license"} Legal -->|Open-weights (R1, Llama, Qwen)| OK["Proceed"] Legal -->|Closed API (GPT-5, Claude)| Stop["STOP: Terms prohibit\ndistillation from APIs\nUse open-weights teacher"] OK --> Prompts["1. Collect Prompts\nReal domain prompts\nProduction logs, support tickets\n1K-5K (narrow) or 20K-100K (broad)"] Prompts --> Generate["2. Generate Teacher Outputs\nTeacher generates responses\nFor CoT: includes reasoning trace\n+ final answer"] Generate --> Filter["3. Filter for Correctness\nMath/code: verify vs ground truth\nOpen-ended: quality filtering\nDrop hallucinations"] Filter --> Dedupe["4. Dedupe & Clean\nROUGE-L near-duplicate filtering\nDrop length/format outliers\nSpot-check 5% for quality"] Dedupe --> Split["5. Split Data\n85-90% training\n10-15% evaluation\nNever evaluate on training data"] Split --> Train["6. Train Student\nSFT on teacher outputs\nUse LoRA for efficiency\nStandard next-token prediction"] Train --> Eval["7. Evaluate\nCompare student vs teacher\nCompare vs base model\nTest on actual task distribution\nCheck for regression"] Eval --> Deploy["8. Deploy\nStudent model: 7B-32B\nServe with vLLM/Ollama\n5-50x cheaper inference"]

Source: redwerk (2026), localaimaster (2026), tanhdev (2026).

Distillation vs Fine-Tuning vs RAG

Approach	What It Changes	Best For	Upfront Cost	Inference Cost
Distillation	Model capability	Small model matching large model	High	Lowest
Fine-tuning	Model behavior	Tone, style, format, domain	Moderate	Moderate
RAG	Model knowledge	Current info, citations	Low	Higher
Hybrid	All three	Production agents	Highest	Low at scale

Source: localaimaster (2026), ailearnings (2026).

R1-Distill Results

Model	Size	AIME 2024	Runs On	Cost vs R1
DeepSeek R1 (teacher)	671B (37B active)	79.8%	Data center	1x
R1-Distill-Qwen-32B	32B	~68%	RTX 4090	~1/20x
R1-Distill-Qwen-14B	14B	~59%	RTX 4090	~1/48x
R1-Distill-Llama-8B	8B	~50%	Laptop	~1/84x
R1-Distill-Qwen-7B	7B	~45%	Laptop	~1/96x

Source: localaimaster (2026), tanhdev (2026). Approximate values — exact scores vary by benchmark version.

Implementation Guide

Phase	What to Do	Timeline
1. Choose teacher	Open-weights: R1, Llama, Qwen. Check license	1 day
2. Choose student	Match architecture family. Qwen, Llama, Mistral	1 day
3. Collect prompts	Real domain prompts: 1K-5K (narrow), 20K-100K (broad)	1-2 weeks
4. Generate teacher outputs	Teacher generates responses. For CoT: include reasoning	1-3 days
5. Filter and clean	Verify correctness, dedupe, drop outliers, spot-check 5%	3-5 days
6. Split data	85-90% train, 10-15% eval	1 hour
7. Train student	SFT with LoRA. 30-90 min for 1K examples (7B)	1-3 hours
8. Evaluate	Compare vs teacher, vs base, on actual task distribution	1-2 days
9. Check for regression	Test on general tasks — ensure no catastrophic forgetting	1 day
10. Deploy	Serve with vLLM or Ollama. 5-50x cheaper than teacher	1-2 days
11. Monitor	Track quality, latency, cost vs teacher in production	Ongoing

Best Practices

Use open-weights teachers — DeepSeek R1 (MIT), Llama 3, Qwen 2.5, Mistral. Closed-model APIs (GPT-5, Claude) prohibit distillation in their terms. The safe approach is open-weights (redwerk 2026).
Quality over quantity — 1K-5K well-curated samples beat 50K noisy ones. The student learns whatever is in the data, including the teacher's biases and hallucinations. Time spent on filtering usually exceeds time spent on training (redwerk 2026).
Use real domain prompts — production logs, support tickets, redacted user queries. Diversity matters more than raw volume past a certain point. If real prompts aren't available, generate seeds and have a human review (redwerk 2026).
Filter for correctness — for math/code, verify answers against ground truth. For open-ended, use quality filtering. Drop hallucinations. Spot-check a random 5% slice (redwerk 2026).
Include CoT for reasoning — distill not just the answer but the chain-of-thought. This is how R1-Distill transfers reasoning capability. The student learns to structure its thoughts, decompose tasks, and self-correct (tanhdev 2026).
Use LoRA for training — train the student with LoRA/QLoRA for efficiency. 30-90 minutes for 1K examples on a single GPU. The student's adapter can be merged for zero inference overhead (localaimaster 2026).
Prove no regression — after distillation, test the student on general tasks to ensure it hasn't forgotten general capabilities. Compare against the base model on general benchmarks (redwerk 2026).
Consider on-policy distillation for long-form — GKD (student generates, teacher labels) produces better long-form and reasoning quality. 2-3x compute cost but worth it for high-stakes models (localaimaster 2026).

For related topics, see our how to fine-tune LLM, LoRA vs QLoRA, small language models, AI reasoning models, and evaluate fine-tuned LLM guides.

FAQ

What is on-policy distillation (GKD) and when should I use it?

On-policy distillation, also known as GKD (Generalized Knowledge Distillation), is a distillation approach where the student generates completions from its own distribution, and the teacher labels each position in the student's completion. This addresses the exposure bias problem of off-policy (sequence-level) distillation. The exposure bias problem: in standard sequence-level distillation, the student trains exclusively on teacher-generated data. At test time, the student generates from its own distribution, which may differ from the teacher's. Errors accumulate because the student rarely learns from its own induced states. The student is trained on teacher trajectories but deployed on student trajectories — a train-test mismatch. How GKD works: (1) Student generates — at each training step, the student generates a completion from the prompt using its own current policy. (2) Teacher labels — the teacher computes logit distributions for each position in the student's completion. (3) Distillation loss — compute KL divergence between teacher and student distributions on the student-generated sequence. (4) Update — the student is updated to better match the teacher on its own generated outputs. The difference from off-policy: off-policy distillation trains on teacher-generated sequences. On-policy distillation trains on student-generated sequences with teacher labels. The student learns from what it would actually produce at inference, getting teacher-level guidance per token. Advantages: (1) Better generation quality — especially for long-form and reasoning. The student learns to correct its own errors. (2) Reduced exposure bias — training and inference distributions match. (3) Dense token-level feedback — the teacher provides guidance at every token position. Disadvantages: (1) 2-3x compute cost — need a teacher forward pass per training step. The teacher must score every student-generated sequence. (2) More complex implementation — need to coordinate student generation and teacher scoring in the training loop. (3) Slower training — student generation + teacher scoring per step is slower than off-policy SFT. (4) Requires white-box teacher — need teacher logits, so open-weights only. Implementation: GKD is implemented in TRL via GKDTrainer. Configure the teacher model, student model, and training data. The trainer handles the student generation and teacher scoring loop. When to use GKD: (1) Long-form generation — the exposure bias problem is most severe for long sequences. GKD's advantage grows with sequence length. (2) Reasoning tasks — where the student needs to learn multi-step reasoning. GKD helps the student learn from its own reasoning errors. (3) High-stakes models — where the 2-3x compute cost is justified by quality gains. (4) When you have an open-weights teacher — GKD requires logit access. (5) When off-policy distillation quality is insufficient — if sequence-level distillation produces a student that's good but not great, GKD can close the gap. When NOT to use GKD: (1) Short-form tasks — for short outputs (classification, extraction), the exposure bias is minimal. Off-policy is sufficient. (2) API-only teachers — GKD requires logit access. Use sequence-level distillation with API teachers. (3) Cost-sensitive projects — 2-3x compute cost may not be justified. (4) Quick prototyping — start with off-policy, upgrade to GKD if quality is insufficient. Prefix on-policy distillation: a 2026 optimization. Research shows that GKD's learning signal is assigned unequally towards early tokens. Aligning the prefix is often sufficient for the student to complete the remaining reasoning correctly. Prefix OPD achieves performance close to full OPD while reducing training compute significantly — supervising only the first part of each student trajectory. The key: 'On-policy distillation addresses the exposure bias problem by training on student-generated sequences with teacher labels. It produces better generation quality, especially for long-form and reasoning, at 2-3x compute cost.' Use GKD for long-form and reasoning tasks where off-policy quality is insufficient. Start with off-policy, upgrade to GKD if needed (localaimaster 2026, arxiv 2026)."
- question: "What is DistilBERT and what can I learn from it?"
answer: "DistilBERT is the most famous example of model distillation in NLP. Created by HuggingFace, it demonstrated that BERT could be compressed by 40% and run 60% faster while retaining 97% of its language understanding capabilities on the GLUE benchmark. It is the proof point that distillation works and the reference case every distillation guide cites. What DistilBERT did: (1) Teacher — BERT (110M parameters). The original Google language model. (2) Student — DistilBERT (66M parameters). 40% smaller. (3) Method — hidden-state distillation. The student mimics the teacher's intermediate layer activations, not just the output. A triple loss: (a) Distillation loss — KL divergence between teacher and student output distributions. (b) MLM (Masked Language Modeling) loss — standard cross-entropy on masked tokens. (c) Cosine embedding loss — align student and teacher hidden states. (4) Architecture — removed token-type embeddings and pooler from BERT. 6 layers instead of 12. (5) Result — 40% smaller, 60% faster, 97% of GLUE quality. (6) Training — trained on the same data as BERT. No additional data needed. What DistilBERT proved: (1) Distillation works — a smaller model can match a larger model's performance with minimal quality loss. (2) Hidden-state distillation is effective — matching intermediate activations, not just outputs, preserves more of the teacher's capability. (3) The quality ceiling is high — 97% retention is close to perfect. For most tasks, the difference is negligible. (4) Speed and size improvements are real — 40% smaller and 60% faster makes the model practical for deployment. (5) No additional data needed — distillation can work with the same training data as the teacher. Lessons for LLM distillation in 2026: (1) Start with a strong teacher — BERT was the best model available. Use the best open-weights teacher you can (R1, Llama 3, Qwen 2.5). (2) Match architecture when possible — DistilBERT worked because the student was the same architecture as the teacher, just smaller. For LLMs, distilling within the same family (Qwen to Qwen, Llama to Llama) is easier than cross-family. (3) Use multiple loss signals — DistilBERT used three losses (distillation, MLM, cosine embedding). For LLMs, combine distillation loss with standard cross-entropy. (4) Remove unnecessary components — DistilBERT removed token-type embeddings and pooler. For LLMs, consider removing unnecessary components from the student architecture. (5) 97% is the ceiling — DistilBERT retained 97% of BERT's quality. This is the best case. For LLMs, expect 85-97% retention depending on the task and method. (6) The floor is lower than you think — when teams skip data quality and evaluation, the result is a small model that is confidently wrong on the inputs that matter. The key: 'DistilBERT shrank BERT by 40% and ran 60% faster while keeping 97% of its language understanding on GLUE. That is the ceiling. The floor, when teams skip the steps, is a small model that is confidently wrong on the inputs that matter.' DistilBERT is the proof that distillation works. But it's also a reminder that the ceiling is 97%, not 100% — and the floor is much lower if you cut corners on data and evaluation (redwerk 2026, localaimaster 2026)."
- question: "How do I evaluate a distilled model?"
answer: "Evaluating a distilled model requires comparing it against both the teacher model and the base student model on your actual task distribution. Without proper evaluation, you don't know if distillation actually transferred capability or just produced a different model. Evaluation steps: (1) Compare against the teacher — run the student and teacher on the same test set. The student should capture 85-97% of the teacher's quality. If the gap is larger, the distillation didn't work well — check data quality, training config, or try a different method. (2) Compare against the base student — run the distilled student and the original (non-distilled) student on the same test set. The distilled model should outperform the base model. If it doesn't, distillation added no value. (3) Compare against fine-tuning — compare the distilled model against the base model fine-tuned on your task data. If fine-tuning achieves similar results, distillation may not have been necessary. (4) Test on actual task distribution — published benchmarks (MMLU, GLUE) are useful for comparison, but your actual task distribution is what matters. Create a test set that reflects what the model will see in production. (5) Test for regression — evaluate the distilled model on general tasks (not just your specific task) to detect catastrophic forgetting. If general performance drops significantly, the distillation has degraded general capabilities. (6) Use task-specific metrics — for classification: accuracy, precision, recall, F1. For generation: BLEU, ROUGE, BERTScore. For reasoning: AIME, GPQA, SWE-Bench. For code: pass@k. Choose metrics that reflect your task's success criteria. (7) Human evaluation — for subjective tasks (tone, style, quality), use human evaluators. Have 3+ evaluators rate responses. Calculate inter-annotator agreement. (8) LLM-as-judge — use a frontier model to evaluate responses. Provide scoring criteria and have the judge model rate responses. Faster than human evaluation but less reliable. (9) Measure inference cost — track tokens/second, latency, GPU memory, and cost per query. The whole point of distillation is cheaper inference — verify you're actually getting it. (10) Measure quality-to-cost ratio — compare quality (accuracy, F1) per dollar of inference cost. The distilled model should have a much better quality-to-cost ratio than the teacher. (11) A/B testing — deploy the distilled model alongside the teacher. Route a percentage of traffic to the distilled model. Compare user engagement, satisfaction, or task completion rates. (12) Regression testing — maintain a set of test cases that the model must pass. Run these after every retraining. The common failure mode: the student is confidently wrong on inputs that matter. The student produces well-structured, plausible-sounding outputs that are incorrect. This happens when: (a) The teacher's hallucinations were included in the training data. (b) The training data didn't cover the full distribution of inputs. (c) The student overfit to the training data and doesn't generalize. Prevention: (1) Filter teacher outputs for correctness — don't include hallucinations in training data. (2) Cover the full input distribution — include diverse prompts that cover the variety of inputs. (3) Hold out evaluation data — never evaluate on training data. (4) Test on real production inputs — not just benchmark data. (5) Monitor in production — track quality metrics and user feedback. The key: 'The training part is the easy 20 percent. The data and evaluation work is the 80 percent that decides whether the project delivers or stalls.' Evaluation is not optional — it's the only way to know if distillation worked. Budget as much time for evaluation as for data preparation and training combined (redwerk 2026, localaimaster 2026)."