Large models are expensive to run. Small models are cheap but less capable. Knowledge distillation is the technique of training a small “student” model to reproduce the behavior of a large “teacher” model — capturing much of its ability in a fraction of the size. It is one of the main reasons today’s small models are surprisingly good.

The original idea

Distillation was popularized by Geoffrey Hinton, Oriol Vinyals and Jeff Dean in the 2015 paper Distilling the Knowledge in a Neural Network. Their insight concerned what a model’s outputs contain.

A classifier trained on images produces a probability for every class. For a picture of a dog, it might say “dog: 0.9, wolf: 0.08, car: 0.0001”. The correct label alone says only “dog”. But the full distribution — the soft targets — also reveals that this dog looks somewhat like a wolf and nothing like a car. Hinton and colleagues called this hidden information “dark knowledge”.

Training a student to match the teacher’s soft probabilities, rather than only the hard labels, transfers more information per example. A temperature parameter softens the distributions further so that small probabilities carry more signal.

Distillation for language models

For language models, distillation takes several forms:

  • Logit distillation: the student learns to match the teacher’s probability distribution over the next token, the closest analogue to the original method. This requires access to the teacher’s outputs in detail, so it is used mainly by labs distilling their own models.
  • Sequence-level distillation: the teacher generates complete answers, and the student is fine-tuned on them as if they were human-written examples. This only needs the teacher’s text outputs.
  • Reasoning distillation: the teacher’s step-by-step reasoning traces are used as training data so the student learns to reason similarly.

A well-known early example is DistilBERT (2019), which kept most of BERT’s language-understanding performance while being 40% smaller and 60% faster, according to its authors. More recently, DeepSeek released small models distilled from its R1 reasoning model, reporting that they acquired strong reasoning abilities from the teacher’s outputs.

Why distillation matters

  • Cost: a distilled model can serve millions of requests at a fraction of the price.
  • Speed: smaller models respond faster, which matters for real-time applications.
  • On-device AI: phones and laptops can run distilled models locally.
  • Specialization: a student can be distilled for one task, such as classification or extraction, and become excellent at it.

Limits

The student rarely surpasses the teacher on broad abilities; it inherits the teacher’s errors and biases too.

Narrowing: a student trained on a teacher’s outputs for one domain may lose general abilities.

Terms of service: many commercial providers prohibit using their model outputs to develop competing models. Distilling from a closed model can therefore breach contracts, and the issue has led to public disputes between AI companies. Open-weight models with permissive licenses — some explicitly allow distillation — avoid this problem.

Data quality: generated data can contain subtle errors; filtering and verification remain necessary.

How teams use it in practice

A common pattern: prototype with a large model, collect its outputs on real inputs, filter them, then fine-tune a small open model on the result. If evaluation shows the student matches the teacher on the target task, switch production traffic to the cheaper model and keep the large one for hard cases.

Common mistakes to avoid

  • Distilling from a model whose terms forbid it. Read the provider’s usage policy before using outputs to train other models.
  • Training on unfiltered outputs. Teacher mistakes become student habits; verify or filter generated examples.
  • Evaluating only on the teacher’s style. Test the student on fresh, real inputs, not just the generated dataset.
  • Expecting a general model. Distillation works best for a defined task or domain.

Key takeaways

  • Distillation trains a small student to imitate a large teacher model.
  • Soft probability targets carry more information than hard labels, as Hinton and colleagues showed in 2015.
  • For LLMs, students often learn from teacher-generated answers and reasoning traces.
  • Distilled models are cheaper, faster and suitable for on-device use.
  • Check licenses and terms: many providers forbid distilling their outputs into competing models.

Frequently asked questions

Is distillation the same as fine-tuning?

Distillation is a kind of training in which the targets come from another model rather than from human labels. In practice it is often implemented as fine-tuning on teacher outputs.

Can a distilled model be better than its teacher?

Occasionally on a narrow task, especially with filtered data, but generally the student approaches rather than exceeds the teacher.

It depends on the provider’s terms of service, which often prohibit using outputs to build competing models. Open-weight models with permissive licenses are the safer choice.