In March 2022, researchers at DeepMind published a paper with a simple but consequential message: the large language models of the day were too big for the amount of data they had been trained on. The paper, Training Compute-Optimal Large Language Models, introduced a model called Chinchilla that outperformed much larger rivals by rebalancing size and data. It changed how the entire industry plans training runs.

The question

Training a language model costs compute. Given a fixed compute budget, you can spend it on a bigger model (more parameters) or on more training data (more tokens). Which split produces the best model?

Earlier work, notably the 2020 scaling-laws paper from OpenAI, had suggested that model size should grow faster than data. Following that advice, labs built ever-larger models — GPT-3 had 175 billion parameters, Gopher 280 billion, Megatron-Turing NLG 530 billion — but trained them on roughly 300 billion tokens.

What the DeepMind team did

Led by Jordan Hoffmann, the team trained over 400 language models, ranging from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They used three different analysis approaches to estimate, for each compute budget, the combination of model size and data that minimized loss.

The finding: scale them together

All three approaches agreed: model size and training tokens should be scaled in roughly equal proportion. Double the compute, and you should increase both parameters and tokens by about the same factor.

Their results implied a rule of thumb that became famous: roughly 20 training tokens per parameter for compute-optimal training. By that measure, a 175-billion-parameter model should be trained on around 3.5 trillion tokens — more than ten times what GPT-3 had seen.

The proof: Chinchilla vs. Gopher

To test the prediction, DeepMind trained Chinchilla: 70 billion parameters on 1.4 trillion tokens, using the same compute budget as its own Gopher model (280 billion parameters, 300 billion tokens).

Chinchilla outperformed Gopher, GPT-3 and other larger models on a wide range of benchmarks, including an average accuracy of 67.5% on MMLU, a notable improvement over Gopher at the time. And because it was four times smaller than Gopher, it was also cheaper to run and fine-tune.

Why it mattered

Data became the bottleneck. If models need far more tokens, high-quality data becomes as valuable as GPUs. This triggered intense investment in data collection, cleaning, deduplication and, later, synthetic data.

Smaller models got better. Meta’s LLaMA models (2023) took the lesson further, training relatively small models on more than a trillion tokens and showing they could compete with much larger ones — making capable open-weight models practical.

Inference economics. A smaller model trained on more data is cheaper every time it answers a user. For products serving millions of requests, that saving dwarfs the extra training cost.

Beyond Chinchilla

The 20-tokens-per-parameter figure is often misunderstood as a target. It describes the best use of training compute only. Today, many labs deliberately overtrain smaller models far past that ratio, sometimes by an order of magnitude or more, because the extra training makes a model that is cheaper to serve for its whole life.

Other caveats:

  • The exact ratio depends on data quality, architecture and tokenizer.
  • Repeating data for many epochs gives diminishing returns, so the supply of fresh, high-quality text matters.
  • Later replication attempts debated details of the original fits, while broadly confirming that data had been undervalued.

The takeaway

Chinchilla did not invent a new architecture. It changed priorities. After 2022, the question for model builders was no longer simply “how big?”, but “how much good data, and how will we serve this model?” — a shift that shaped every major model generation since.

Frequently asked questions

What does “compute-optimal” mean?

For a fixed amount of training compute, it is the combination of model size and number of training tokens that gives the lowest loss. It says nothing about the cost of running the model afterwards, which is why products often prefer smaller models trained for longer.

Is the 20 tokens per parameter rule exact?

No. It is an approximate result from DeepMind’s experiments with a particular architecture, dataset and tokenizer. Treat it as a rule of thumb for compute-optimal training, not a law. Data quality, repetition and architecture all shift the best ratio.

Why did the paper matter for open-source AI?

It showed that a smaller model trained on much more data can beat a far larger one. That made capable models cheaper to train and run, which helped organizations such as Meta release competitive open-weight models that could run on more modest hardware.

Does this mean data matters more than model size?

Both matter, and the paper’s point is balance. Before Chinchilla, many labs had over-invested in size relative to data. After it, collecting, cleaning and generating high-quality data became one of the main priorities in model development.