Context windows — the amount of text a model can consider at once — have grown from a few thousand tokens to hundreds of thousands or even millions. It’s tempting to paste everything in and let the model sort it out. Our take: long context is a powerful tool, but more context is not automatically better.

The problems with stuffing the prompt

Attention gets diluted. Research such as Lost in the Middle (2023) found that models were often better at using information at the beginning or end of a long input than in the middle. Newer models handle long inputs much better, but performance on complex tasks can still degrade as irrelevant material grows.

Distraction. Irrelevant or contradictory documents can lead models to wrong answers. Quality of context matters more than quantity.

Cost. Most models charge per input token. Sending a 500-page document with every question is expensive, even with caching.

Latency. Processing long inputs takes time, slowing responses.

Harder debugging. When an answer is wrong, it’s difficult to know which part of a huge context misled the model.

When long context shines

  • Reading a whole document that must be understood as a unit — a contract, a book chapter, a codebase module.
  • Cross-referencing many parts of a document, where splitting would break connections.
  • One-off analysis where building a retrieval system isn’t worth it.
  • Long conversations and agent sessions that need memory of earlier steps.

When retrieval works better

Retrieval-augmented generation selects only the most relevant passages for each question. It is usually better when:

  • the knowledge base is large and growing;
  • questions target specific facts;
  • you need citations to exact sources;
  • cost and speed matter at scale.

Many production systems combine both: retrieval to select relevant material, and long context to include enough of it.

Practical tips

  1. Curate before you paste. Remove irrelevant sections, duplicates and boilerplate.
  2. Put instructions and the question clearly — often at the end of long inputs, and repeated if needed.
  3. Ask for citations to sections or pages to check the model used the right parts.
  4. Test with your real documents: check whether answers are correct for facts located throughout the text, not just at the start.
  5. Use prompt caching where available for repeated long contexts.

The bottom line

Long context removes many technical limits, but it doesn’t remove the need to give models the right information. Think of the context window as a desk, not a warehouse: keep on it what the task needs.

A simple test you can run

To see how a model handles your long documents:

  1. Take a long document you know well — for example a 100-page report.
  2. Write ten questions whose answers appear in different places: early, middle and late sections, plus a few that require combining information from two sections.
  3. Paste the full document and ask the questions one at a time in fresh conversations.
  4. Score the answers and note which positions cause errors.
  5. Repeat with a retrieval approach that provides only relevant passages, and compare accuracy, cost and speed.

This takes under an hour and tells you more about your situation than any general claim about context length. Many teams discover that a combination works best: retrieval to find the right sections, with enough surrounding context to preserve meaning.

The cost arithmetic

Long contexts multiply costs quickly. If a document is 200,000 tokens and you ask 50 questions about it, you may pay for 10 million input tokens unless you use caching or retrieval. Retrieving a few thousand relevant tokens per question can reduce that by orders of magnitude. For one-off analysis, the convenience of long context may be worth it; for a product used by many people, efficiency matters.

What to watch next

  • Long-context benchmarks that test reasoning, not just retrieval.
  • Prompt caching and pricing for long inputs.
  • Hybrid retrieval and long-context architectures.
  • Memory systems for long-running agents.

Key takeaways

  • Large context windows are powerful but not automatically better.
  • Irrelevant material can distract models and reduce accuracy.
  • Long inputs increase cost and latency.
  • Retrieval is often better for large knowledge bases and specific facts.
  • Curate context and test with facts spread through your documents.

Frequently asked questions

What is a context window?

The maximum amount of text, measured in tokens, that a model can consider in a single request, including both input and output.

Does long context replace RAG?

Not entirely. Long context helps with whole-document tasks; retrieval remains more efficient and precise for large knowledge bases.

How do I know if my model handles long context well?

Test it with your own documents, asking questions whose answers appear in different positions.