Every large language model has a working memory called the context window: the total amount of text it can consider at once. Vendors now advertise windows of hundreds of thousands, even millions, of tokens. But fitting text into the window and actually using it well are two very different things.
Tokens, not words
Models read text as tokens — fragments that average roughly three-quarters of an English word. A 200,000-token window holds around 150,000 words, about the length of two novels. Code, tables and non-English text often use more tokens per word.
Attention is the real bottleneck
Inside the model, a mechanism called attention lets each token weigh every other token. Researchers have repeatedly found that recall is strongest at the beginning and end of a long prompt and weaker in the middle, a pattern sometimes called lost in the middle.
Practical rules for long prompts
Put your instructions at the start and repeat the key question at the end. Use headings and clear delimiters around documents. When accuracy matters, retrieval — sending only the most relevant passages — still tends to beat stuffing everything in.
Key takeaways
- Tokens, not words
- Attention is the real bottleneck
- Practical rules for long prompts
