Why there are limits: memory, compute and context
P10.orientation.04 · Audience: guest, it-ml, language-pro
Language models can feel limitless, so their sudden limits are confusing: a model forgets what you said earlier, stumbles on a very long document, or gets slower and more expensive as the task grows. None of this is a bug — it follows from three plain facts about any machine that processes text: it has a limited working memory, every step costs something, and it only ever sees a fixed window of your conversation. No maths. Your glossary is one click away.
🗣️ From a linguist's perspective: a model works like an interpreter's short-term memory
A simultaneous interpreter can only hold so much of a speech in mind at once. Say too much without a pause and the earliest words slip away — not from carelessness, but because working memory is finite. A language model has the same constraint, made explicit: a fixed context window, the span of text it can attend to at one time. Go past it and the earliest parts fall out of view, exactly like the start of an over-long sentence for the interpreter.
A model does not remember your whole conversation the way a person remembers a relationship. At each moment it can only look at a fixed span of recent text — its context window, measured in tokens (the text pieces from the previous module). Think of it as a desk of a fixed size: papers you set down fit until the desk is full, and then, to add a new sheet, an old one slides off the edge.
Worked example — the model that “forgot”. You give a translation model a glossary at the start of a long chat, work through many paragraphs, and near the end it ignores a glossary term. It did not lose interest — the glossary scrolled out of the window. The fix is practical, not magical: restate the key term when it matters, or keep the working text within the window. Knowing the limit exists tells you how to work with it.
📚 Go Further
The limits, in plain terms.
- Context window — the fixed span of recent text (in tokens) a model can attend to at once; its working memory. See your glossary.
- Compute — the time, hardware and energy a model spends per token; more text means more of it. See your glossary.
- Practical habit: when a model “forgets”, ask first whether the information simply left its window — then restate it, rather than assuming the tool failed.
Ask the mentor about this module
Ask a question about this content. The mentor explains and grounds its answer in what you are studying; asking is recorded as a learning signal, not a grade.