What a context window actually holds
2 hours ago
You typed a line twenty messages ago. Later in the same chat, the model cannot use that line. People treat a huge advertised window as a bigger brain. ChatGPT and Claude advertise that window. The window is a working desk. Fitting on that desk is different from being read.
Ask
Ask about this presentation
Answers are generated from this presentation.
Chapters
Show transcriptHide transcript
A long chat goes blank
You typed a line twenty messages ago. Later in the same chat, the model cannot use that line. People treat a huge advertised window as a bigger brain. ChatGPT and Claude advertise that window. The window is a working desk. Fitting on that desk is different from being read.
A pasted PDF can miss the paragraph
You paste a PDF and wait for the answer. The model misses the one paragraph that answered you. That paragraph still sat inside the advertised window. A PDF is a stack of sheets. Those sheets share the same desk as the chat.
The context window is a working desk
A context window is the working desk. Every page the model can look at sits on that desk. The next reply is written from that pile. The reply itself sits on that desk too. Anthropic’s docs call this working memory. Working memory is the pile in use right now. The training library sits next door in a closed cabinet. Training is the older reading that taught the model to write. That library stays closed during a chat. OpenAI’s help page measures the same desk in pieces. A token is a numbered piece from the last episode. The desk is measured in those pieces. One request has a hard piece budget. The reply has a separate cap on how long it can run. The standing instructions already sit on the blotter before you type. Those standing instructions are the system prompt.
Standing instructions occupy the blotter
The diagram starts with the empty desk. The blotter already holds the standing instructions. Those standing instructions occupy space before you type a word. Uploaded pages occupy space too. The rest of the stack has to share the leftover room.
Each chat turn lays down two pages
Each chat turn lays down two sheets. Your message is one sheet. The model’s reply is the second sheet. Turn one stacks both. Turn two stacks both again. The current question lands on top. A gap has to stay open for the answer being written. Anthropic’s own diagram draws that accumulation.
Pages that overflow slide off
The stack can grow taller than the desk. Extra sheets then slide off the far edge. OpenAI’s help page names three moves. You can shorten the input. You can split the input. You can summarize the input. Anthropic’s API returns an error that the prompt is too long. The prompt is the pile you sent. Overflow is a fall off the edge.
The middle of the desk can go dim
The stack can still fit. The stack can still fail you. A 2023 paper by Liu and colleagues measured this. Liu and colleagues called the finding Lost in the Middle. Lost in the Middle means the model gets the answer right most often when the answering page sits at the start. The model also gets the answer right when the answering page sits at the end. The model misses more often when the answering page sits in the middle. Liu and colleagues gave a 2023 ChatGPT model, GPT-3.5-Turbo, twenty Wikipedia pages. Exactly one of those pages answered the question. The remaining nineteen pages did not answer. With an empty desk, GPT-3.5-Turbo answered fifty-six point one percent of the questions correctly. With only the answering page on the desk, GPT-3.5-Turbo answered eighty-eight point three percent of the questions correctly. With the answering page tenth of twenty, GPT-3.5-Turbo answered fifty-three point eight percent of the questions correctly. That middle share sits under the empty-desk share. The answering page still fit. GPT-3.5-Turbo scored worse on that middle page than on an empty desk. From the first of those twenty pages to the tenth, the share of correct answers dropped more than twenty points. The small U on this desk is that sag. The five dots are the paper’s tabulated points. A thirty-page version of this test sits under the empty-desk share on the longer twin of GPT-3.5-Turbo. That twin answered fifty point five percent of the questions correctly at the tenth of thirty, against fifty-six point zero percent on an empty desk. The shorter GPT-3.5-Turbo cannot hold that prompt.
A bigger window can still sag
This picture is roughly right. Here is where the picture breaks. GPT-3.5-Turbo and its longer twin overlay on the same ten-page and twenty-page prompts. On twenty pages, the first-page shares of correct answers sit on top of each other. The tenth-page shares of correct answers sit on top of each other. Claude-1.3 is a 2023 Claude model. Claude-1.3 and its much larger twin overlay the same way. A longer advertised window did not lift the middle. Those 2023 shares stay a 2023 measurement. A 2026 product page is a different measurement. The positional sag is the finding. Anthropic’s current docs name that sag context rot. Extra pages on the desk make the middle dimmer. Liu and colleagues put the question before the pages and after the pages. That twice-asked question nearly fixed a lookup of random ID codes. The twice-asked question barely moved the Wikipedia-page questions. Stuffing the desk is a weak strategy. Liu notes the sag resembles a human list-memory curve. The paper stops at the resemblance.
You typed a line twenty messages ago
You typed a line twenty messages ago. That line can still sit on the desk and still go unread. A context window is a working desk. Every prior turn shares one stack. The standing instructions share that stack. The uploaded pages share that stack. The reply being written shares that stack. The desk is measured in numbered pieces. Sheets that overflow slide off the far edge. Sheets that still fit can still go unread. The start and the end stay bright. The middle can go dim. Liu’s 2023 measurement is that U. The number on the box counts how many pages fit. Reading the pages in the middle is a different job. How those pieces become meaning is the next Foundations lesson.
