Nate Meyvis

Thinking about context

The context window is one of the new fundamental ideas in programming. Here is a nice Claude-centric overview of it.

Programming now requires a strong working knowledge of what the context window is, both in general and in a particular session. As with so many AI topics, however, it’s hard to know how to improve: it’s fundamentally a bit mysterious, and it’s changing so fast.

How should we think about context?

It’s tempting to think of context by analogy to RAM: It’s what’s more available to the system; information gets loaded into and purged from it; it’s more limited than distantly stored information; and so on.

The analogy is, however, dangerous. RAM is a resource; it’s basically always good to have more of it; and it’s basically always good to be using the RAM you have.1 Each of these aspects of the analogy can mislead us:

First, available space in the context window is a resource, but the context itself is content that will be used in future turns of a session. It’s tempting to envision a process of choosing something from the context intelligently and selectively, but this would require first looking at the whole thing.

The underlying architecture, moreover, uses pairwise relationships of everything in the context (see this useful document from Anthropic). Adding to the context, therefore, has a quadratically dilutive effect on it. The Nth bit of context taxes the first N–1 in a way that adding RAM doesn’t, so using your context is not a free lunch.

This, by the way, relates to Simon Willison's recent observation2 that, as tools have gotten massively better in many dimensions, context windows have not grown proportionately. It also helps us understand why tools sometimes come with recommendations not to use the maximum size of the context window.3

If nothing else, remember that "context rot" doesn't (just) mean that it's bad to have false or misleading things in the context window. Rather, it means that adding to the context limits, or at least changes, the effect of what's already there.

How can we get better at thinking about context?

More concretely, how can we think better about this?

  1. Here as always, look!: Claude offers a /context command that shows you how you're using the context window. I've often been surprised at what's there.
  2. Consider /compacting before the window runs out; the option to give Claude instructions about how to compact is useful.
  3. Try /forking a conversation, compacting one and not the other, and comparing future responses.
  4. This document from Anthropic is also useful. I can't read it without being struck by how much this is a discipline in its infancy.
  5. So: stay tuned, because this is all likely to change quickly.

  1. This is a simplification, about which I welcome correction. But almost every computer person has, for example, had the experience of thinking “oh, no, I’m almost out of RAM, even though I’m only doing X,” only to learn that the system is just using the RAM efficiently and ready to stop using it on X if something more important comes along.↩

  2. If memory serves. Unfortunately, I can't find the reference.↩

  3. Here's an example.↩

#generative AI #software