The Million-Token Mirage
Why enterprise AI needs a context graph.
There is a strange double rhythm to artificial intelligence right now. On one beat, the industry celebrates ever-larger context windows — a million tokens, two, four — as though capability were simply a matter of wider doorways. On the other beat, the people actually pushing AI into production keep walking into a wall that wider doorways do not break. The disappointment is rarely about a model's reasoning. It is about what gets lost on the way in: the relationships between the pieces you dropped into the prompt.
A salesperson approves a nonstandard discount on a Friday-night call. A doctor writes an off-label prescription with a one-line note. A support engineer escalates a ticket because she remembers the same bug from a different customer two years earlier. A general counsel agrees to an unusual indemnification clause because, last quarter, a similar one held up in arbitration. Each of these is a decision. Each is the kind of decision that makes a company itself. And almost none of them are durably captured by the software the company runs on.
This gap has, in the last six months, quietly acquired a name. A small but growing chorus of investors, founders, and database engineers has begun calling it the context graph: a structured, machine-readable record of an organization's reasoning, not just its outputs. Foundation Capital, in a widely circulated essay this spring, called context graphs "AI's trillion-dollar opportunity." Aaron Levie, the CEO of Box, debated the thesis on his podcast with Foundation's Joanne Chen and PlayerZero's Animesh Koratana and ended up agreeing — somewhat against his initial reflex. Will Lyon, a product manager at Neo4j, has taken to describing context graphs in the most deliberately plain way possible: "the missing why." In his definition, a context graph is "a knowledge graph that contains all of the information necessary to make decisions throughout the organization."
That definition is calm by design. It is also, I suspect, the most consequential idea in enterprise software right now.
To see why, consider the shape of the problem. Every company is sitting on two fundamentally different kinds of data. The first kind — the easy kind — is state: the closed deal, the approved invoice, the resolved ticket, the shipped order. State is what Salesforce, Workday, ServiceNow, SAP, and Snowflake were designed to preserve. The second kind — the hard kind — is the reasoning that produced that state. Why this customer got a forty-percent discount when the published floor was twenty. Why this account received custom payment terms. Why this candidate was hired despite a weak case round. Why this incident was triaged the way it was.
That reasoning, today, lives almost everywhere except in software. It lives in Slack threads, deal-desk calls, Zoom recordings no one has rewatched, two-line follow-up emails (”approved per our chat”), and the heads of the senior people most likely to leave. The moment those moments end, the reasoning evaporates, and the company is left with a state record that — to a future employee, or to a future model — is uninterpretable without the human chain of memory that produced it.
A model with a million-token window does not solve this. It just gives you a larger room in which to be confused. As Emil Eifrem, the CEO of Neo4j, has been putting it lately: the question is not whether you have enough information; it is whether the information has any structure to its relationships. Without that structure, longer context becomes, in his blunt phrase, a larger pool of noise.
What systems of record were never built to remember
The most useful frame I have heard for this comes from Aaron Levie, and it is, of all things, a story about travel agents.