Latent vs. Deterministic Space

Sources#

• The New Physics of Business — Garry Tan, Y Combinator

Summary#

Garry Tan's design discipline for agent systems (practitioner-opinion): be deliberate about where the computation is actually happening, because it always happens in one of two places — and "all of the AI engineering problems we run into, it's usually because something is happening in one side of the equation that should be in the other."

• Latent space — the LLM itself. What it's for: taste, judgment, "understanding what a human actually wants when they say something vague," the non-deterministic calls. You steer it with markdown (Agent Context Files).
• Deterministic space — what engineers already know: the code the agents write, external storage, verifiable state.

The worked example: seating 800 people#

Tan's live case (YC Startup School): seat 800 of 6,000 attendees so each person's neighbors are the perfect people for them to meet. The division of labor:

• The multi-dimensional array of 800 seats — the state — "must not live in the context window." It belongs in deterministic space.
• The LLM does the human part: judging who should meet whom — the thing a human organizer would otherwise do by printing 800 pages and shuffling them in a big room for a month.

Combined, "a couple hundred dollars worth of tokens and probably 10 minutes" — a task that was economically impossible six months prior. The example generalizes: latent space supplies judgment per decision; deterministic space holds the state and enforces the constraints.

Why this framing earns a page#

It compresses several harder-won lessons in this wiki into one diagnostic question — which side should this computation be on?

• State out of the context window is the working rule behind Context Window Smart Zone (the smart-zone budget is spent on judgment, not storage) and behind this vault's own architecture (LLM-as-Compiler Knowledge Base: the wiki holds the state; build.py/lint.py do the deterministic bookkeeping; the LLM does only the interpretive compile).
• Steering latent space with markdown is the Agent Context Files pattern named as one half of a two-sided architecture rather than a standalone trick.
• The bug taxonomy — "something happening on the side it shouldn't" — covers both familiar failure classes: LLMs doing arithmetic/state-tracking that belongs in code (hallucinated bookkeeping), and brittle code hard-coding judgment that belongs in the model (the Software 3.0 point — Karpathy's MenuGen "shouldn't exist" because the paradigm-native version pushes the whole task into latent space).
• It is the architecture-level cousin of Planning / Execution Division of Labor: that page splits decisions between human and agent; this one splits computation between model and code.

Connections#

• Agent Context Files — markdown as the steering mechanism for the latent side
• Context Window Smart Zone — the capacity argument for keeping state out of the window
• Software 3.0 — Karpathy's paradigm frame for the same boundary; his MenuGen example is the inverse bug (deterministic app doing latent-space work)
• Planning / Execution Division of Labor — the human/agent decision split; this page is the model/code computation split
• Agent Harness Engineering — harness design is largely the engineering of this boundary: what the model sees vs. what the scaffold enforces mechanically
• LLM-as-Compiler Knowledge Base — this vault as an instance: deterministic generators and linters around a latent compiler
• AI-Native Organization — the org-level thesis from the same talk; the org mapping presumes each encoded process knows which side its steps run on
• Garry Tan — the framing's author

Open Questions#

• Tan asserts the "wrong side" diagnosis covers most AI-engineering bugs. Does any incident/failure taxonomy (agent postmortems, eval failure analyses) actually classify failures by computation-locus, and what fraction lands in each side?
• The seating example prices latent-space judgment at "a couple hundred dollars of tokens" for 800 seat assignments. As models absorb more deterministic capability (Harness Shrinkage as Models Improve), does the economically-optimal boundary move toward latent space, or does state-out-of-context remain invariant?

Sources#

• The New Physics of Business — Garry Tan, Y Combinator — Garry Tan, "The New Physics of Business," AI Engineer, 2026-07-17, §latent space vs. deterministic space (8:38–10:53)