sunk cost

I keep running into the same fork in the road when I am working in Cursor, Claude Code, or Copilot, namely the question of whether I should continue the session I already have open or start a new chat and re-establish context from scratch. By the time I am asking myself that question the session usually has twenty or thirty turns in it and I have already spent an hour getting the agent oriented on my project structure and constraints, which makes starting over feel wasteful. What typically breaks the inertia of maintaining context for me is the moment the agent ignores a constraint I set ten prompts ago, or proposes a rewrite that awkwardly merges the approach I abandoned with the approach I asked for, at which point I realize the chat’s accumulated history is working against the given session and tokens are starting to be wasted.

The underlying question I am trying to answer in this post is when a clean context state outweighs the tokens already sitting in a session. I wrote about pieces of this before in the end of writing code and the beginning of directing it where I noted that long sessions lose track of earlier instructions and that discrete, well-scoped tasks produce better results than one mega-conversation. I keep making the wrong call anyway and paying for it in tokens and frustration, which is partly what motivated me to write this and why I suspect a lot of people maintain chat state longer than they should, because the sunk cost of all those prior turns feels real even when the model can no longer maintain KV attention and degrades notably.

prior tokens poison future iterations at a certain point

Large language models are autoregressive by design, which means each token a model emits is sampled from a probability distribution conditioned on every token that came before it in a given session, including system prompts, tool outputs, diffs, error messages, and so on, and prior tokens do not disappear when you change direction mid-session because they continue to weight the model’s output statistically in ways that are predictable, which is maddening if you are trying to get your code to compile. For example, I noticed this effect in action recently when I changed direction after spending multiple turns building a script one way, realized the approach was flawed, and asked my agent to rewrite it using a different framework, at which point the result was a Frankenstein hybrid with imports from dead artifacts, naming conventions that made no sense, and logic that tried to satisfy my prompt but was clearly garbled, because old code acts as a statistical anchor that a clean session removes by giving a model no prior tokens to be confused by.

a million tokens on a spec sheet, thousand in practice

Model providers advertise context windows in the hundreds of thousands or millions of tokens, and Paulsen (2025) draws a useful distinction between the Maximum Context Window, which is what the model technically accepts, and the Maximum Effective Context Window, which is the point where additional tokens begin degrading output quality. In testing across several frontier models Paulsen found accuracy degraded severely around 1,000 tokens for some problem types, that several top-tier models failed with as little as 100 tokens in context on certain tasks, and that all models fell far short of their advertised maximum in some cases by over 99 percent, which should make anyone who has been stuffing entire repositories into a single chat feel at least slightly uneasy about what they were actually buying with all those tokens. Liu et al.’s “Lost in the Middle” documents a related phenomenon where models recall information best when it appears at the very beginning or very end of the input with performance dropping sharply when relevant information sits in the middle, producing a U-shaped attention pattern that maps uncomfortably well onto an agent coding session where your system rules and .cursor/rules sit at the top, your latest message sits at the bottom, and the constraint you established on turn eight ends up buried under tool outputs, stack traces, and three rounds of diffs.

When I find myself re-stating something like “remember, we are using Btrfs, not ext4” or “do not touch the auth module,” I take that as my signal that the effective context window has been exceeded for the task at hand regardless of what the provider’s spec sheet claims, and I have started treating those moments the same way I treat a compiler error because something structural is wrong with the session and repeating the constraint louder rarely fixes it when the underlying problem is that the model’s attention mechanism has already distributed its weight elsewhere. The fix in those cases is usually to compile what still matters into a fresh prompt and move on rather than continuing to invest tokens in a session that has already demonstrated it cannot hold the constraints you care about, which is a habit that feels wasteful until you compare the cost of one compiled restart against the cost of five more correction turns in a session that keeps drifting.

sessions that crawl by afternoon

Every message in a long session resends the full history through the prefill phase before the model writes a single new token, and the KV cache that stores past token representations grows linearly with context length in ways that increase memory pressure and time-to-first-token, which means that when I am doing rapid iteration where I fix one line, run tests, and fix again, a 30,000-token session requires paying to re-read 30,000 tokens before the model touches the one line I care about. This experience is also evident through the lag I notice in Cursor when a session that started snappy in the morning takes several seconds to begin responding by afternoon even for trivial edits. I wrote about the billing side of this in building with ai is costly where I argued that tokens bill whether the output is useful and that a long session which has drifted off-task charges you for the drift along with the correction, and compaction features where the tool summarizes older turns to squeeze them back into the window help with the infrastructure limit but do not restore the model’s ability to follow constraints that got summarized away because the summarized version is a lossy compression of what you actually said and the model treats the summary as ground truth going forward, which is how you end up arguing with a paraphrase of yourself from forty-five minutes ago.

I have had Claude Code compact a session mid-debug and then confidently misremember which of my three attempted fixes actually ran, which is a special kind of hell when you are already rate-limited and trying to narrow in on a bug that only reproduces under a specific configuration you explained in detail three turns before the compaction happened. These types of experiences are part of why I no longer treat compaction as a substitute for the kind of deliberate prompt compilation I describe later in this post because compaction optimizes for fitting history into a window rather than preserving constraints and failure trajectories, so auto-compact should also be a signal to start a new chat.

when sunk cost doesn’t apply

Continuity earns its keep when re-establishing state would cost more than the attention penalty of a long history. Deep iterative debugging is the clearest case because a model needs the stack trace, previous failed attempts, and your system configuration to understand what not to try next, and if you spent two thousand tokens establishing specific domain logic, custom API endpoints, or schema rules then keeping the session alive until that module is finished makes sense because those turns function as few-shot examples of the exact tone, formatting, and logic you want the model to continue producing, which matters when you have spent real effort steering it away from the em-dash-heavy slop it defaults to. Cursor and similar tools partially substitute for chat history through open files, @ references, rules files, and git state that give the agent project context without requiring every prior turn to remain in the window, and I have .cursor/rules files in most of my repos now precisely because they persist constraints across sessions without me re-stating them every time I open a new chat, which means the decision to keep a session live mainly concerns the conversation-specific failure trajectory and whether contextual value lives in the thread itself, in the failed attempts and the narrowing search. If enough context can be gleaned from the code itself, git history, or other rules and commands, then a fresh session is usually better.

Domain switching is the other clear signal for me to create a new chat since moving from debugging a kernel panic to writing user-facing documentation, or from implementing a payment flow to refactoring CSS, means the technical context from the first task adds nothing to the second and actively competes for the model’s attention. I start a new chat for new domains almost always even if the old session still has plenty of nominal context window left on paper, because I would rather spend five minutes compiling a fresh prompt than twenty minutes watching the model cite debugging context while trying to write API documentation.

factor keep session start fresh
task status iterating on the same specific problem when changing approach or switching tasks
model behavior following constraints reliably hallucinating or ignoring recent instructions
latency and cost TTFT and token cost acceptable session feels sluggish or expensive per edit
domain same technical context throughout moving from debugging to docs, or kernel to frontend

copy your state, open a new chat, move on

The most helpful rule I internalize is that if I can copy the current state of my code, paste it into a new chat with a three-sentence preamble, and lose nothing critical, then I start fresh. Using prompt compilation and summary tools makes it less painful to constantly create new chats because they help gather the current file state, any constraints, failures, and next steps into one self-contained message rather than forcing me to reconstruct all of that from memory. I built Quality Prompts partly because I kept doing this by hand and wanted a systematic way to generate structured prompts from a brief idea, though a plain text block in a new Cursor chat works fine for most tasks and I still use that more often than I admit because the overhead of opening a tool sometimes feels like more work than just writing the three sentences myself even when I know a tool would probably produce something more thorough.

Sunk cost is a real force in agent sessions because the hour you spent orienting the model and the twenty turns of debugging history feel like assets you would be throwing away if you closed the tab, and that feeling is not imaginary even when the session has already stopped producing useful output. The sunk cost fallacy is the part of your brain that treats those spent tokens as a reason to keep going, and in agent workflows the fallacy is usually pointing you in the right direction because the tokens you already paid for cannot be recovered by sending more tokens into a degraded session. When this happens, the only rational move is to extract whatever state still matters, write it down, and start over. I have closed sessions that contained genuinely good work in the transcript and felt a pang of loss doing it, but every time the fresh chat reached the same working state in fewer turns than it would have taken to corral the old one back on track, I feel validated. And you will too. Take my word for it.

If this is not new for you, I am curious, what do you use as your signal to start a new chat?