Picture the AI bill at the end of the quarter. The line item the finance team has been quietly flagging for three months is the line item for token costs in the engineering department. The vendor has been promoting the larger context window as the productivity feature developers have been asking for, and the larger context window has been quietly multiplying the per request cost by somewhere between three and seven depending on the model. The engineering leader who saw the bill grow from a small line item to a board level conversation is the engineering leader who finally asked the question nobody had asked. The question is whether the extra context window was earning the bill, or whether the developers had quietly routed every task through the largest model because the largest model was the easiest model to pick.
Here is the working version of the field guide. The shorter version is the part the engineering leader and the team actually have time to read.
What the cost actually is
Token cost sits at the top because token cost is the line item the bill shows. Vendors charge per token, the per token price scales with the context window, and the vendors have been raising the per token price on the larger windows because the larger windows cost the vendor more to serve. A prompt that fits in a 32k window can run five to ten times as much when it lands in a 200k window, even before the model has generated a single response token. The compound effect is the part the bill captures and the part the engineering leader has been quietly trying to control.
Latency sits second because latency is the part users have been quietly blaming on the AI. A larger context window means the model has more input to process on every request, and more input means more time before the first token streams back. A 200k call on a long document can take fifteen to forty five seconds before the response starts, and the user experiences that delay as the AI being slow, when the AI is doing exactly what the user asked. The smaller window does the same job in two to four seconds. The difference is the window, not the model.
Vendor lock in sits third because that is the part the engineering leader has been quietly trying to break. Developers have been standardising on the model with the larger window. That standardisation makes switching expensive. Switching is a move the leader has to fund at a moment when the bill is what the board is asking about. The lock in compounds. Every month the larger window remains the default is another month of code, prompts, and workflows built around the assumption it is always available.
What the extra window actually enables
Full codebase context is the use case vendors use to justify the larger window. A user can paste the entire repository into the prompt, ask the AI to refactor a function across the files the refactor touches, and get a coherent response because the model has the full picture. Smaller windows force file by file work, which works for narrow refactors and breaks for cross cutting changes. The tradeoff is real, the productivity gain is real, and the productivity gain only applies to a fraction of the prompts sent in a day.
Long document analysis is the second use case. Logs, reports, policy documents, the kind of artefact that does not fit in a 32k window. The larger window means the document can be handed to the AI in a single shot for a summary, a gap analysis, a risk review. Smaller windows require chunking the document, pasting each chunk, stitching the response. The labour saved is meaningful on long documents and the labour saved is invisible on short ones.
Multi turn conversation is the third use case. A long running context can be carried across the turns needed to refine a draft, debug a function, work through a design. The smaller window forces more restarts, which costs time and which costs accuracy because the model loses the earlier turns. The tradeoff is real. The tradeoff is also the one users have been quietly paying the token cost for, on conversations that did not need the long context in the first place.
How to control the bill
Route by task. The AI platform vendors ship a router for this. A simple classifier puts the easy prompts on the small model, the moderate prompts on the medium model, and the genuinely large context jobs on the large model. Anthropic, OpenAI, Google all expose a routing layer or a routing pattern. The leader who has set the router up is the leader who has cut the bill by 40 to 70 percent without changing the developer workflow. The leader who has not set the router up is the leader paying the largest model to summarise short emails.
Summarise before sending. A running summary of the long document can be kept, pasted instead of the full text, and the smaller window handles the prompt. The summary costs a few thousand tokens to build. The savings on every subsequent call are real. The pattern is the one a single sprint can ship. Most teams have not bothered to set it up because the AI platform does the routing badly enough that no one has noticed the bill yet.
Set a budget alert. The finance team can configure a per team spend cap in the AI platform, the alert fires when the cap is hit, and the engineering leader knows which team has been over using the larger window. The leader who has set the alert is the leader who can intervene before the next quarterly review. The leader who has not set the alert is the leader explaining the bill to the board three months too late.
The bottom line
Three habits, set once. Stop reaching for the top shelf on every task. Compress the input. Cap the bill. None of this is a research problem. All of it is a discipline problem. The leader who has the discipline ships the next quarter without the surprise line item. The leader who does not has the conversation at the next board meeting.
Sources & Further Reading
All claims in this article are sourced from primary documentation, vendor advisories, and reputable security researchers.
Spotted an error? Email the editor. Corrections are issued with a visible correction note.
Editorial standards. Every article on humanrequired.org is reviewed by a human editor before publication. AI may assist with drafting or research; final editorial control is human. Read the full standards.
