AI System Design Numbers: Estimate Before You Build

A model price tells you what a token costs. It does not tell you what your feature costs. The missing part is usually the workload: growing chat history, repeated tool results, retries and output nobody can use.

I want an estimate I can challenge in a design review, not a pricing table I have to trust. The rates below are deliberately hypothetical. The arithmetic is the reusable part.

An AI feature estimate needs workload counts before it needs vendor prices. Start with how often the feature runs, what each request sends, what it returns and how often it needs another attempt. Then attach rates to those quantities.

Store the model identifier and the date of the price lookup beside the calculation. Tokenizers, model limits and billing rules can change; none should be treated as a cross-provider constant.

A six-turn chat sends more input than six copies of its opening prompt when the client includes the growing transcript. The following example sends 15,510 input tokens and generates 2,100 output tokens per conversation, before retries or tool use.

Assume a 1,500-token system prompt, a 60-token user message each turn and a 350-token reply. The first request contains 1,560 tokens. Each later request adds one previous user message and reply, or 410 tokens.

Input across six turns
= 6 × 1,560 + 410 × (0 + 1 + 2 + 3 + 4 + 5)
= 15,510 tokens

Output across six turns
= 6 × 350
= 2,100 tokens

At 100,000 monthly active users and eight conversations each, that becomes 12,408 million input tokens and 1,680 million output tokens. Here are three invented rate cards applied to exactly that workload, without caching:

These are not current prices or claims that the three options produce equivalent answers. The table isolates price sensitivity. Provider routing only saves money when the cheaper route still completes the task to the required standard.

A tool loop can multiply input volume because each model call may include earlier tool calls and results. With the assumptions below, thirteen requests send 126,360 input tokens, although the final transcript contains only 15,240. The repeated input is the important budget line.

Assume 4,000 instruction and tool-definition tokens, a 200-token task, twelve tool calls with 120 generated tokens and 800 returned tokens each, then a 500-token final answer.

Input = 13 × 4,200 + 920 × (0 + 1 + ... + 12)
      = 126,360 tokens
Output = 12 × 120 + 500
       = 1,940 tokens

At hypothetical rates of $5 per million input tokens and $25 per million output tokens, the model portion is about $0.68. Tool fees, browser compute, storage and failed attempts are additional costs. A one-call answer to the same prompt is cheaper, but it has not necessarily done the same work.

I would budget maximum steps, maximum cost and an elapsed-time limit together. A dollar cap alone does not stop an inexpensive loop from keeping someone waiting.

Prompt caching changes the estimate when repeated input is eligible for reuse and the billed read rate is lower than processing it fresh. Savings must include any write premium and cache misses. Stable text at the beginning helps, but usage records decide whether it worked.

For an illustrative prefix used six times, suppose a cache write costs 1.25 times normal input and each read costs 0.1 times. A 1,500-token prefix then costs 1,500 × (1.25 + 5 × 0.1) = 2,625 input-token equivalents instead of 9,000. That is a saving on this prefix, not on the entire run.

Use the actual provider's minimum prefix length, lifetime and rates before applying that example. Anthropic's prompt-caching documentation is one provider-specific reference, not a universal billing contract.

Embedding storage starts with vector count multiplied by dimensions and numeric precision. For pgvector's float32 vector type, the documented value size is four bytes per dimension plus eight bytes. That calculation excludes table and index overhead, replicas, backups and working memory.

The distinction between five million documents and forty million chunks is larger than many model-selection decisions. Calculate chunk count from a representative corpus, including overlap, rather than multiplying page count by a universal token ratio.

There is also an index constraint to check before a backfill: pgvector's HNSW index supports up to 2,000 dimensions for vector, and 4,000 for halfvec. The limits and storage formula are in the pgvector README, checked on September 7, 2026. Storage support and index support are different limits.

Reducing dimensions or precision is a quality trade-off. Measure retrieval recall on your own questions before treating the smaller footprint as a free saving. That is the same pattern behind sizing an architecture before building it: make the assumption visible, then test the one that could reverse the decision.

Optimise the stage that dominates the user's waiting time on your measured workload. Retrieval, queueing, model startup, generation and tool execution can each become the bottleneck. A public benchmark for a different corpus, region or concurrency level cannot identify yours.

For example, a 20 ms retrieval saving matters differently in a 100 ms search endpoint than in a 15-second agent run. It can still matter at high request volume or at the tail. “The model is slower” is not a reason to ignore a database bottleneck without measuring it.

Streaming also creates two clocks: first useful content and completed usable output. Rendering structured output progressively helps the first clock; validation and tool completion determine the second.

A review-ready estimate shows its assumptions, its range and the measurement that could change the design. Separate model spend from infrastructure, and divide total spend by useful completed results. A low token rate is not a win if the system needs repeated repairs.

My final check is simple: could another engineer reproduce the number without asking what I meant? If not, the estimate needs another line of explanation, not another decimal place.

Keep the worksheet. Refresh the rates. Replace the guesses with measurements as soon as the feature has real traffic.