The pilot cost almost nothing. Somebody put a model behind a form, it answered a hundred questions well, and the invoice at the end of the month was small enough that nobody read it carefully.
Then the same thing goes to five hundred people, or onto the public website, and the bill stops being a rounding error. Not because anything went wrong, but because the pilot was never the thing being priced. Almost every budget I am shown for an assistant is a pilot invoice multiplied by an optimistic number, and that is not how the cost behaves.
Here is what actually drives the annual number, roughly in the order that surprises people.
The model is rarely the largest line
Start with the counter-intuitive part. For most business assistants, what you pay the model provider is meaningful but not dominant. It is simply the most visible cost, because it arrives as a single itemised invoice with your name on it and everything else does not.
What is genuinely large, and rarely budgeted:
- The systems it reads from. Search infrastructure, a vector database if you use one, and the pipeline that keeps them current.
- Keeping the content correct. Somebody has to notice when a policy changes and make sure the assistant stops repeating the old one.
- Reading what it said. A human reviewing a weekly sample of conversations. This is the line most often cut, and cutting it is how an assistant is wrong for four months without anyone knowing.
- Change. Not incidents — ordinary change. A new product, a new rule, a new department wanting it to do something else.
Three of those four are people, not infrastructure. Which means an assistant is mostly an operating commitment wearing the costume of a software purchase, and that is the reframing that makes the rest of the number make sense.
What actually moves the model bill
Within the model line, the drivers are not the ones people expect.
How much context you send, not how much it answers. Every request typically carries instructions, retrieved documents and conversation history. On a retrieval-based assistant the input dwarfs the output, and the single most effective saving available is retrieving five relevant passages instead of twenty mediocre ones.
How many turns a conversation takes. A question resolved in one exchange is dramatically cheaper than the same question resolved in six, because each turn re-sends the history. Assistants that ask clarifying questions feel helpful and cost multiples.
Which model, for which step. Using your most capable model for every task is the most common and most expensive mistake. Classification, routing and extraction rarely need it. Anthropic’s guidance on building effective agents makes the general version of this point: start with the simplest thing that works and add complexity only when it demonstrably improves the outcome.
Whether anything is cached. If every request re-sends the same long instructions, you are paying for the same tokens continuously. Both major providers document the fix — OpenAI’s prompt caching and Anthropic’s — precisely because the pattern is so common.
The practical consequence: two teams building the same assistant on the same model can differ several-fold on the model line alone, purely from retrieval discipline and model selection.
The cost that arrives with success
The number nobody models is what happens when people actually use it.
Usage is not evenly distributed. A small group of people will use an internal assistant many times a day while most use it occasionally, and a public one will find one question that a thousand people ask in a week. Budget for a peak, not for an average, and know what the ceiling costs before you find out.
Set a hard spend limit from the first day. Not a monitoring alert — a limit that stops. It is the same class of control as the off switch in the four questions before you connect a model to a live system, and it is needed for the same reason: the failure mode that matters is the one nobody is watching at three in the morning.
The obligations that come with the data
An assistant that reads your documents is processing whatever is in them, and that has a cost attached even when nothing goes wrong.
The risk categories are well catalogued: the OWASP Top 10 for LLM applications lists sensitive information disclosure among them, and it is the one with a direct budget consequence, because preventing it is engineering time rather than a setting.
Somebody has to establish what personal data it can reach, whether the answers can leak one customer’s information to another, how long conversations are retained, and where they are stored. Under the federal personal data protection framework these are not optional questions, and the checklist in what the data protection law asks before you connect a model is the version I use.
The budget line is not legal advice. It is the engineering work of restricting what the assistant can retrieve, and the periodic check that the restriction still holds after six months of people adding documents to the shared drive.
The three numbers to establish before anyone quotes
None of the above can be estimated without these, and all three are knowable in an afternoon from the pilot you already ran.
- Conversations per month, at the ceiling. Not the average. How many people can reach it, multiplied by how often the heaviest tenth of them will use it.
- Turns per conversation. Take fifty real pilot conversations and count. The number is almost always higher than the team believes, and it multiplies everything downstream.
- Documents retrieved per turn. Look at the actual configuration rather than asking. This is the number most often set to a default nobody revisited, and it is the cheapest one to fix.
Those three, multiplied out, give you the run line. Everything else in the budget is either fixed or a person, and both of those are easier to estimate honestly.
A structure that produces a defensible number
When I am asked for an annual figure, I do not produce one. I produce four, because a single number hides the assumptions that will actually determine it.
- Build. One-off. Integration, retrieval, the interface, evaluation before launch.
- Run. Monthly and usage-driven. Model calls, search infrastructure, hosting.
- Keep correct. Monthly and fixed. Content maintenance, sampled review, the evaluation set updated as things change.
- Change. Quarterly. New capabilities, new sources, whatever the second year brings.
Measurement belongs in the same structure. The NIST AI Risk Management Framework treats measuring and managing as continuous functions rather than a launch gate, and that is a budgeting statement as much as a governance one: whatever you decide to watch, you are committing to watch it every month for as long as the assistant runs.
Then state the assumption underneath the run line — conversations per month, turns per conversation, documents retrieved per turn — and show what the number does if each doubles. A budget that survives that question is a budget. One that does not is a quote for a pilot.
The same discipline applies to every part of this: the standing cost is the thing to establish before committing, which is why it is one of the six checks in the technical due diligence I run before anyone buys or builds.
What I would do to halve it
In order of return, before anyone renegotiates a contract.
Fix retrieval first. Most assistants send far too much context. Cutting retrieved passages from twenty to five usually improves answers and cuts the input bill at the same time, which is the rare change with no trade-off.
Route by difficulty. Send the easy majority to a smaller model and keep the capable one for what needs it.
Cache the fixed parts. Long standing instructions belong in a cache, not in every request.
Reduce the turns. Look at the conversations that took six exchanges and find what the assistant failed to establish in the first one. This improves the experience and the bill together.
Then look at the contract. Committed-use pricing is worth discussing once you have a stable usage pattern, and worthless before, because you will be committing to the wrong number.
That order matters. Every one of the first four is engineering you control; the fifth depends on somebody else agreeing. Teams almost always start with the fifth. This is a substantial part of what I am asked to do as an AI consultant and inside fractional CTO work, and it is usually the difference between a project surviving its second budget cycle and quietly being switched off.
If the yearly number is acceptable and you want the thing built, my company builds them: custom AI assistants at Tothiq.
Frequently asked questions
Can you give a figure for a typical assistant?
Not honestly, and anyone who does without asking about your volumes is guessing. The band is enormous — the same functional assistant can differ several-fold on the model line alone, from retrieval discipline and model selection, before you count the people keeping it correct. What I can tell you is the shape: build is one-off and smaller than expected, running costs are dominated by people rather than compute, and the largest single risk is unmodelled usage growth.
Is self-hosting a model cheaper?
Sometimes, at high and steady volume, and it changes what you are buying rather than simply lowering it. You trade a usage-based bill for fixed infrastructure plus the people who run it, so it only pays back above a threshold you should calculate before committing. It is often the right answer for data residency reasons rather than cost ones, and that is a better basis for the decision.
Our vendor quoted a flat monthly price. Is that better?
It is more predictable, which has real value. Read what it covers: usage above a threshold, model upgrades, content maintenance, and what happens when you want a change. A flat price with a low usage cap and change requests billed separately is a variable price with extra steps. Predictability is worth paying for, but only if it is actually predictable.
How do we stop the bill running away overnight?
A hard daily ceiling that stops rather than alerts, per-user rate limits, and a dashboard someone actually looks at weekly. Alerts are not controls — they depend on a person being awake and willing to act. Set the ceiling above your expected peak but below the number that would matter, and treat hitting it as information rather than as a failure.
What is the most commonly forgotten cost?
The person who reads a sample of what it said. It is small, it is easy to cut because nothing visibly breaks when you do, and its absence is how an assistant gives a wrong answer for months. Budget it as a fixed line from the first month, not as something to add once the assistant is proven.