The gap between an AI demo and an AI product
Almost any AI feature can be made to work once.
The hard part is the ten-thousandth input: a document formatted unlike any example in the prompt, at a cost per request that still makes sense at volume. That is where most AI features stall.
The work that closes the gap is unglamorous, and it belongs in the build rather than a phase that follows it:
- An evaluation set built from real inputs, with known-correct outputs.
- Accuracy scored before and after every prompt, model or parsing change.
- Control over how much text is sent to the model each time, so speed and cost stay predictable.
- An interface where a wrong answer is recoverable, not silently trusted.
The failure mode that costs most is not a feature that never works. It is one that ships, works, and then decays. Models behave differently between versions, providers retire the version you were using, and real inputs drift away from the examples the prompts were written against.
Without measurement, none of that is visible until a customer complains — by which point trust in the feature is gone.
What we build
Retrieval-augmented generation
RAG is the right shape when a model needs to answer from your data rather than from what it learned in training. Most of the engineering is in finding the right material to hand it, not in the writing:
- How documents are chunked.
- Which embedding model is used.
- Whether vector search alone is enough, or hybrid keyword search.
- How results are ranked before they reach the model.
We work with PostgreSQL and pgvector, Pinecone and equivalents, chosen on how much data there is and what you already run. We measure how good the retrieval is separately from how good the answer is, so when something is wrong it is clear which half caused it.
Retrieval quality also decides cost. Returning four relevant passages instead of forty cuts both the bill and the wait, and usually improves the answer at the same time, because models reason better over less noise.
Where the set of documents is small enough, the honest answer is that none of this is needed and everything fits in a single prompt. That recommendation costs us work and saves the client money. We make it anyway.
Document intelligence
Document intelligence covers extraction, classification and comparison over documents that are inconsistent by nature — contracts, reports, statements, filings.
A contract review system, for example, reads an incoming agreement, flags non-standard clauses, and reports deviations from a written playbook in minutes rather than days. We have written up how we approach that class of system in our NDA review agents note.
The engineering challenge is never summarisation; it is being right about the things that carry consequences, and staying right as the standard being compared against changes.
This is where evaluation pays for itself. The correct answer for a given document is knowable, so accuracy can be measured rather than argued about. A change that improves one document class while quietly degrading another gets caught before release.
Agent workflows
Agents are worth building where a task genuinely requires several steps, tools and decisions, and worth avoiding where a single well-specified call would do.
Where they fit, the design questions are which tools the agent may call, how failure and retry are handled, what the cost ceiling per run is, and where a human sits in the loop. We are conservative here on purpose: an agent that occasionally takes a wrong action confidently is worse than a workflow that asks.
Cost matters as much as design. An agent that loops has no natural stopping point, so limits on steps, spend and elapsed time are part of the specification rather than an afterthought.
We also prefer agents whose working you can see. If you can follow how it reached an answer, you can judge it. If it only hands you a conclusion, you either take it on faith or throw it away — there is nothing in between.
Choosing a model, and being able to change it
You should not be locked into one AI provider.
Which model a feature runs on is a practical decision, not a loyalty one. It comes down to three questions: what it costs per request, how quickly it answers, and whether your data is allowed to leave your country or your own network. That last question usually settles it — if the data cannot go to a US provider, nothing else about the choice matters.
Providers change their prices, and better models arrive every few months. You can only act on either if switching is cheap. So the code that talks to a provider lives in one place. The rest of the product neither knows nor cares which model is behind it — changing provider is a small job, not a rebuild.
Switching is only safe if you can show the new model is at least as good. That is what the evaluation set is for: run both against the same real examples and compare the scores.
You own the prompts and the evaluation sets, not just the source code. Those are the parts that took longest to get right, and the parts that would cost the most to recreate somewhere else.
Where this fits in an engagement
AI features are usually scoped during a Discovery Sprint, where the proof-of-concept deliberately targets the AI component because it carries the most technical risk.
They are then built as part of an MVP build and maintained under an ongoing retainer, since model behaviour, pricing and usage patterns all shift after launch. You own the code, the prompts, the evaluation sets and the infrastructure configuration — 100%, from the first commit. See ProfileIQ for how this works in production.
Sometimes the answer is no. Some ideas do not need a model at all — a query, a rule or a well-designed form serves them better. Others need one but cannot live with how often today’s models still get it wrong.
We would rather say that in week one of a sprint than build something that demonstrates well and quietly loses your users their confidence in the product.