Building an AI MVP: what to build first, and what to leave out
What belongs in a first AI product release: one workflow, an evaluation set, a hosted model, human review where it matters, and logging. Plus what to leave out.
An AI MVP has the same job as any MVP: get a real product in front of real users quickly enough to learn something. The difference is that AI features fail in quieter, stranger ways than ordinary software, so a few things that would be "later" in a normal product need to be in from day one.
One workflow, done properly
Pick the single workflow that proved itself in the idea phase and make it excellent. One input, one useful output, one clear next step for the user. A product that does one job reliably beats one that does five jobs some of the time.
Build the evaluation set before the feature
Before tuning prompts or adding features, collect a set of real examples with the answer you'd expect, and a simple way to score results: a rubric, a checklist or an exact match where that's possible. Run it every time you change a prompt, a model or the data the model sees.
This is the AI equivalent of automated tests. Without it, every change is a guess, and you'll find regressions from customers instead of from a report.
Start with a hosted model
For almost every first release, a hosted model through an API is the right call. It's fast to build on, quality is high, and you can switch providers later if you keep the model behind a small interface in your code. Training or fine-tuning your own model is a later decision, made when your evaluation set shows a gap prompting can't close.
Keep a person in the loop where errors cost money
If a wrong output is expensive or embarrassing, design the product so a person approves it: a draft to edit, a suggestion to accept, a queue to review. Make that review fast: show the source the answer came from, highlight what the model was unsure about, and make accepting or fixing one click.
Human review isn't a failure of the AI. It's often the product: people buy the time saved, not the autonomy.
Ground answers in your own data
When the model needs facts about your business or your customer, give it the relevant documents at the moment it answers (often called retrieval-augmented generation) and show citations. Respect permissions: a user should only ever see answers drawn from data they're allowed to see.
Log everything from day one
Store each request, the context the model saw, its output, the cost and what the user did next: accepted, edited, ignored. That log is your best source of new evaluation examples, your early warning for quality drift, and the data you need to set a sensible price.
What to leave out
- Custom model training, until evaluations prove you need it.
- Multi-agent systems for a task one well-structured call can do.
- Ten integrations. Ship the one your first customers use.
- A chat box when a button would do. Open-ended chat is hard to evaluate and hard for users to learn; a focused action is usually better.
- An admin panel for everything. A spreadsheet and a script are fine for month one.
A short launch checklist
- The evaluation set passes at a level you'd be happy to show a customer.
- Failures degrade gracefully: a clear message, never a confident wrong answer with no source.
- You know the cost per task and have a monthly spend alert.
- Data handling is written down: what's stored, where, and for how long.
- Someone owns watching the logs for the first few weeks.
That's the shape of the build phase at DevTrust. AISiteFix is a live example: an AI product designed, built and shipped end to end.