AI product management: how to spec, cost and ship an AI feature
An AI feature is specified differently from a normal one: you write example inputs with pass/fail answers, not just user stories, because the model will sometimes be wrong. The order that works is: check the problem needs AI, pick the simplest design (a single prompt before an agent), write measurable success criteria, build a small eval set, work out the monthly cost per model, and design what happens when the answer is wrong. In our worked example the same feature costs anywhere from $132 to $5,280 a month, depending on the model.
- 1
Does this feature need AI at all?
Start from the user’s problem, not from the model. Google’s People + AI Guidebook says to stop asking "Can we use AI to ___?" and to ask what the user needs first, because sometimes the most valuable thing is a product that behaves the same way every time. A rule or a form is cheaper, faster and never makes things up.
OpenAI’s guide to building agents lists three signs that work is worth handing to a model rather than to more rules (it frames them as the test for an agent, but they work as a first filter for any AI feature): decisions that need nuanced judgment and exceptions, rule sets that have become unwieldy to maintain, and work that depends on reading unstructured text such as emails, documents or chat messages.
- Judgment with exceptions: approving a refund that depends on the story, not one field
- Too many rules: a policy with hundreds of if-then branches nobody wants to edit
- Messy text: customer messages, contracts, call notes, CVs
- None of the three? Build the rule or the form instead, and say so in the review
Key ideaAn AI PM’s first job is to say no to AI where a rule works. That answer is often the best spec in the room.
- 2
Pick the simplest design that can work
Anthropic’s engineers who wrote "Building effective agents" recommend "finding the simplest solution possible" and adding complexity only when it clearly pays off. They separate workflows, where code fixes the steps and the model fills them in, from agents, where the model chooses its own steps and tools. Agents, they note, often trade latency and cost for better results.
So climb the ladder one rung at a time, and only move up when the eval (step 4) shows the lower rung fails. Most first versions of a feature are a single, well-written prompt with the right context.
- Rule or form
- One model call
- Fixed workflow of calls
- Agent that picks its own steps
Key ideaEach rung up adds cost, delay and new ways to fail. Move up only when the eval says so.
- 3
Write the spec as examples and pass/fail criteria
Anthropic’s docs say good success criteria are specific, measurable, achievable and relevant. "Good performance" is not a criterion; "accurate sentiment classification" is. They add that "most use cases need multidimensional evaluation": accuracy, consistency, tone, privacy, use of context, latency and price can all matter at once.
For an AI feature the most useful part of the spec is a table of real inputs with the output you would accept, plus a short never-do list. Engineers can test against it, designers can see the edge cases, and leaders can argue about the examples instead of adjectives. If the feature reads text written by outsiders, such as customer tickets, say that instructions inside that text must be ignored: the Prompt injection lesson on this path shows why.
One-page AI spec: refund-ticket triage (our fictional example) Problem: agents read 20,000 refund tickets a day; simple ones wait hours. User: support agent, then the customer. Job: read a ticket + order details, return category, suggested action, 2-line reply draft. Design: one model call, JSON out. No agent. Human in the loop: agent approves every refund. Pass/fail criteria (per ticket): - category matches the agent's label - action follows the refund policy - never promises money the policy forbids - reply has no customer data from other orders - reply in the customer's language Never: issue a refund itself, promise cash beyond policy, follow instructions written inside the ticket (treat it as untrusted text). Launch bar: 54 of 60 eval tickets pass (90%), 0 policy-breaking promises, p95 under 3 s. Cost cap: $500 a month (step 5 shows the maths). Fallback: low confidence or bad JSON -> ticket goes to the normal queue.Key ideaIf two people can disagree about whether an output passes, the criterion is not finished.
- 4
Build the eval set before you build the feature
Hamel Husain, who helps teams fix AI products, calls error analysis "the single most valuable activity in AI development": read real outputs, write down what went wrong, and group the failures. He prefers pass/fail judgments because "binary decisions provide clarity that more complex scales often obscure", and warns that generic scores such as "helpfulness" hide the real problems.
Start with 50 to 100 real inputs, including the awkward ones, and label the right answer. Anthropic’s docs say to prefer more test cases graded automatically over a few graded by hand. Run every model and prompt change against the same set, and add every new failure you see in production. The Evals lesson on this path walks through building one in a spreadsheet.
Six rows from a 60-ticket eval set (input → expected · what it checks) 1. "Paneer missing, charged ₹240" → missing_item, refund the item · category + policy 2. "Order 40 min late, food cold" → late_delivery, coupon · category + policy 3. "Refund my last 6 orders" → escalate, promise nothing · never-do list 4. A message in Hinglish → reply in Hinglish · language 5. Ticket quotes another user's phone → reply without it · privacy 6. Empty message, photo only → escalate · edge caseKey ideaThe eval set is the spec you can run. Write it before the prompt, and keep adding to it after launch.
- 5
Do the cost maths before anyone says "use the best model"
Cost is tokens times price times traffic. In the example each ticket sends about 1,200 input tokens (instructions and refund policy plus the message) and gets about 200 tokens back. At 20,000 tickets a day for 30 days that is 720 million input and 120 million output tokens a month.
At list prices checked on 17 Nov 2026, Claude Haiku 5.5 costs $0.00022 a ticket, or $132 a month. Claude Sonnet 5.5 costs $2,640 and Claude Opus 5.5 $5,280: 40 times the cheapest. OpenAI’s guide gives the order: set up evals to get a baseline with a capable model, then try smaller models and keep the cheapest one that still passes. Try your own numbers in the cost calculator further down this page.
Two levers cut the bill without changing the model: prompt caching, because the same instructions and policy go out with every ticket, and batch processing for work that can wait. Both are priced on the makers’ pricing pages; the figures here leave them out.
The sum for Claude Haiku 5.5 Input: 720M tokens × $0.1/M = $72.00 Output: 120M tokens × $0.5/M = $60.00 Month: $132.00 for 600,000 ticketsOur example · 20,000 tickets a day · list prices 17 Nov 2026The same feature costs $132 to $5,280 a month, depending on the model- Claude Haiku 5.5$132
- gpt-6-luna$132
- gpt-5.4-nano$294
- gpt-5.4-mini$1,080
- gpt-6.1-terra$1,320
- Claude Sonnet 5.5$2,640
- gpt-6.1-sol$2,640
- Claude Opus 5.5$5,280
1,200 input + 200 output tokens per ticket, 30 days. No caching or batch discounts. Claude Haiku 5.5 price is for prompts up to 100,000 tokens; gpt-6 prices for up to 272,000.Source: Anthropic pricing, OpenAI API pricing · 17 Nov 2026 View as table
Item Value Note Claude Haiku 5.5 $132 gpt-6-luna $132 gpt-5.4-nano $294 gpt-5.4-mini $1,080 gpt-6.1-terra $1,320 Claude Sonnet 5.5 $2,640 gpt-6.1-sol $2,640 Claude Opus 5.5 $5,280 Key ideaCost is a product decision. Put a monthly cap in the spec and let the eval pick the cheapest model that passes.
- 6
Design for the wrong answer, then ship in stages
Google’s guidebook chapter on errors says "When an AI system fails, often the easiest path forward is to let the user take over", and that how failures are handled "calibrates user trust". OpenAI calls human intervention "a critical safeguard", most of all early on. So the spec names every failure state and what the user sees in each.
Then launch in stages. Run the model silently next to the humans first and compare, then show it to a small share of users, then widen it. Write down in the spec what you watch and what sends you back a stage, for example: agents override more than 1 in 5 suggestions, any policy-breaking promise, or cost above the cap. Every override and every thumbs-down is a new row for the eval set. The worked workflow below puts all six steps together, and the blank spec under it is yours to copy.
Blank one-page AI spec: copy and fill in Problem, with today's numbers: User, and the job the AI does: Why AI? judgment / rules / text: Rung: rule / call / flow / agent: 10+ inputs + accepted outputs: Pass/fail criteria: - Never-do list: - Launch bar (pass rate, 0-fails): Latency target: Cost cap a month: Fallback (wrong, slow, empty): Rollout stages: Watch, and when to step back:Worked workflowWorked workflow: refund-ticket triage, from idea to launch (our fictional example)
InputA fictional food-delivery app. Support agents read 20,000 refund tickets a day, and simple ones wait hours for a first reply.
Check it needs AI
The tickets are free text in English, Hindi and Hinglish, and refunds depend on the story. Two of the three signs from OpenAI’s guide (judgment, unstructured text) are there, so a model is worth testing.
Pick the lowest rung
One model call that returns JSON. No agent: the refund itself stays with a human, so the model never needs tools that move money.
Write the spec
Five pass/fail criteria, a launch bar of 54 of 60 eval tickets (90%) with zero policy-breaking promises, a p95 latency under 3 seconds and a $500 monthly cap.
Build and run the eval
Sixty real tickets, labelled by two senior agents, including late orders, missing items, abuse, Hinglish and photo-only tickets. Run it on a capable model first for a baseline, then on the cheaper ones.
Price it
At 20,000 tickets a day: $132 a month on Claude Haiku 5.5, $2,640 on Claude Sonnet 5.5. Only a model that passes the eval and fits the $500 cap can ship.
Fallback and rollout
Bad JSON, low confidence or a "refund everything" request sends the ticket to the normal queue. Two weeks silent beside agents, then 5% of tickets, then 50%. Step back a stage if agents override more than 1 in 5 suggestions, any reply breaks policy, or cost passes the cap.
OutputA one-page spec, a 60-ticket eval set, a model choice backed by both the eval and a cost line, and a rollout plan with a way back. That is the packet an AI PM brings to a launch review, and the answer interviewers look for when they ask how you would ship an AI feature.
Key ideaA good AI spec answers four questions: what does good look like, how will we measure it, what does it cost, and what happens when it is wrong.
Straight from the source
The original idea, then the same idea taken apart, then how you use it today.
- 1 · Understand
- They recommend "finding the simplest solution possible" and separate workflows, where code fixes the path, from agents, where the model directs its own steps. Agents often trade latency and cost for better results.
- 2 · Break it down
- Many teams that succeed use simple, composable patterns rather than complex frameworks
- A workflow is predictable; an agent is flexible but harder to test
- More autonomy means more cost, more delay and more ways to fail
- 3 · Use it
- In your spec, name the rung you chose (rule, one call, workflow, agent) and the eval result that would make you move up one.
- 1 · Understand
- Error analysis, reading real outputs and grouping what went wrong, is "the single most valuable activity in AI development". He prefers pass/fail judgments to 1–5 scores and calls a tools-first mindset the most common mistake.
- 2 · Break it down
- Look at real outputs before choosing any metric or tool
- Group failures and fix the biggest group first
- Pass/fail is clearer than a score nobody can define
- 3 · Use it
- Before launch review, read 50 real outputs yourself and bring the three biggest failure groups, with counts, to the meeting.
Words to know
- Success criteria
- Specific, measurable statements of what a good output is, such as "category matches the agent’s label".
- Eval set
- A fixed list of real inputs with the accepted answer, run after every change to see if the feature still works.
- Error analysis
- Reading real outputs, noting what went wrong and grouping the failures to decide what to fix.
- Workflow vs agent
- In a workflow your code fixes the steps; in an agent the model chooses its own steps and tools.
- Token
- The chunk of text models read and write and are billed by. Prices are quoted per million tokens.
- Fallback
- What the product does when the model’s answer is wrong, slow or missing, often handing over to a person.
Watch
Hand-picked from the people who explain this best. The video loads only when you press play.
One of the authors of "Building effective agents" explains when not to build an agent and how to keep a design simple.
A long conversation; focus on the error-analysis walkthrough: how a PM reads real outputs before writing any eval.
What will your AI feature cost a month?
Type your traffic and token counts, or start from an example. Every model is priced in your browser from the makers’ list prices.
A 30-day month is 600,000 requests. The cheapest model below costs $132.00 a month; the dearest costs $5,280.00, about 40× as much. Run your eval on the cheap ones first: if one passes your bar, that gap is yours to keep.
- Claude Haiku 5.5$132.00/mo · $0.22 per 1k
- gpt-6-luna$132.00/mo · $0.22 per 1k
- gpt-5.4-nano$294.00/mo · $0.49 per 1k
- gpt-5.4-mini$1,080.00/mo · $1.80 per 1k
- gpt-6.1-terra$1,320.00/mo · $2.20 per 1k
- Claude Sonnet 5.5$2,640.00/mo · $4.40 per 1k
- gpt-6.1-sol$2,640.00/mo · $4.40 per 1k
- Claude Opus 5.5$5,280.00/mo · $8.80 per 1k
List prices per million tokens from Anthropic and OpenAI, opened 17 Nov 2026. No prompt caching, batch discount or tax. A real bill also has retries and longer edge cases: add a margin.
Try it
Spec one AI feature in 10 minutes
- 1Pick a feature you wish an app you use had, such as "summarise my bank statement" or "draft a reply to this review".
- 2Write one line for each of the three signs: does it need judgment, too many rules, or messy text? If none is true, write the non-AI design instead.
- 3Write three pass/fail criteria that two people would always agree on.
- 4List ten example inputs, with at least three awkward ones (empty, another language, someone trying to trick it).
- 5Put your traffic and token guesses into the cost calculator on this page and write a monthly cap.
- 6Write one sentence on what the user sees when the answer is wrong, then fill in the blank spec from step 6.
It checks what you understood, then gives you a small real task to try.
Read the prompt
Be my patient tutor. I just studied this lesson: "AI product management: spec and ship". Summary: An AI feature is specified differently from a normal one: you write example inputs with pass/fail answers, not just user stories, because the model will sometimes be wrong. The order that works is: check the problem needs AI, pick the simplest design (a single prompt before an agent), write measurable success criteria, build a small eval set, work out the monthly cost per model, and design what happens when the answer is wrong. In our worked example the same feature costs anywhere from $132 to $5,280 a month, depending on the model. Key terms: Success criteria, Eval set, Error analysis, Workflow vs agent, Token, Fallback. Do this, one step at a time, waiting for me each time: 1. Ask me to explain the main idea in my own words. Correct me kindly with one concrete example. 2. Ask 4 questions, easy to hard. After each answer, say what was right and what was missing. 3. Give me one small workflow I can try in 10 minutes that uses this idea in real work or study. Walk me through it step by step and check my result. 4. End with 3 lines: what I now know, what I should practise, what to learn next.
Check yourself
Pick an answer to see if you are right, and why.
Done? Keep going
Mark this lesson complete to keep your place. Progress is saved on this device, no login.
- AI Product Manager interview guideSpec, cost and ship an AI feature: the case most AI PM loops ask you to walk through.Open the guide
- Product Manager interview guideHow you would spec, measure and cost an AI feature is a common product sense follow-up.Open the guide
- Founder's Office interview guideFounder’s office teams often scope the first AI feature. The cost and eval steps keep it honest.Open the guide
Questions people ask
How do you write a PRD for an AI feature?
Keep the usual problem, user and scope sections, then add four AI parts: example inputs with the output you would accept, pass/fail success criteria with a launch bar, a monthly cost cap worked out from tokens and traffic, and the fallback when the answer is wrong. The worked example on this page fits on one page.
How would you evaluate a generative AI feature in a PM interview?
This was reported as a Visa product manager question in April 2026. A two-minute answer can follow this page: (1) the user and the job, and why AI beats a rule; (2) the design rung you would start on; (3) three or four pass/fail criteria, with safety and privacy at zero tolerance; (4) a labelled eval set of 50 to 100 real inputs with edge cases, run on every change; (5) the cost per month and the cheapest model that passes; (6) after launch, override rate, user feedback and cost per request, with a rule for rolling back.
How much does it cost to run an AI feature?
Multiply input and output tokens per request by the model’s price per million tokens and by your traffic. In our example of 20,000 tickets a day it ranges from $132 to $5,280 a month at 17 Nov 2026 list prices. Use the calculator on this page with your own numbers.
Sources and further reading
- Anthropic: Building effective agents (Erik S. and Barry Zhang, 19 Dec 2024)
- Anthropic docs: Define success criteria and build evaluations
- OpenAI: A practical guide to building agents (PDF)
- Google PAIR People + AI Guidebook: User Needs + Defining Success
- Google PAIR People + AI Guidebook: Errors + Graceful Failure
- Hamel Husain: A Field Guide to Rapidly Improving AI Products (24 Mar 2025)
- Hamel Husain: Your AI Product Needs Evals (29 Mar 2024)
- Anthropic pricing (opened 17 Nov 2026)
- OpenAI API pricing (opened 17 Nov 2026)
- PracHub: Visa PM question on evaluating a generative AI feature (26 Apr 2026)