AI Problem-Framing Worksheet
Scope the use case before you build the system. These eight questions help you agree on the business outcome, compare approaches, and decide what evidence would make you continue, change course, or stop.
One page. No email wall. By Rajiv Shah, author of AI Problem Framing for AI Practitioners.
Use it before you scope an AI project
Bring this worksheet to a kickoff with the person who owns the outcome and someone who will use the output. It works for an ML model, a generative AI application, or an agent. Start with the decision someone needs to make, then consider what to build.
Write a short answer to each question. When you don't know, record the assumption and how you'll test it. Agree on the metric and stop criterion with the owner before building. Revisit the answers when new evidence changes the plan.
The questions below preserve the original worksheet's three sections. They cover decisions from the GOATS Loop: Goal, Operating Assumptions, Alternatives, Trade-offs, and Signals. Use the copyable template for project notes or the PDF for a meeting.
Section 1: Define the problem
1. WHY
What business outcome are we trying to improve, and what does the problem cost today?
Time, money, errors, or churn. An estimate is fine. Is it more than the system costs to build, run, and be wrong?
What is the metric? Which candidate metric did we reject, and why?
Same goal, different metrics, different systems. The metric you pick is the system you get.
2. WHAT
What are we proposing to build? Is that a problem or a solution?
If you named a tool (chatbot, RAG, agent), write the problem underneath it instead.
What are we explicitly NOT building?
Three non-goals. An unstated non-goal is an invitation the system will accept.
3. WHO + ATOMIC UNIT
Who will use it, and who owns the decision it supports?
The user acts on the output. The owner's budget moves if it works. Name both. Then list who else cares: different seats often want different metrics.
What does the system decide about, one at a time?
The atomic unit is one ________________________
Examples: one customer-month, one support ticket, one passage, one task step.
Section 2: Define the approach
4. HOW
How will it work? What simpler approach should we compare it against?
5. TRADE-OFFS
What are we optimizing for, and what are we willing to give up?
6. ASSUMPTIONS
What has to be true for this approach to work?
Include the signal: does the information this decision needs exist, and is it available at decision time?
Section 3: Define the evidence
7. SIGNALS
What early evidence would tell us we are on track or off track?
Start from the top three ways this fails in production. Off-track signals come from failure modes. Write down both on-track and off-track evidence.
8. GOOD ENOUGH
What is good enough to test? Good enough to ship? What number makes us stop?
Pre-commit the stop with the owner: a number and a time horizon, agreed before building. Give test, ship, and stop their own criteria.
Run the four quick tests
Run these before you call it done. These are the worksheet's framing checks; the diagnostics for a running system come later.
- The Oracle Test: If this system was 100% accurate right now, what would still fail?
- The Perfect-Unit Check: If every unit came back perfect, would the goal move? If not, the unit is wrong.
- De-escalation: How would you solve this without any AI, and would it be good enough?
- The Feynman Test: Can you explain what this is for using no AI words at all?
Completed example: a support-policy assistant
This is an illustrative adaptation of the composite policy-bot example in Chapter 2 of the free book and the Signals Canvas. It is a plan, not a customer result. The cost estimate, non-goals, approach, and success and stop thresholds come from those materials. The rejected metric, test gate, and week-one target below are added to show how a team could fill the remaining fields. Choose targets for your own workflow with its owner.
1. Why this project exists
Support agents need to tell customers which policy applies without repeatedly interrupting the policy team. In the book's composite, the policy team spends about a third of its week answering the same questions by chat. Compare the value of that time with the build cost, running cost, and cost of a wrong answer before funding the project.
Use answer accuracy on a weekly production sample, with fewer routine questions reaching the policy team as the outcome check. Reject deflection rate as the main metric: keeping a question away from a person doesn't establish that the answer was right. Track escalations as a guardrail so a higher accuracy score doesn't hide more work for the policy team.
2. What is in scope
The problem is finding the right policy and giving the support agent an answer they can check before sending. The proposed solution retrieves policy documents and drafts an answer with citations. The three non-goals are a customer-facing bot, access to account data, and decisions about policy exceptions.
3. Who acts, and on what unit
Support agents use the answer. The support director owns the outcome. The policy team cares about interruptions and policy accuracy, while engineering owns the retrieval and document-update process. The atomic unit is one policy question from one support agent, graded as a complete answer with its supporting citations.
4. How to approach it
Compare keyword search plus the existing FAQ template with one-shot retrieval and a cited draft. Keep a support agent reviewing the answer before it reaches a customer. Evaluate both approaches on the same questions. Add iterative retrieval as an alternative if questions need information that one retrieval call can't find.
5. What to trade off
Prioritize checkable answers and a small initial build. Accept human review and less autonomy. One-shot retrieval is cheaper and simpler than iterative retrieval, but it risks missing an answer spread across multiple documents. Record that limitation and test it explicitly.
6. What has to be true
The policy documents must be current and accessible when the question arrives. One retrieval call must find the documents needed for enough questions to justify the approach. Support agents must read the citations before sending. Sample questions in the form agents will actually submit, including questions that span multiple policies, and check the source documents with the policy team. Treat each of these as an assumption to test.
7. What early evidence to look for
The top three failure modes are missing a needed document, retrieving a superseded policy, and agents losing trust after confident wrong answers. In week one, label the documents needed for a 50-question sample and check whether the top five results include all of them. For this illustration, agree on at least 45 of 50 as the retrieval target; missing it calls for investigation before the pilot expands. Review document versions and watch whether agents check citations or go back to the policy team's chat.
The on-track evidence is retrieval meeting that target, current sources, and agents using the cited drafts. Missing documents, outdated policies, or agents bypassing the assistant are off-track evidence.
8. What earns a test, a launch, or a stop
Test: Run a supervised pilot once the source documents have been checked and the week-one retrieval target is met. Keep the FAQ fallback available and require support agents to review every draft.
Ship: By day 60, answer accuracy is above 80% on the weekly production sample, routine policy questions to the policy team fall, and escalations fall rather than rise. In this example, shipping still means an internal assistant with human review.
Stop: Accuracy under 70% after two tuning cycles means revisiting the approach with the support director. Switch to keyword search plus the FAQ template while deciding whether a different approach is justified. Agree on what counts as a tuning cycle and its review date before starting. Agents routing around the assistant is another reason to review the plan, even if the accuracy number passes.
Check the example before building
The Oracle Test exposes the trust problem: even a correct answer won't help if the agent can't check it. The Perfect-Unit Check asks whether correct, cited answers to individual policy questions reduce interruptions; if the unanswered work is really exception approval, change the unit or scope. De-escalation keeps keyword search and the FAQ template in the comparison. The Feynman version is: help support staff find the right policy and check their answer before replying to a customer.
Copy the worksheet
Copy the text below into your project notes and replace each answer line. The copy button copies only this template. Keep the unknowns visible until you have evidence.
AI FRAMING WORKSHEET
Scope the use case before you build the system.
SECTION 1: DEFINE THE PROBLEM
1. WHY
What business outcome are we trying to improve, and what does the problem cost today?
Answer:
What is the metric? Which candidate metric did we reject, and why?
Answer:
2. WHAT
What are we proposing to build? Is that a problem or a solution?
Answer:
What are we explicitly NOT building?
Three non-goals:
3. WHO + ATOMIC UNIT
Who will use it, and who owns the decision it supports?
Users, owner, and other stakeholders:
What does the system decide about, one at a time?
The atomic unit is one:
SECTION 2: DEFINE THE APPROACH
4. HOW
How will it work? What simpler approach should we compare it against?
Answer:
5. TRADE-OFFS
What are we optimizing for, and what are we willing to give up?
Answer:
6. ASSUMPTIONS
What has to be true for this approach to work?
Does the information exist and is it available at decision time?
Assumptions and tests:
SECTION 3: DEFINE THE EVIDENCE
7. SIGNALS
What early evidence would tell us we are on track or off track?
Top three failure modes:
On track:
Off track:
8. GOOD ENOUGH
What is good enough to test? Good enough to ship? What number makes us stop?
Test:
Ship:
Stop (number, time horizon, owner):
QUICK TESTS
The Oracle Test: If this system was 100% accurate right now, what would still fail?
Answer:
The Perfect-Unit Check: If every unit came back perfect, would the goal move? If not, the unit is wrong.
Answer:
De-escalation: How would you solve this without any AI, and would it be good enough?
Answer:
The Feynman Test: Can you explain what this is for using no AI words at all?
Answer:Take the answers into the GOATS Loop
Use the full framework to compare alternatives and test the assumptions you wrote down. Chapter 2 of the free book walks through those decisions in depth. Use the Signals Canvas to agree on success, stop, and early-warning signals with the owner.
Download the one-page worksheet or browse all free worksheets and templates.