AI Product UX Audit: A 50-Point Checklist

How do you audit the UX of an AI product?

You audit an AI product by running realistic tasks through every place the model touches the user and checking how the interface handles variable output: good answers, slow answers, partial answers and wrong ones. The method is the same as any UX audit, with scope, evidence, severity and fixes, but the checklist is different.

Run a standard UX audit checklist too; this list covers only what it misses: 50 checks in six groups, from the prompt box to human handoff. It's for shipped products. If your AI feature isn't live yet, start with the UX readiness checklist for adding AI to a SaaS product.

Why do generic UX audit checklists miss AI failures?

Generic checklists assume the same input always produces the same screen, and AI features break that assumption. A model can answer the same question well on Monday and badly on Tuesday, take two seconds or twenty, and sound equally sure either way. A checklist that inspects screens can't see any of that.

The failures look different too. In an AI product, the worst failure is often a fluent, well-formatted answer that's wrong, and nothing on the screen signals a problem. The audit has to test the answer, not just the interface around it.

That's why heuristic evaluation alone undercounts AI issues. You need tasks with known correct answers to catch confident mistakes, plus checks on cost, limits and handoff.

How do you run an AI UX audit step by step?

Run the audit as a task-based review with a fixed test set, so findings are repeatable and you can re-test after fixes. The steps below take you from scoping to a ranked report.

1. 1

   Map every AI surface

   List each place the model produces output: chat, inline suggestions, generated documents, summaries, agent actions and background jobs.

2. 2

   Build a test set of 20–40 tasks

   Include easy tasks, hard ones, ambiguous ones, tasks the product should refuse, and tasks where you know the right answer. Write each one down so you can rerun it exactly.

3. 3

   Run each task and record everything

   Capture the input, the wait, the output, any error and what you could do next. Screen recordings help with streaming issues.

4. 4

   Score against the 50 checks

   Mark each check pass, fail or not applicable per surface. A check can pass in chat and fail in an inline suggestion, so score them separately.

5. 5

   Rank by severity, not by count

   Confident wrong answers, irreversible actions without confirmation and surprise charges go first. Cosmetic issues in the prompt box go last.

6. 6

   Write fixes as specific changes

   Each finding gets the exact copy, state or control to add. Use a UX audit report template so engineers can turn findings into tickets.

AI UX audit checklist: prompt input and onboarding

The prompt box is where users decide whether the product understands them, so these checks focus on getting a good first request in. Ten checks:

1. The input says what the AI can do here, in one line, before the user types.
2. Example prompts are real tasks for this product, not generic "Ask me anything" filler.
3. The input shows what context the AI will use (this document, this project, the whole workspace).
4. Users can attach or reference the data the task needs without leaving the flow.
5. Long prompts stay readable: the box grows, and nothing gets cut off silently.
6. Character or token limits are shown before the user hits them, not after submit.
7. Users can edit and resubmit the last prompt without retyping it.
8. Keyboard behavior is clear: Enter sends or adds a line, and the hint says which.
9. A first-time user can get one useful result without reading documentation.
10. The product says plainly what it can't do, so users don't waste their first tries.

Which streaming and response-state checks matter?

Streaming and loading checks test whether users can tell what the AI is doing while they wait. A slow answer is acceptable; an unexplained one isn't. Eight checks, which build on the patterns in loading states and optimistic UI:

1. Something visible happens within a moment of submit, before the first token arrives.
2. Multi-step work shows its current step (searching, reading files, drafting), not one generic spinner.
3. Streaming text doesn't make the page jump or scroll the user away from what they're reading.
4. Users can stop a response mid-stream, and the stop control is visible during the whole stream.
5. A stopped or partial answer is marked as partial, not presented as complete.
6. Screen readers get progress updates without hearing every token, following WCAG 2.2's status messages criterion.
7. Long-running tasks keep going if the user navigates away, and tell them where to find the result.
8. Regenerate keeps the previous answer available instead of replacing it with no way back.

How do you audit AI errors and wrong answers?

Error checks cover the system failing (timeouts, rate limits, outages) and the model failing (wrong or off-topic answers). The second is harder to see and usually more severe. For message copy, see error state UX. Eight checks:

1. Timeouts and provider outages show a plain-language message with a retry, not a raw error code.
2. The user's prompt survives every error, so nothing has to be retyped.
3. Rate-limit messages say when the user can try again.
4. On test tasks with known answers, wrong answers are flagged or hedged rather than stated with full confidence.
5. The AI declines out-of-scope requests clearly and suggests what it can do instead.
6. Users can correct an answer in place (edit, refine, "that's wrong because…") and the correction sticks.
7. Feedback controls (thumbs, report) ask for a reason in one tap, and say what happens with the report.
8. Generated content that will be published or sent is marked for review before it leaves the product.

What trust and transparency checks should an AI UX review include?

Trust checks test whether users can see why the AI answered as it did, so they trust it the right amount, not more. Eight checks:

1. Answers based on documents or the web cite their sources, and each citation opens the exact passage.
2. Citations actually support the sentence they're attached to (spot-check ten on your test set).
3. The product distinguishes retrieved facts from generated suggestions.
4. Users can see what data the AI read for this answer, and can exclude something.
5. AI-generated content is labelled where it could be mistaken for human-written content.
6. Privacy is explained at the point of use: whether prompts are stored and whether they train models.
7. Tone stays consistent with the product, and the AI doesn't claim feelings or certainty it can't have.
8. Uncertainty is shown in words ("I couldn't find this in your files") rather than hidden.

How do you check cost, usage and limit visibility?

Cost checks test whether users know what an AI action costs in credits, quota or money before they take it. A surprise limit mid-task feels like a broken product. Eight checks:

1. Users can see remaining credits or quota from the place they use the AI, not only in billing settings.
2. Expensive actions (long documents, large batches, premium models) show their cost before running.
3. Warnings appear before the limit, not at the moment work is blocked.
4. Hitting a limit saves the user's work and explains the options: wait, upgrade or reduce the task.
5. Model choice, if offered, explains the trade-off in plain words (faster, cheaper, more thorough).
6. Team admins can see and cap usage per person or workspace.
7. The pricing page and the in-product meter use the same units and names.
8. Failed or stopped generations don't silently consume credits, or the product says that they do.

How do you audit human handoff and user control?

Handoff checks test whether users can take control back from the AI, and reach a person when the AI can't help. They matter most in agent features that act on the user's behalf, which our guide to human-in-the-loop design covers in depth. Eight checks:

1. Actions with side effects (sending, deleting, paying, publishing) ask for confirmation and show exactly what will happen.
2. Every action the AI takes on its own is listed in a log the user can review.
3. Users can undo AI actions, or the product says clearly when an action can't be undone.
4. Users can set how much the AI does on its own, per action type.
5. The AI hands off to a human when it fails repeatedly or the user asks, without a dead-end loop.
6. Handoff passes the conversation and context along, so the user doesn't repeat themselves.
7. Users can switch off the AI feature and still finish the task manually.
8. Shared or team outputs show which parts the AI produced and who approved them.

If the problems run deeper than individual screens, our guide to AI product UX covers the principles behind these checks, and our product design service redesigns AI products end to end, with words, design and code from one team.