Production-readiness checklist

Before an AI agent goes live, test whether it can fail safely.

Use this checklist to assess analytics agents, SQL copilots, RAG assistants, and enterprise AI workflows before users start trusting their answers.

Does the agent ask a clarification question when the request is underspecified?

Does it choose the right tool, or does it answer from memory when it should call a tool?

Are filters, dates, IDs, user scopes, and query parameters correct?

Does it respect access controls at retrieval time and answer time?

Does the agent retrieve the right source material, and does it know when context is weak?

Does it use the company’s actual definitions for revenue, churn, active users, margin, conversion, and retention?

For analytics agents, do joins, aggregation grain, time windows, and filters match the intended question?

Can the agent show why it answered the way it did, and whether the evidence supports the claim?

Does it avoid turning correlation into causation?

If a tool call fails, does it retry, ask for help, or disclose uncertainty instead of hallucinating a result?

If the user changes scope, does the agent update its assumptions or keep using stale context?

Does the agent still pass critical workflows after prompt, model, retrieval, or tool changes?

Audit output

What a reliability audit should produce.

A useful audit gives the product, engineering, and data teams evidence they can act on, not just a vague quality opinion.