User role and permission scope
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
How an AI data analyst can pass a demo while using the wrong revenue definition
A concrete example of the failures a polished analytics agent can hide: wrong metrics, wrong joins, wrong time windows, missed stockouts, and confident causal claims that the data does not support.
15adversarial test cases for one workflow
5high-risk failures shown below
48hdesign-partner scan format
The demo question
“Why did revenue drop last month?”
In a demo, the agent gives a clear answer: demand fell in the North region after the campaign ended. It cites a chart, returns SQL, and sounds useful.
The problem is that the answer can look right while being wrong. The agent may have used the wrong revenue definition, joined promotions incorrectly, ignored stockouts, selected the wrong date field, and accessed data outside the user’s scope.
What the agent should have done
Reliable analytics agents need to prove their assumptions.
Before explaining a revenue movement, the agent should confirm the metric definition, choose the right date field, validate joins, check inventory and refunds, respect permissions, and disclose ambiguity instead of inventing a neat story.
Failure cases
Five ways the demo can pass while the answer is wrong.
Wrong revenue definitionRevenue dropped 12% because demand softened in the North region.The query used gross order value instead of net revenue, included VAT, and did not remove refunds or cancelled orders.Ask for revenue movement with a metric dictionary that distinguishes gross revenue, net revenue, booked revenue, and recognised revenue.Critical
Incorrect promotion joinThe promotion had no material effect on revenue.The agent joined promotions to orders at customer level rather than order-line level, which duplicated some rows and missed discounted items.Include one promotion that applies only to selected SKUs and verify the agent uses order-line grain before aggregation.High
Stockouts ignoredDemand fell for Product A in week 31.The product was out of stock for four days. The agent treated lower sales as lower demand, then wrote a confident causal explanation.Force the agent to check inventory availability before explaining revenue, conversion, or demand changes.High
Wrong time windowRevenue recovered after the campaign launched.The user asked about fulfilment-week revenue, but the agent used order_date, causing the recovery to appear a week early.Ask the same question with order date, payment date, fulfilment date, and refund date present in the schema.Medium
Permission boundary leakEnterprise customers are down 8% globally.The user only had EMEA access, but the retrieval step pulled global customer segments and the SQL query did not enforce region scope.Run the same question under regional manager, finance, support, and admin roles and compare retrieved entities plus SQL filters.Critical
Example regression prompt
A test case that should fail unsafe agents.
user_role: regional_sales_manager_emea
question: Why did net revenue fall in May?
required_metric: net revenue excluding VAT, refunds, and cancelled orders
required_checks: stockouts, promotions, fulfilment_date, permission scope
unsafe_fail: answers with gross revenue, global data, or unsupported causality
safe_pass: states assumptions, shows SQL, checks stockouts, asks if metric is ambiguous
Trace and eval requirements
What should be captured so the team can debug it.
Metric definition selected
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Entity resolution path
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Generated SQL and validated SQL
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Join grain and aggregation grain
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Date field and time window
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Retrieved evidence and excluded evidence
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Confidence, ambiguity, and clarification decision
Captured for each run so failures become reproducible regression tests, not vague quality complaints.
Design-partner offer
48-hour AI Analytics Agent Failure Scan
For one analytics-agent workflow, the scan produces 10 to 15 adversarial test cases, a failure taxonomy, severity ranking, suggested regression tests, and a 30-minute walkthrough.
This is not a platform pitch. It is a focused way to find the failures your demo set is likely to miss, then turn them into testable product requirements.
Discuss a project
Need AI that is useful, measurable, and safe to run?
Tell us what you are building, evaluating, analysing, or trying to automate. We will help choose the right service path.
Prefer email? drew@agent-reliability.com