app.acme.com
Hi. What can I help with?
SlackAPIYour real interface
It talks to your agent where your users do, in chat, in Slack, or over the API.
AI agent testing
Test your AI agentthe way your users do.
Smoketest talks to your agent through its real interface, then checks what happened in the system behind it. You get the result, the transcript, and every tool call on the ticket.
Book a callTalk to our founder. Test a real ticket the same day.
Acme › ACM-544In Progress
The assistant cancels a plan when asked
When a customer asks the assistant to cancel, it confirms the date and ends the plan at the close of the billing period.
Activity
SKSara K assigned this to Smoketest
Smoketest Read the ticket, 3 comments, and PR #1358. The change touches the cancel tool.
Ticket3 commentsPR #1358Playbook, 2 rules
Test plan2m 26s
1Asks before it cancelsReplyPassed
2Says when the plan endsReplyPassed
3Calls cancel_plan onceTool callPassed
4Billing shows the plan endingBillingFailed
Failed · 3 of 4 checks passed · 2m 26s
The assistant said the plan was cancelled, but cancel_plan returned an error it did not read. Billing still renews on December 1. Turn 4 in the transcript.
Not tested: refund requests, because the ticket only covers cancelling.
RecordingStepsReasoning
Smoketest moved this to In Progress and assigned Kevin
Billing says the plan still renews
You get the result and the transcript on the ticket.
It checks what your agent did, as well as what it said.
A good reply can hide a bad outcome, so it looks behind every answer.
SlackAPI- 1find_accountemail: "dana@acme.com"
- 2get_planid: "acct_204"
- 3cancel_planat_period_end: true
Tool calls
It checks which tools your agent called, in what order, and with what.
Subscription acct_204
BeforeProRenews Dec 1
AfterProEnds Dec 1
What changed
It looks in your system afterwards to see whether the work really happened.
What does the Pro plan cost?Pro is $399 a month.Actually, can I pay yearly?Yes. I can switch you now.Wait, not yet. Which plan am I on?You are on Pro, billed monthly.
Long conversations
It follows up, changes its mind, and goes off topic, the way people do.
Show me Dana's invoices.I can't share another customer's account.Refused
What it should refuse
It asks for things your agent must not do, and checks that it says no.
7 of 8ways of asking got it right
More than one run
It asks the same thing several ways, because one good answer proves little.
You move the ticket. Smoketest does the rest.
Your columns, assignees, and workflow stay as they are. People only step in where a person is needed.
1 Hand off
2 Read
3 Plan
4 Test
5 Report
Developer
Moves the ticket to Ready for QA
Fixes what failed, with the transcript in hand
Smoketest
Reads the ticket and the pull request
Writes the conversation and what must be true
Talks to your agent and looks behind each reply
Posts the result and moves the ticket
QA lead
Sets the rules it plans from
Answers when a ticket needs a look
- 1 Hand offDeveloperMoves the ticket to Ready for QA
- 2 ReadSmoketestReads the ticket and the pull request
- 3 PlanSmoketestWrites the conversation and what must be trueQA leadSets the rules it plans from
- 4 TestSmoketestTalks to your agent and looks behind each reply
- 5 ReportDeveloperFixes what failed, with the transcript in handSmoketestPosts the result and moves the ticketQA leadAnswers when a ticket needs a look
Every ticket comes back Passed, Failed, or No Result.
If a person moves a ticket while it is testing, it stops and never moves it back.
Acme › ACM-544In Progress
The assistant cancels a plan when asked
The assistant confirms the date and ends the plan at the close of the billing period.
Smoketest3 min ago
Failed 3 of 4 checks passed on PR #1358, 2m 26s
- Asked before it cancelled
- Said the plan ends on December 1
- Called cancel_plan once
- Billing shows the plan endingExpected an end date, Billing says it renews on December 1. Turn 4 in the transcript.
- Not tested: refund requestsThe ticket only covers cancelling.
RecordingStepsReasoning
Smoketest moved this issue to In Progress and assigned Kevin
Keep a passed ticket as a regression test.
Your coverage grows with every sprint, and a new model or prompt runs against everything you saved.
Ticket test
Written from the ticket and its acceptance criteria. Lives for the sprint.
When a ticket reaches QARegression
Save it to your library and run it on deploys, pull requests, and pushes.
On every deployProduction monitor
Point it at production and get an alert when a flow breaks or recovers.
Hourly, daily, or weekly
A QA teammate with no queue.
| QA team member | Smoketest | |
|---|---|---|
| Picks up a ticket | When it reaches the top of their queue | As soon as it reaches QA |
| Talks to the agent | By hand, a few messages at a time | Through its real interface |
| Looks behind the reply | Opens the logs if something seems off | Every time, in the same test |
| Repeats a scenario | Once or twice, if there is time | As many times as you set |
| Proof on the ticket | A pasted chat and a comment | Result, transcript, and tool calls |
| Judgment calls | Makes them | Leaves a note and waits for a person |
Your QA team stays in charge. Smoketest takes the first pass, and they keep the calls that need a person.
See what the first pass costs you each year.
Someone checks every ticket today. Move the sliders to see what those hours add up to, and what Smoketest gives back.
Back in your budget each year$52,0001,040 hours a year at $50 an hour
Book a callValued at $50 per hour, the U.S. median wage for QA analysts and testers (BLS, May 2025), over 26 two-week sprints. First pass only, before your Smoketest plan.
One team member for everything you ship.
When a ticket touches more than your agent, the same ticket runs on the other surfaces too.
| Surface | Sandbox | What it checks | Proof you get | |
|---|---|---|---|---|
| Web apps | SandboxCloud browser at any width | What it checksFlows, forms, logins, downloads | Proof you getVideo, steps, transcript | → |
| Mobile apps | SandboxiOS simulator, Android emulator | What it checksScreens, gestures, deep links | Proof you getVideo, steps, device log | → |
| APIs | SandboxAPI runner against staging | What it checksStatus, contracts, auth, side effects | Proof you getRequest and response log | → |
| MCP servers | SandboxMCP client with a real model | What it checksSchemas, auth, real tool calls | Proof you getTool-call transcript | → |
| AI agents | SandboxYour agent's real interface | What it checksReplies, tool calls, side effects | Proof you getConversation transcript | → |
Building something else? One team asked us to test their voice agent, so we are adding it. Propose your surface
Testing an AI agent comes down to five checks.
Whether a person or an agent does it, a good pass covers the same ground. Smoketest works out which of these a ticket needs, then runs them.
- 1
The job itself
Give it the task your users give it, and see whether the task gets done.
- 2
The tools
Check it called the right tools, with the right arguments, in the right order.
- 3
The system behind it
Look at what changed, and compare that with what the agent said.
- 4
The limits
Ask for what it must refuse, and check when it hands over to a person.
- 5
The long way round
Follow up, change your mind, and ask the same thing another way.
Questions about testing AI agents
It runs scenarios through your agent's real interface, then checks its replies, its tool calls, and what changed in your system.
Keep them. Evals score your model on a set of examples. Smoketest takes each ticket and runs it through the product, so it also finds what breaks around the model, like a tool call that failed.
It checks what has to be true, such as what the agent did and what changed in your system. The exact words can vary, and you can run a scenario as many times as you need.
Yes. It talks to your agent from the outside, the way a user does, so it does not matter what your agent is built on.
It checks through the screens and APIs your team already uses, with test accounts you set up. We go through this with you on the call.
Yes. Point a saved test at production on a schedule, and you get an alert when it breaks or recovers.
It starts at $399 a month for 1,000 runs. Each run is up to 10 minutes, and a longer test counts as two runs or more, one for every 10 minutes. If your team needs more, we design a Scale plan with you, so you stop buying extra runs.
Every result links to the recording, the steps, and the reasoning, and says what it did not test, so you can check its work the way you would check a teammate's.
Your next agent ticket comes back tested.
Book a call and we will run one ticket through your agent with you. You get the result and the transcript the same day.
Ready for QA
Done
ACM-544
The assistant cancels a plan when asked
Passed, recording