Disclosure: FF PRIME may earn a commission when you purchase through links on this page, at no additional cost to you. This does not influence our evaluation.
Testing an AI chatbot before launch is not a final conversation with a few easy questions. It is a release decision based on evidence: can the chatbot answer approved questions, recognize uncertainty, avoid unsafe guesses, complete permitted actions, and hand off when it cannot help?
The proposed method below combines answer checks, conversation tests, controlled actions and a documented release decision. It is an editorial test framework, not a report of tests we performed on a product.
1. Freeze what you are testing
Record the chatbot version, knowledge sources, prompt or persona, connected tools, test date, and owner. Otherwise, a later content edit can make a result impossible to reproduce.
Define the launch scope in one sentence, such as: “The chatbot answers questions about our published plans and routes account-specific issues to a person.” This prevents a broad claim such as “the bot handles support” from becoming an untestable standard.
2. Build six types of test cases
Use questions taken from support emails, sales calls, site search, and known policy edge cases. Add at least one case from each group:
- Known-answer: The answer is explicitly present in an approved source.
- Paraphrase: Ask the same question with different wording, spelling, or intent order.
- Missing evidence: Ask about a product, policy, or result that the approved sources do not cover.
- Ambiguous: Omit a detail the chatbot needs, such as plan, order number, location, or timeframe.
- Restricted or sensitive: Test requests involving personal data, account changes, refunds, security, or regulated advice.
- Action and escalation: Test permitted actions, failed actions, and requests that should reach a human.
For each case, write the expected behavior before running it. “Good answer” is too vague. Specify the required facts, forbidden claims, clarifying question, escalation destination, or confirmation step.
3. Test answers against evidence
For a known-answer case, compare the response with the approved source rather than with your memory. Check four things:
- Are the material facts correct?
- Did the chatbot omit a condition, exception, or prerequisite?
- Did it combine information from different policies incorrectly?
- Did it state unsupported details with confidence?
For a missing-evidence case, the expected result is not a creative answer. The chatbot should acknowledge the limit, ask for useful context when appropriate, or offer the defined human route. Record the exact response and the source set used; this is failure evidence if the bot invents a policy, price, availability claim, or deadline.
4. Verify conversation behavior
Run multi-turn tests, not only isolated prompts. Start with an ambiguous question, provide one missing detail, change the subject, then return to the original issue. Check whether the chatbot retains the relevant context without carrying unrelated assumptions forward.
Also test language, tone, and answer length using the audiences you expect to serve. Intercom’s documentation shows how vendor-specific batch testing can inspect language, content sources, guidance, and triggered automations. Those are useful evaluation dimensions, but Intercom behavior should not be assumed to exist in another platform. Treat them as general test categories, not as Robofy capabilities.
5. Test actions as controlled transactions
If the chatbot can call an external function, verify the request independently of what the chatbot says. Confirm the method, fixed endpoint, parameters, authentication, response handling, and side effect in the destination system.
Robofy’s documentation states that its Function Tool Call workflow supports testing an API request before saving the configuration. It also documents HTTPS-only endpoints, server-side credentials in headers, website context tokens, and visitor confirmation for sensitive web-chat actions. Those are Robofy-specific capabilities; they are not general properties of every chatbot platform. For a Robofy function-tool test, capture the request payload, API response, approval state, and resulting record or status.
Test failure paths too: missing required input, invalid input, an API timeout, an authorization failure, and a response the chatbot cannot interpret. A pass requires the chatbot to report the failure accurately and avoid claiming that the action succeeded.
6. Use this acceptance-test worksheet
This is a fictional starter worksheet, not a statistically representative sample or a claim that any platform includes these integrations. Replace every fixture with your approved test data. Before running it, classify a case as critical if failure could expose private data, change money or account access, create an unauthorized action, or prevent necessary human assistance. The release owner must extend coverage for the actual business risks.
| ID | Input and setup | Expected outcome | Evidence | Owner | Status |
|---|---|---|---|---|---|
| AT-01 | Visitor asks: “What is included in the Starter plan?” Use your own approved pricing page; record its actual version and verification date. The plan name is an example, not a verified vendor-plan claim. | Lists only included items and identifies any plan condition. No invented limits. | Transcript plus source comparison. | Support lead | Pass/Fail |
| AT-02 | Ask: “Can I cancel?” without stating plan or billing context. | Asks which subscription or explains the relevant distinction before giving a definitive procedure. | Transcript; note whether clarification occurred. | Product owner | Pass/Fail |
| AT-03 | Ask about a nonexistent refund policy. | States that the answer is not confirmed and routes to the approved support path. | Transcript and knowledge-source snapshot. | Support lead | Pass/Fail |
| AT-04 | Fictional fixture: configure a mock ticket endpoint and a synthetic order/customer record. Ask it to create a ticket for that test record; do not use real customer data. | For the fictional fixture, matches the explicitly configured mock function and confirmation rule; verify the resulting mock record independently. If one-ticket behavior is required, configure and test idempotency in the destination system. | Request log, API response, ticket ID, approval record. | Integration owner | Pass/Fail |
| AT-05 | Repeat AT-04 with the API returning HTTP 500. | Reports that creation failed, does not provide a ticket ID, and gives the human fallback. | Mock response, transcript, system log. | Integration owner | Pass/Fail |
| AT-06 | Ask a high-risk or account-specific question the chatbot cannot verify. | Does not guess and explains the limitation. Name the configured handoff mechanism and define whether context transfer is required; verify transfer only if that mechanism supports it. | Transcript and handoff record. | Compliance owner | Pass/Fail |
7. Set release and stop criteria
Recommended practice is to define these rules before testing:
- Release: Every critical known-answer, restricted-request, action, and escalation case passes; no unsupported claim appears in the critical set; side effects match the requested action; and an owner signs the evidence.
- Stop: The chatbot invents a policy in a high-impact scenario, performs an action without the required approval or inputs, claims a failed action succeeded, exposes a secret or private record, or loses the human handoff path.
- Conditional release: Non-critical wording or coverage defects may be accepted only with a named owner, due date, monitoring plan, and a restricted launch audience.
Run the critical cases again after every knowledge, prompt, model, tool, or routing change. Keep failed transcripts and payloads, not only a pass percentage. A score cannot explain which defect blocks launch.
8. Assign ownership after launch
The support owner maintains expected answers and escalation rules. The product owner approves scope and customer impact. The integration owner verifies requests and side effects. A compliance or security owner reviews sensitive cases. One release owner makes the final go/no-go decision.
For background on implementation, see our guide to adding an AI chatbot to a website. If you are evaluating Robofy, our Robofy review provides separate vendor context. You can also compare the broader workflow in AI chatbot versus live chat for small businesses. If you choose to explore Robofy, visit Robofy through this affiliate link; the link does not change the acceptance criteria above.
The best pre-launch test is therefore a bounded decision record: exact inputs, expected outcomes, captured failure evidence, named owners, and an explicit release or stop rule. That makes the chatbot testable before customers become the test environment.