Skip to Content
DocsTesting & QATest Cases

Test Cases

A test case defines a single scenario to test against your AI voice agent. Each case simulates a caller with a specific persona who says specific things, and success is measured against criteria you define.

Creating Test Cases

There are two ways to create test cases: manually or with AI generation.

Manual Creation

  1. Open a test suite
  2. Click Add Test Case
  3. Fill in the fields:
FieldDescription
NameA short, descriptive name (e.g., “Interested buyer asks about pricing”)
ScenarioWhat the test caller should say and how the conversation should unfold. Write this as a narrative description of the simulated call.
PersonaThe personality and tone of the simulated caller. This affects how the AI caller responds to your agent.
Success CriteriaWhat constitutes a pass for this test case. See Success Criteria Types below.
Max TurnsMaximum number of conversation exchanges before the test is stopped. Prevents infinite loops.
  1. Click Save

Example Test Case

  • Name: Interested lead asks for pricing
  • Scenario: The caller is interested in the product and asks about pricing options. They want to know about monthly vs annual plans and whether there is a free trial.
  • Persona: Friendly and direct. Business owner in their 40s. Asks clear questions and expects specific answers.
  • Success Criteria: keyword_match — transcript must contain “pricing”, “plan”, and either “trial” or “free”
  • Max Turns: 10

AI Generation

Instead of writing every test case by hand, you can describe your scenarios in plain English and let AI generate the test cases.

  1. Open a test suite
  2. Click Generate Test Cases
  3. In the text area, describe the scenarios you want to cover. For example:
Generate test cases for a dental office receptionist agent:
- Patient calling to schedule a cleaning
- Patient calling to cancel an appointment
- Patient asking about insurance coverage
- Angry patient complaining about a billing error
- Caller asking for the office address and hours
  1. Click Generate
  2. Review the generated test cases. Each one will have a pre-filled name, scenario, persona, success criteria, and max turns.
  3. Edit any cases as needed, then click Save All
💡

AI generation works best when you describe the caller’s intent and emotional state. Instead of “test appointment booking,” write “patient who is nervous about dental work calls to schedule their first cleaning in 3 years.”

Success Criteria Types

Each test case uses one of three success criteria types to determine if the test passes or fails.

keyword_match

The simplest criteria type. The test passes if the call transcript contains specific words or phrases.

Configuration:

  • Keywords — A list of words or phrases that must appear in the transcript
  • Match mode — all (every keyword must appear) or any (at least one keyword must appear)

Example: Keywords = ["appointment", "confirmed"], Match mode = all The test passes only if both “appointment” and “confirmed” appear somewhere in the transcript.

sentiment_check

The test passes if the call ends with a specific sentiment.

Configuration:

  • Target sentiment — positive, neutral, or negative

Example: Target sentiment = positive The test passes if the overall call sentiment is rated as positive by the AI analysis.

custom

Free-form criteria evaluated by AI. Use this for nuanced requirements that cannot be expressed as simple keyword matches.

Configuration:

  • Criteria description — A natural language description of what constitutes a pass

Example: Criteria = “The agent must correctly explain the return policy, including the 30-day window and the requirement for original packaging, without providing incorrect information.”

The AI evaluator reads the transcript and determines whether the criteria were met, providing a detailed explanation of its reasoning.

Managing Test Cases

Enable/Disable Individual Cases

You can disable a test case without deleting it. Disabled cases are skipped during test runs.

  1. Open the test suite
  2. Find the test case
  3. Toggle the Enabled switch

This is useful when a test case is temporarily irrelevant (e.g., testing a seasonal promotion that has ended) but you want to keep it for future use.

Editing a Test Case

  1. Open the test suite
  2. Click the test case name
  3. Modify any fields
  4. Click Save

Deleting a Test Case

  1. Open the test suite
  2. Click the test case name
  3. Click Delete
  4. Confirm the deletion

Best Practices

Write Specific Scenarios

Vague scenarios produce inconsistent results. Instead of “caller asks a question,” write “caller asks whether the product supports integration with Salesforce and wants to know the setup time.”

Use Realistic Personas

The persona affects how the simulated caller behaves. A “rushed executive” will respond differently than a “curious first-time buyer.” Match your personas to the real callers your agent will encounter.

Set Appropriate Max Turns

  • Simple scenarios (address lookup, hours inquiry): 4-6 turns
  • Medium scenarios (appointment booking, product inquiry): 8-12 turns
  • Complex scenarios (complaint handling, multi-step processes): 12-20 turns

Setting max turns too low may cause the test to stop before the scenario completes. Setting it too high wastes time on conversations that have clearly gone off track.

Combine Criteria Types

For important test cases, consider using custom criteria that encompass both content and tone requirements. For example: “The agent must book the appointment AND maintain a professional, empathetic tone throughout the conversation.”

Last updated on
Test SuitesTest Runs