Skip to content

Create test scenarios

Generate realistic conversations from a short seed and run each one against your agent:

# test_scenarios.py
from monocle_test_tools import MonocleValidator
from travel_agent import build_travel_agent

SEED = {
    "agent": "A travel booking agent that books flights between airports and hotels in a city.",
    "goals": [
        "Book a flight from Bombay to Hyderabad on 22 Oct 2026",
        "Book a hotel in Mumbai for 27 Nov 2026",
    ],
    "out_of_scope": ["refunds", "visa advice"],
    "max_turns": 8,
}

@MonocleValidator().monocle_scenario_testcase(SEED)
async def test_scenarios(scenario):
    await MonocleValidator().test_scenario_async(build_travel_agent(), "langgraph", scenario)

export MONOCLE_TEST_AGENT_MODEL=openai:gpt-4.1-mini   # plays the user (required)
export MONOCLE_TEST_JUDGE_MODEL=openai:gpt-4.1-mini   # judges each reply (defaults to the same)
export MONOCLE_EXPORTER=okahu OKAHU_API_KEY=okh_...
pytest test_scenarios.py

A model plays the user, and a judge decides after every reply whether the scenario's success criteria are met. Each conversation lands in Okahu as a session you can evaluate and label.

Preview — not yet in a published release

Scenario generation (monocle_scenario_testcase) is on Monocle main and is not in a released package yet (the current release is 0.8.15). Install the development version from the Monocle repo main branch to use this page. Everything in the other pages works on the released packages.

Write the seed

Before you write tests, write down who uses the agent, what they want, what they should not do, and the data those situations involve. The seed captures exactly that.

  • Goals — phrase them the way a user would, with real values. The generator splits each goal into its details (source, destination, date) and builds variations around them.
  • out_of_scope — what the agent should decline. Each entry becomes a scope-escape test.
  • max_turns — optional, defaults to 8.

Hand-written scenarios

When you need precise control, write the scenarios yourself and pass the list as the seed:

SEED = [{
    "test_name": "book_flight_impatient",
    "scenario": "Book a flight from source to destination.",
    "persona": "An impatient user who gives one detail at a time.",
    "params": [
        {"name": "source", "value": "San Francisco", "description": "source airport", "is_initial": True},
        {"name": "date", "value": "22 Oct 2026", "description": "travel date"},
    ],
    "success_criteria": "The agent confirms the flight is booked.",
    "max_turns": 6,
    "target_description": "A travel booking agent that books flights and hotels.",
}]
  • Held-back details — a param without is_initial is withheld until the agent asks for it.
  • Optional details — set "required": false for a detail the agent may skip.
  • Test data — vary the values to cover what your users bring: ordinary, edge, and invalid.

What gets generated per goal

About fifteen scenarios per goal, each its own pytest case:

Kind Test id contains What it checks
Personas persona.<name> Five styles: power user, newcomer, developer, manager, terse
Held-back details withhold.<detail> It asks for a missing detail instead of guessing
False premise false_premise It handles a request for a capability it does not have
Red team redteam.<attack> Prompt extraction, unauthorized actions, indirect injection
Out of scope scope.scope_escape.<topic> It declines each out_of_scope topic

Run a subset with -k, e.g. pytest -k redteam or pytest -k withhold.

When a scenario fails

A failed scenario reports the full turn-by-turn transcript with the judge's verdicts. The reasons: criteria violated, maximum turns used up, the agent raised an error, or a required detail was never asked for (the agent made it up).

Notes

  • Fresh agent per scenario — build the agent inside the test so each scenario gets its own memory.
  • Cached set — scenarios are generated on the first run and cached in scenarios.json, reused until the seed changes, so runs stay comparable.
  • Plan for cost — each scenario is a multi-turn conversation with a judge call per turn, so cost grows with goals × scenarios × turns. Start with one goal and -k, then widen.
  • Manual testing counts too — run your app with tracing on and use it as your users would; those traces land in Okahu next to the generated ones and you label them the same way.

With Claude Code

The okahu@skills plugin can plan and write these tests: monocle-test-plan reads your app and writes a scenario/checks table; monocle-test-gen turns a scenario into a pytest test.

Next

Check agent behaviour — the assertions and evals you put inside these tests.