Check agent behaviour

Most tests mix a few kinds of check in one chain:

import pytest

@pytest.mark.asyncio
async def test_books_flight_correctly(monocle_trace_asserter):
    await monocle_trace_asserter.run_agent_async(
        travel_agent, "google_adk", "Book a flight to Mumbai on 22 Nov")

    monocle_trace_asserter.called_tool("book_flight").contains_input("Mumbai")  # calls, inputs
    monocle_trace_asserter.under_token_limit(2000)                              # limits
    monocle_trace_asserter.with_evaluation("okahu") \
        .check_eval("hallucination", expected="no_hallucination")              # eval

Pick the right kind of check

Type	Checks	Methods
Calls	The right tool or sub-agent was used, how often, nothing forbidden	called_tool, called_agent, does_not_call_*, count / min_count / max_count
Inputs and outputs	The arguments passed; what the answer contains or means	contains_input, contains_output, has_output with with_comparer(...)
Limits	Token and time budgets	under_token_limit, under_duration
Span details	Attributes, events, which turn something happened in	where, has_attribute, has_event, has_scope
Code-based checks	Keywords, patterns, valid JSON, exact match, similarity, computed locally	keyword_presence, regex_match, json_validity, exact_match, bert_score
Evals (LLM-as-a-judge)	What needs judgement: correct, grounded, relevant, safe, on-topic	check_eval

The rule: if you can state the question exactly, use one of the first five. They are fast, cheap, and give the same answer every run. If it needs judgement, use an eval.

Calls, inputs, outputs, limits

@pytest.mark.asyncio
async def test_books_the_right_flight(monocle_trace_asserter):
    await monocle_trace_asserter.run_agent_async(travel_agent, "google_adk", "Book a flight to Mumbai")
    monocle_trace_asserter \
        .called_tool("book_flight", agent_name="flight_booking_agent") \
        .contains_input("Mumbai") \
        .contains_output("booked")

@pytest.mark.asyncio
async def test_stays_within_limits(monocle_trace_asserter):
    await monocle_trace_asserter.run_agent_async(travel_agent, "langgraph", "Summarize this document")
    monocle_trace_asserter \
        .called_agent("summarizer_agent") \
        .under_token_limit(500) \
        .under_duration(5.0, units="seconds", span_type="agent_invocation")

What you want to check	Use
The right tool was called (and how often)	called_tool(name, agent_name=…, count= / min_count= / max_count=), called_any_tool(…)
The right sub-agent was used	called_agent(name, count=…), called_any_agent(…)
Something was not done	does_not_call_tool(name), does_not_call_agent(name)
The right arguments were passed	called_tool("book_flight").contains_input("Mumbai")
The answer contains / matches something	contains_output(…); for meaning, not exact text: with_comparer("similarity").has_output(…)
No wasted calls	called_tools(max_count=…), called_tool(name, max_count=…)
Token and time budgets	under_token_limit(n); under_duration(limit, units="ms"/"seconds"/"minutes", span_type=…)
Something happened in a given turn	called_tool("book_flight").has_scope("turn_id", "2")

Selectors narrow what follows. called_agent("x").called_tool("y") checks only tool calls made inside agent x.

Code-based checks

exact_match, regex_match, keyword_presence, json_validity, and bert_score run locally, no model needed. Check that a reply mentions "flight" and never leaks a password or a stack trace:

{"span_type": "agentic.turn",
 "eval": {"eval": "keyword_presence",
          "eval_options": {"required_keywords": ["flight"],
                           "forbidden_keywords": ["password", "traceback"]},
          "args": ["output"],
          "expected_result": {"required_coverage": 1.0, "forbidden_absent": 1.0},
          "comparer": "metric"}}

Evals (LLM-as-a-judge)

An eval is the check for what needs judgement: whether the answer is correct, grounded, relevant, safe, or on-topic. It is an LLM judge with a fixed rubric and a fixed set of labels. Two flavors, used the same way.

Built-in evals

Area	Evals
Task and answer	correctness, answer_relevancy, argument_correctness, mcp_task_completion, conversation_completeness, knowledge_retention, summarization
Grounding (RAG)	hallucination, contextual_precision, contextual_recall, contextual_relevancy
Safety and behaviour	toxicity, bias, pii_leakage, misuse, offtopic, role_adherence
User experience	sentiment, frustration

monocle_trace_asserter.with_evaluation("okahu") \
    .check_eval("hallucination", expected="no_hallucination") \
    .check_eval("bias", expected="unbiased") \
    .check_eval("pii_leakage", not_expected="pii_leakage")

# A session-level eval, accepting either of two labels
monocle_trace_asserter.with_evaluation("okahu") \
    .check_eval(fact_name="agentic_sessions", eval_name="role_adherence",
                expected=["excellent_adherence", "good_adherence"])

If you narrow the trace first (for example called_agent("x")), the eval covers only that part.

Custom evals

A custom eval is your own template: a name, a judging prompt, and a set of labels, such as refund_disclosure → disclosed / not_disclosed / not_applicable. Saved in Okahu, it is versioned and available to every app in your account.

monocle_trace_asserter.with_evaluation("okahu") \
    .check_eval("refund_disclosure", expected="disclosed") \
    .check_eval(template_path="custom_eval_templates/refund_disclosure.json", expected="disclosed")

Reach for a custom eval only when no built-in fits and no exact check can decide the question.

Control what the agent faces

Mock tools

Test the agent's decisions without calling real systems, and simulate failures:

from monocle_test_tools import MockTool, ToolType

monocle_trace_asserter.with_mock_tool(MockTool(
    name="adk_book_flight", type=ToolType.ADK,
    response={"status": "success", "message": "Flight booked from {{from_airport}} to {{to_airport}}."}))
await monocle_trace_asserter.run_agent_async(
    travel_agent, "google_adk", "Book a flight from San Francisco to Mumbai for 26 Nov 2025.")
monocle_trace_asserter.called_tool("adk_book_flight").contains_output("Flight booked")

{{arg}} in a response is filled with the tool's argument value. Set raise_error=True to make a tool fail, then check that the agent retries sensibly or tells the user honestly instead of claiming success.

Multi-turn tests

@MonocleValidator().monocle_multi_turn_testcase(multi_turn_cases)
async def test_multi_turn(multi_turn_case):
    await MonocleValidator().test_multi_turn_agent_async(travel_agent, "google_adk", multi_turn_case)

Each case has a session_id and a list of turns. Use {previous_output} in a turn's input to feed in the last answer. Memory carries across turns for ADK, LangGraph, Strands, LlamaIndex, and Microsoft Agent Framework.

Generate tests with Claude Code

With the okahu@skills plugin, two skills write these tests for you:

• monocle-test-gen writes a specific test you describe: the agent, the input, and what to check (tools, sub-agents, output, limits, or an eval). Point it at your test folder.
• /ok-generate-tests ./my-app finds your agents and tools and writes tests by category: routing, input and output validation, performance, quality (evals), and orchestration. It never overwrites existing files.

Use monocle-test-gen for the behaviours your plan calls out, and /ok-generate-tests for broad coverage.

Next

Label, review & regress — record your own judgement, and turn every failure into a test the next version must pass.