Skip to content

Evaluate AI agents

A test that runs your agent, checks what it did, and judges its answer:

import pytest

@pytest.mark.asyncio
async def test_books_the_right_flight(monocle_trace_asserter):
    await monocle_trace_asserter.run_agent_async(
        travel_agent, "langgraph", "Book a flight to Mumbai on 22 Nov")

    # Exact checks: fast, and the same answer every run
    monocle_trace_asserter.called_tool("book_flight").contains_input("Mumbai")
    monocle_trace_asserter.under_token_limit(2000)

    # A judgement call: let an eval decide
    monocle_trace_asserter.with_evaluation("okahu") \
        .check_eval("hallucination", expected="no_hallucination")

Two kinds of check in one test: exact assertions in code for what you can state precisely, and a judge for what needs judgement. Run it on every version, then over live traffic.

Why evaluate at all

A demo that works once is not proof. The same prompt can make an agent pick a different tool, take a different path, or return an answer that reads fine and is wrong. Repeatable checks are the only way to know a new version is better, not worse.

The four pieces

Piece What it is You use it to
Monocle (monocle_apptrace) Open-source tracing for AI apps Record every agent, tool, model, and retrieval step as a trace
Monocle test tools (monocle_test_tools) A pytest plugin Run your agent, assert on its trace, and call evals from tests
Okahu The platform behind it Store traces, run the evals, keep labels, watch tokens, latency, and errors
okahu@skills A Claude Code plugin Guided workflows: instrument an app, write tests, author evals

Monocle is open source and runs in your code. Okahu is the hosted service the traces and evals go to. You can read traces and run evals from the portal, a VS Code extension, or an API.

An eval is one LLM-as-a-judge check: a judging prompt with a fixed set of labels, like hallucination → hallucination / no_hallucination. Okahu ships a set of built-in evals, and you can write your own.

The scorecard you are building

You grade the agent on six things. Each grade combines more than one signal: exact checks in code, an eval where the answer needs judgement, and your own labels to settle disagreements.

Dimension Question
Accuracy Did it complete the task?
Reliability Does it take the right steps, every time?
Quality Does it behave well?
Security Can it be misused?
Cost Does it use too many tokens?
Performance Is it fast enough?

Illustration only

The dashboard below is a mock-up drawn for this guide. It is not an Okahu product screen.

An example agent scorecard: six dimensions, each pass, warn, or fail, with trend lines
An example scorecard for a travel agent. Illustration for this guide, not an Okahu product.

Where to go next

The pages follow the workflow end to end:

  1. Quickstart — run your first eval on a traced agent.
  2. Create test scenarios — generate realistic conversations to test against.
  3. Check agent behaviour — every assertion type, with code.
  4. Label, review & regress — record your judgement and turn failures into tests.
  5. Monitor in production — keep the checks running over live traffic.
  6. The agent scorecard — roll it all into one view of agent health.
  7. Reference — environment variables and the API.