Skip to content

Label, review, and turn failures into tests

Every failure you find should become a test the next version must pass. Generate one from a trace:

python -m monocle_test_tools generate_test --trace-id <id> --workflow-name travel-agent \
    --trace-source okahu --eval-source okahu --eval hallucination=no_hallucination \
    > test_regression_refund.py

The generated test asserts the agents and tools called, token and time limits, and the evals you name.

Your labels are the reference

Your own judgement is the reference. A label records it and lets you review the eval's verdicts (and later train an eval that grades the way you would).

POST https://eval.okahu.co/api/v1/eval/label?trace_id=<id>&fact_name=traces&workflow_name=travel-agent
{"result": {"label": "refund_disclosure", "value": "not_disclosed",
            "explanation": "Refund issued in turn 3 before any policy was stated", "category": "manual"}}
  • label is the eval name, value is the verdict.
  • Always include an explanation — training uses it, and it tells reviewers why.
  • Manual labels win. When a manual label and an eval verdict disagree, reports show the manual label as the authoritative result.

Review results

Where What you get
POST https://api.okahu.co/api/v1/workflows/{name}/evals/report Per trace and eval: the authoritative (manual) result, the latest verdicts, and a summary. Body: {"fact_name": "traces", "eval_names": ["refund_disclosure"], "fact_ids": ["<id>"], "category": ["manual", "llm"]}. Up to 300 ids, or a time window.
Quality view (portal) Each eval's verdicts split into LLM, Human, and Needs Human Review, plus traces not yet evaluated

Turn failures into tests

From a whole conversation. A multi-turn conversation (such as one scenario run) arrives as a session; pass its session id so every turn is included:

python -m monocle_test_tools generate_test --session-id <session_id> --workflow-name travel-agent \
    --trace-source okahu --eval-source okahu --eval hallucination=no_hallucination \
    > test_booking_conversation.py

Tighten what was generated. A generated test describes what one run did, not what every run must do. Before you keep it:

  • Remove incidental checks — calls and output text that happened but are not required.
  • Add what must never happen — e.g. does_not_call_tool for forbidden actions.
  • Add the evals that fit, built-in or custom.

From a window of traces in Okahu:

from monocle_test_tools import setup_test_cases

CASES = setup_test_cases(source="okahu", workflow_name="travel-agent",
                         start_time="2026-10-01", end_time="2026-10-21", check_eval="hallucination")

@pytest.mark.parametrize("testcase", CASES)
def test_evals_still_hold(monocle_trace_asserter, testcase):
    monocle_trace_asserter.with_trace_source(testcase=testcase, workflow_name="travel-agent")
    monocle_trace_asserter.with_evaluation("okahu").check_eval(testcase=testcase)
  • Freeze the set — save with json.dump([c.model_dump() for c in CASES], f) and reload with setup_test_cases(source="local", path="cases.json"), so it doesn't change under you.
  • Replay against a new version — run_agent_async(agent, "langgraph", testcase=tc) checks that the new agent still handles the same inputs.

From a CSV of known cases: decorate a test with @monocle_csv_cases("cases.csv").

Next

Monitor in production — keep running these checks over live traffic.