Monitor in production

Once the agent is live, keep running the same checks over real traffic. Schedule an eval to run on a cron, in plain language:

POST https://api.okahu.co/api/v1/schedules
{"prompt": "Run hallucination on travel-agent traces from the last 24h and summarize",
 "cron_expr": "0 9 * * 1-5", "timezone": "America/Los_Angeles"}

The question shifts from "is this version ready?" to "is it still behaving as it did at release?"

What to watch

Need	Use
Latency & token budgets	Set SLAs per app in the portal, or PUT https://api.okahu.co/api/v1/apps/{app} with "sla": {"performance": {"default": {"value": 5, "unit": "sec"}}, "budget": {"traces": {"value": 1000, "unit": "tokens"}}}. Each trace is marked meeting or missing its SLA.
Trends	Per-app measures: latency and p95, tokens and p95, tool/agent invocations, turns per session, session duration. GET /v1/insights/apps/{app}/facts/{fact}; histograms at GET /v1/apps/{app}/histogram.
Errors	Error groups cluster similar runtime errors and summarize each: GET https://ml-analysis.okahu.co/api/v1/insights/apps/{app}/error_groups?start_time&end_time.
Quality on live traffic	Run evals over a recent window on a schedule. Review the Quality view, and label a sample yourself to confirm the eval still agrees with you.
Eval spend	POST https://api.okahu.co/api/v1/usage/query with {"metrics": ["eval_tokens"]}.
Ask questions	Kahu in VS Code ("why did errors spike yesterday?"), or /kahu <question> in Claude Code.

Schedules run on a cron (up to 100 active). Keep a weekly schedule that re-runs your key evals and posts a summary.

Close the loop

Feed what you find back into testing. A production failure becomes a regression test; new user behaviour becomes a new scenario. That is what keeps the pre-launch and production scorecards connected.

Next

The agent scorecard — roll all of this into one view of the agent's health.