Knowledge check

Test your understanding of Monocle, test tools, and Okahu for CI/CD pipelines with agentic AI workloads.

---

1. What does the Fluent API's does_not_call_tool() assertion validate?

A. That the tool function has been removed from the codebase B. That the tool returned an error when called C. That the agent correctly decided NOT to invoke a specific tool based on its reasoning D. That the tool is not registered in the agent's tool list

Correct! Negative assertions test that the agent's reasoning is correct — it should refuse to act when conditions are ambiguous or invalid, not just succeed when conditions are right.

Incorrect. Negative assertions test that the agent's reasoning is correct — it should refuse to act when conditions are ambiguous or invalid, not just succeed when conditions are right.

2. What is the main advantage of the Okahu Cloud approach over embedding Monocle JSON in GitHub issues?

A. It is faster to set up B. It does not require an Okahu account C. AI agents can dynamically query traces via MCP tools and run quality assessments, with no size limits D. It stores traces in the GitHub repository

Correct! The Cloud approach lets agents use MCP tools (okahu/get_traces, okahu/analyze_error_with_ai, etc.) to dynamically query and analyze traces, with no 50KB issue body limit.

Incorrect. The Cloud approach lets agents use MCP tools (okahu/get_traces, okahu/analyze_error_with_ai, etc.) to dynamically query and analyze traces, with no 50KB issue body limit.

3. What does .with_evaluation("okahu").check_eval("hallucination", "no_hallucination") do in a test?

A. Sends the trace to Okahu's quality assessment API, which verifies the agent's response is grounded in context B. Checks if the agent mentioned the word "hallucination" in its response C. Validates that the LLM model used is not known to hallucinate D. Compares the agent's output against a stored golden response

Correct! The hallucination check sends the complete trace to Okahu's quality assessment API, which verifies the response is grounded in the given context, tools, and user request.

Incorrect. The hallucination check sends the complete trace to Okahu's quality assessment API, which verifies the response is grounded in the given context, tools, and user request.

4. How does Monocle tag traces so they can be looked up by CI/CD run?

A. It adds the run ID to the trace file name B. It sets a scope attribute scope.git.run.id = github_{GITHUB_RUN_ID} on every span via OTel baggage C. It creates a separate Okahu application per run D. It stores the run ID as a plain span attribute that is searchable by default

Correct! Monocle uses set_scopes() to propagate the run ID via OTel baggage. Okahu indexes scope.* attributes as searchable facts, enabling ?fact_ids=github_{RUN_ID} queries.

Incorrect. Monocle uses set_scopes() to propagate the run ID via OTel baggage. Okahu indexes scope.* attributes as searchable facts, enabling ?fact_ids=github_{RUN_ID} queries.

5. What is the difference between called_tool() and called_agent() in the Fluent API?

A. They are interchangeable — tools and agents are the same concept B. called_tool() checks the LLM response text; called_agent() checks span metadata C. called_tool() asserts a specific tool function was invoked; called_agent() asserts a sub-agent was delegated to D. called_agent() only works with supervisor patterns, while called_tool() works with any pattern

Correct! The Fluent API distinguishes between tool invocations (function calls) and agent delegations (sub-agents). Both are visible as spans in the trace.

Incorrect. The Fluent API distinguishes between tool invocations (function calls) and agent delegations (sub-agents). Both are visible as spans in the trace.

6. What quality dimensions does Okahu support beyond hallucination checks?

A. Only hallucination — other metrics require a third-party tool B. Hallucination and latency only C. Hallucination, completeness, relevancy, and other quality dimensions D. It only provides pass/fail without specific metric categories

Correct! Okahu's assessment API supports multiple quality dimensions. You can chain multiple .check_eval() assertions in a single test to validate different aspects of agent behavior.

Incorrect. Okahu's assessment API supports multiple quality dimensions. You can chain multiple .check_eval() assertions in a single test to validate different aspects of agent behavior.

7. What is the four-tier CI/CD progression for agentic AI testing?

A. Unit tests → Integration tests → Load tests → Deployment B. Lint → Build → Test → Deploy C. Basic pytest → Traces embedded in issues → AI agent assignment → Okahu Cloud with MCP tools D. Local testing → Staging → Canary → Production

Correct! Each tier adds capability: Tier 1 runs tests, Tier 2 attaches Monocle traces to issues, Tier 3 assigns AI agents (Claude, Copilot, Kahu) to investigate, and Tier 4 gives agents dynamic trace access via Okahu MCP tools.

Incorrect. Each tier adds capability: Tier 1 runs tests, Tier 2 attaches Monocle traces to issues, Tier 3 assigns AI agents (Claude, Copilot, Kahu) to investigate, and Tier 4 gives agents dynamic trace access via Okahu MCP tools.