Evaluating CartPilot: ADK's Eval Framework

📦 Companion Code: View the complete, runnable code on GitHub: part12 in the cartpilot-adk2-series repository.

Every part of this series has made claims on faith: "Flash is enough for search," "the checkout pipeline validates correctly," "the refund confirmation flow works." Faith doesn't survive a prompt tweak, a model swap, or a tool refactor six months from now. ADK's built-in evaluation framework turns those claims into automated, repeatable tests — the agent equivalent of a regression suite.

The two things an eval checks

ADK's AgentEvaluator scores an agent run against a reference case on two axes:

• tool_trajectory_avg_score — did the agent call the right tools, with sensible arguments, in the right order? Scored by comparing actual vs. expected tool calls, with EXACT, IN_ORDER, or ANY_ORDER matching depending on how strict you need to be.
• response_match_score — how closely does the agent's final text match a reference response? Scored with ROUGE (word-overlap similarity) by default — good for catching gross regressions, not a substitute for human review of tone and nuance.

Building an eval set

The fastest, most reliable way to create real eval cases: use adk web's Eval tab. Have a real conversation with CartPilot, then "Add current session to Eval Set" — this captures the actual user input, actual tool calls, and actual final response as a starting point, which you then review and adjust into your reference expectations. That beats hand-authoring the JSON from scratch, and it means your eval cases reflect real usage patterns rather than what you imagine usage looks like.

For illustration, here's the shape directly:

// tests/search.test.json
{
  "eval_set_id": "cartpilot_search_basic",
  "name": "CartPilot search agent — basic product discovery",
  "eval_cases": [
    {
      "eval_id": "find_water_bottle",
      "conversation": [
        {
          "invocation_id": "inv-1",
          "user_content": {"parts": [{"text": "Do you have any water bottles?"}], "role": "user"},
          "final_response": {
            "parts": [{"text": "Yes — we have an Insulated Water Bottle 1L for $24.50, currently in stock."}],
            "role": "model"
          },
          "intermediate_data": {
            "tool_uses": [{"name": "search_products", "args": {"query": "water bottle", "category": ""}}]
          }
        }
      ],
      "session_input": {"app_name": "cartpilot", "user_id": "eval-user", "state": {}}
    },
    {
      "eval_id": "no_results_handled_gracefully",
      "conversation": [
        {
          "invocation_id": "inv-1",
          "user_content": {"parts": [{"text": "Do you sell snowboards?"}], "role": "user"},
          "final_response": {
            "parts": [{"text": "We don't have any snowboards right now — would you like to see our other outdoor gear?"}],
            "role": "model"
          },
          "intermediate_data": {
            "tool_uses": [{"name": "search_products", "args": {"query": "snowboard", "category": ""}}]
          }
        }
      ],
      "session_input": {"app_name": "cartpilot", "user_id": "eval-user", "state": {}}
    }
  ]
}

The second case matters as much as the first — testing the graceful-failure path (no results found) catches a real class of regression: a prompt change that makes the agent start hallucinating products that don't exist instead of saying so honestly.

Set thresholds separately from the cases themselves:

// tests/test_config.json
{
  "criteria": {
    "tool_trajectory_avg_score": 0.8,
    "response_match_score": 0.5
  }
}

response_match_score at 0.5 (not 1.0) is deliberate — ROUGE penalizes any rewording, and CartPilot's agent shouldn't be locked into one exact phrasing. tool_trajectory_avg_score at 0.8 is stricter, because getting the tool call right matters more than getting the wording right for an agent whose job is fundamentally about taking correct actions.

Wiring it into pytest

# tests/test_cartpilot_eval.py
import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator


@pytest.mark.asyncio
async def test_search_agent_basic_trajectory():
    """Regression gate: search_agent must call search_products with
    sensible args and produce a response matching the reference closely
    enough, for both a successful search and a graceful no-results case."""
    await AgentEvaluator.evaluate(
        agent_module="cartpilot",
        eval_dataset_file_path_or_dir="tests/search.test.json",
    )

pip install pytest pytest-asyncio
pytest tests/ -v

This calls the real Gemini API for every case — it costs real tokens and takes real time, which is expected for an eval suite. If you're running it on every commit, that cost is itself worth tracking with Part 10's cost callback; a bloated eval suite run on every PR can quietly become one of your biggest recurring token expenses.

Where this earns its keep

The value isn't the first green checkmark — it's every regression it catches afterward. Concretely, for CartPilot:

• Before changing a prompt, run the suite, change the prompt, run it again. A drop in tool_trajectory_avg_score for checkout_pipeline's validation step tells you immediately if your wording change accidentally made the model skip the stock check.
• Before swapping a model tier (the decision Part 10 set up), the eval suite is what actually answers "does Flash perform acceptably here?" instead of a gut feeling. Run the same eval set against both tiers and compare scores directly.
• In CI, a failing eval blocks a merge the same way a failing unit test would — pytest's exit code integrates with any CI pipeline unchanged.

What's next

CartPilot's tools have all been code we wrote. Part 13 extends it with a real third-party MCP tool — an external server CartPilot can call for shipping rates — and looks at what governance that connection needs (auth, rate limits, cost attribution) once you're depending on someone else's service.