Resilient Tools: ADK's Error Conventions & Self-Healing Agents

📦 Companion Code: View the complete, runnable code on GitHub: part09 in the cartpilot-adk2-series repository.

CartPilot's tools have all been happy-path so far. In production, get_order_status will occasionally hit a down service, charge_payment will occasionally hit a flaky gateway. What happens next shouldn't be "the conversation breaks" — it should be something the agent can reason about and communicate, the same way a good support rep handles a system outage without panicking.

Convention 1: tools never raise raw exceptions to the model

A model reading a Python traceback doesn't have anything useful to do with it. A model reading {"status": "error", "error_type": "payment_failed", "retryable": false, "message": "..."} can decide what to say and do next. Every tool boundary should convert exceptions into structured data before the model ever sees them.

Convention 2: retries are bounded and live in the tool, not the conversation

Left unconstrained, a model that sees a failure can decide to "just try again" — and again, and again, each attempt a full round-trip. That's not resilience, it's an unbounded cost multiplier with a thin veneer of retry logic. The fix: give the tool itself a fixed, small retry budget, and once it's spent, report failure structurally and definitively — with retryable: false so the model knows not to attempt it again in this turn.

# cartpilot_checkout/tools.py
import random
import time
from functools import wraps


class RetryBudgetExceeded(Exception):
    pass


def with_retries(max_attempts: int = 3, base_delay: float = 0.2):
    """Retries a flaky call with exponential backoff, up to a hard cap."""
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            last_error = None
            for attempt in range(1, max_attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except Exception as exc:
                    last_error = exc
                    if attempt < max_attempts:
                        time.sleep(base_delay * (2 ** (attempt - 1)))
            raise RetryBudgetExceeded(str(last_error))
        return wrapper
    return decorator


def _flaky_payment_gateway_call(order_total: float) -> dict:
    """Simulates an unreliable external payment gateway."""
    if random.random() < 0.3:
        raise ConnectionError("payment gateway timeout")
    return {"transaction_id": f"txn_{random.randint(100000, 999999)}", "amount_charged": order_total}


@with_retries(max_attempts=3)
def _charge_with_retries(order_total: float) -> dict:
    return _flaky_payment_gateway_call(order_total)


def charge_payment(order_total: float) -> dict:
    """Charges the shopper's payment method for the order total.

    Args:
        order_total: The total amount to charge, in dollars.

    Returns:
        On success: {"status": "success", "transaction_id": ..., "amount_charged": ...}
        On failure: {"status": "error", "error_type": "payment_failed",
                     "retryable": bool, "message": str}
    """
    try:
        result = _charge_with_retries(order_total)
        return {"status": "success", **result}
    except RetryBudgetExceeded as exc:
        return {
            "status": "error",
            "error_type": "payment_failed",
            "retryable": False,
            "message": f"Payment could not be processed after multiple attempts: {exc}",
        }

# cartpilot_checkout/agent.py
from google.adk.agents import Agent
from .tools import charge_payment

root_agent = Agent(
    model="gemini-3.5-flash",
    name="checkout_payment_agent",
    description="Handles payment charging during checkout, with resilient error handling.",
    instruction="""
You process payment for CartPilot orders.
- Call `charge_payment` with the order total.
- If status is "success", confirm the charge with the transaction ID.
- If status is "error": apologize briefly, do NOT call `charge_payment`
  again yourself (it already exhausted its own retry budget), and tell
  the shopper their payment method wasn't charged and they can try
  again in a few minutes or use a different payment method.
""",
    tools=[charge_payment],
)

The instruction's explicit "do NOT call charge_payment again yourself" line matters as much as the code — a model that sees retryable: false but isn't told what that means might still try again anyway. State the convention in the instruction, not just in the data shape.

Why retryable earns its place in the schema

Not all failures are equal. A payment gateway timeout might genuinely be worth one retry from the model's perspective if the tool's own retry budget wasn't the issue — but a validation failure ("invalid card number") will fail identically on every attempt. Distinguishing retryable: true from retryable: false in the error shape lets the model make a sensible choice instead of guessing: retry once more itself for a transient-looking failure, or stop and hand off to a human for a structural one.

Where this connects to cost

Every retry is a full tool round-trip, and if the model is the one deciding to retry (rather than the tool's own bounded loop), each attempt often triggers another model generation too, to decide how to retry. A tool-level retry budget converts an open-ended cost risk — "how many times might the model try this before giving up?" — into a fixed, known cost: at most max_attempts calls to the flaky dependency, period, every time. That predictability matters as much as the resilience itself once you're accountable for a cost budget in production, which is exactly where we're headed next.

What's next

We've mentioned cost and token discipline in nearly every part so far as a side note. Part 10 makes it the main event: measuring what CartPilot's conversations actually cost, tracking usage per node, and using that data — not intuition — to decide where Flash is enough and where Pro earns its keep.