Bonus: Bring Your Own Model — Any LLM, Cloud or Local, with ADK 2

📦 Companion Code: View the complete, runnable code on GitHub: part15 in the cartpilot-adk2-series repository.

Every part of this series has run CartPilot on Gemini. That was a deliberate choice for a series about ADK 2 and production Gemini usage — but ADK itself doesn't care. Underneath, ADK's model layer is provider-agnostic: swap Gemini for GPT, Claude, or a model running entirely on your own laptop, and every agent, tool, structured output schema, callback, and multi-agent pattern from Parts 1–14 keeps working unchanged. This bonus part shows exactly how, with one small module you can drop into any ADK project.

How ADK talks to different providers

An agent's model field accepts two shapes:

A native Gemini model string ("gemini-3.5-flash") — ADK talks to the Gemini API directly, no intermediary.
A LiteLlm instance — ADK routes the call through LiteLLM, which normalizes over 100 providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Ollama, vLLM, and more) behind one interface. From ADK's perspective, it's just another model — tool calling, structured output, and streaming all flow through the same code path regardless of what's actually generating the response.

bash

pip install "google-adk>=2.0.0,<3.0.0" python-dotenv "litellm>=1.83.0"

Pin litellm explicitly. A March 2026 supply-chain security advisory affected specific PyPI releases (1.82.7/1.82.8) — check pypi.org/project/litellm/#history for the current safe range before deploying this anywhere real.

One module, every provider

python

# cartpilot_any_llm/model_config.py
import os
from google.adk.models.lite_llm import LiteLlm

_MODEL_IDS = {
    "google": {
        "default": "gemini-3.5-flash",
        "strong": "gemini-2.5-pro",
    },
    "openai": {
        "default": "openai/gpt-4o-mini",
        "strong": "openai/gpt-4o",
    },
    "anthropic": {
        "default": "anthropic/claude-haiku-4-5",
        "strong": "anthropic/claude-sonnet-4-5",
    },
    "ollama": {
        # ollama_chat/ (not ollama/) is the recommended LiteLLM prefix —
        # it maps onto Ollama's native chat API and gives more reliable
        # tool-calling behavior than the older ollama/ prefix.
        "default": f"ollama_chat/{os.getenv('CARTPILOT_OLLAMA_MODEL_DEFAULT', 'llama3.2')}",
        "strong": f"ollama_chat/{os.getenv('CARTPILOT_OLLAMA_MODEL_STRONG', 'qwen2.5:14b')}",
    },
}


def get_model(tier: str = "default"):
    """Returns a model reference for the requested tier, honoring
    CARTPILOT_MODEL_PROVIDER. Native Gemini strings pass through as-is;
    every other provider is wrapped in LiteLlm."""
    provider = os.getenv("CARTPILOT_MODEL_PROVIDER", "google").lower()
    if provider not in _MODEL_IDS:
        raise ValueError(f"Unknown CARTPILOT_MODEL_PROVIDER: {provider!r}")
    if tier not in ("default", "strong"):
        raise ValueError(f"Unknown tier: {tier!r}")

    model_id = _MODEL_IDS[provider][tier]
    if provider == "google":
        return model_id  # native Gemini string — ADK talks to it directly
    return LiteLlm(model=model_id)

Notice "default" and "strong" — this is the same two-tier convention from Part 1 and Part 10, now provider-independent. Whatever CartPilot's cost-tracking data from Part 10 told you about when a specialist needs the stronger tier still applies, no matter which company's model is running underneath.

Using it

Take the Part 2 search agent and change exactly one line:

python

# cartpilot_any_llm/agent.py (excerpt)
from .model_config import get_model

root_agent = Agent(
    model=get_model("default"),  # <-- the only change from Part 2
    name="product_search_agent",
    description="Helps shoppers find products by name, category, or description.",
    instruction="""...""",
    tools=[search_products, get_product_details],
)

Everything else — tools, docstrings, instruction, structured output if you add it — is completely unaware of which provider answers. Control it entirely from .env:

bash

# .env
CARTPILOT_MODEL_PROVIDER=google       # or: openai | anthropic | ollama

# google (native Gemini API)
GOOGLE_API_KEY=your-gemini-api-key-here
GOOGLE_GENAI_USE_VERTEXAI=FALSE

# openai (via LiteLLM)
OPENAI_API_KEY=your-openai-api-key-here

# anthropic (via LiteLLM)
ANTHROPIC_API_KEY=your-anthropic-api-key-here

# ollama (fully local, via LiteLLM — no API key, no cloud calls at all)
CARTPILOT_OLLAMA_MODEL_DEFAULT=llama3.2
CARTPILOT_OLLAMA_MODEL_STRONG=qwen2.5:14b

Going fully local with Ollama

For the ollama provider, nothing leaves your machine:

bash

# Install Ollama: https://ollama.com
ollama pull llama3.2          # a tool-calling-capable model
ollama pull qwen2.5:14b        # a stronger local option, if your hardware handles it

# Ollama serves on http://localhost:11434 by default — LiteLLM's
# ollama_chat/ prefix finds it automatically. Only set OLLAMA_API_BASE
# if you're running Ollama on a different host or port.

Then run CartPilot exactly as before:

bash

CARTPILOT_MODEL_PROVIDER=ollama adk web

Same agent, same tools, zero API keys, zero network calls to any cloud provider — the entire request/response cycle happens on your laptop.

Seeing the swap actually work

A small script drives the identical agent through every configured provider back-to-back, skipping any that aren't set up:

python

# compare_providers.py
import asyncio, os
from dotenv import load_dotenv
load_dotenv()
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.genai import types

QUERY = "Do you have anything for hiking under $30?"
PROVIDERS_TO_TRY = ["google", "openai", "anthropic", "ollama"]
_REQUIRED_ENV = {"google": "GOOGLE_API_KEY", "openai": "OPENAI_API_KEY",
                  "anthropic": "ANTHROPIC_API_KEY", "ollama": None}

async def run_one(provider: str) -> None:
    os.environ["CARTPILOT_MODEL_PROVIDER"] = provider
    required = _REQUIRED_ENV[provider]
    if required and not os.getenv(required):
        print(f"\n--- {provider}: skipped (set {required} in .env to try it) ---")
        return

    import importlib
    import cartpilot_any_llm.agent as agent_module
    importlib.reload(agent_module)  # re-read the provider we just set

    session_service = InMemorySessionService()
    session = await session_service.create_session(app_name="cartpilot_any_llm", user_id="demo-user")
    runner = Runner(agent=agent_module.root_agent, app_name="cartpilot_any_llm", session_service=session_service)

    print(f"\n--- {provider} ---")
    try:
        message = types.Content(role="user", parts=[types.Part(text=QUERY)])
        async for event in runner.run_async(user_id="demo-user", session_id=session.id, new_message=message):
            if event.is_final_response() and event.content and event.content.parts:
                print(event.content.parts[0].text)
    except Exception as exc:
        print(f"(failed: {exc})")

async def main():
    for provider in PROVIDERS_TO_TRY:
        await run_one(provider)

if __name__ == "__main__":
    asyncio.run(main())

bash

python compare_providers.py

You'll see the same question answered by whichever providers you've configured, one after another — the tool call to search_products happens identically regardless of which model requested it.

What actually varies between providers

Swapping the model isn't entirely free — a few things genuinely differ:

Tool-calling reliability varies by model. Gemini and GPT-4-class models call tools reliably with minimal prompting; some smaller local models need more explicit instruction nudging ("always call search_products before answering") to call tools consistently. If you see an agent answering from imagination instead of calling a tool it clearly should, that's model capability, not an ADK bug.
output_schema (Part 4) support depends on the underlying provider's structured-output capabilities. LiteLLM translates ADK's schema request into whatever mechanism the target provider offers — OpenAI's and Gemini's native JSON mode are both solid; some local models handle it less consistently. Test this specifically if you rely on guaranteed structured output on a non-Gemini provider.
Cost and latency characteristics are completely different per provider — Part 10's cost-tracking callback assumed Gemini pricing; adapt PRICE_PER_MILLION_TOKENS per provider (or set it to zero for a fully local Ollama setup, where the real cost is your own hardware, not per-token billing).
Run Part 12's eval suite whenever you change providers, not just when you change prompts. A model swap is exactly the kind of change that eval suite exists to catch — "does this provider still call the right tools in the right order, and produce responses that pass our reference checks?" is an empirical question, not one to assume the answer to.

Where this fits in the bigger picture

This module doesn't replace anything from Parts 1–14 — it sits underneath all of it. Every pattern in this series (structured checkout, human-approved refunds, resilient payments, guardrails, cost tracking, MCP tools) works the same way regardless of which model tier resolves each agent's reasoning. That separation — agent logic in one place, model choice in one small config module — is what makes CartPilot portable across a cloud-only deployment, a fully local one, or anything in between, without touching the parts of the system that actually define what CartPilot does.