Bonus: Comparing Total Cost of Ownership Across LLM Providers

📦 Companion Code: View the complete, runnable code on GitHub: part17 in the cartpilot-adk2-series repository.

Part 15 made CartPilot's model provider swappable with one .env variable. Part 16 built a TCO calculator — for Gemini only. This part puts them together: the same TCO model, computed side-by-side across every provider Part 15 wired up, so "should we run this on Gemini, GPT, Claude, or a local Ollama model" becomes a number you compute instead of a guess based on one provider's list price.

What actually changes between providers, cost-wise

Only one thing: the per-token price. Everything else in Part 16's TCO model — hosting, third-party MCP tool costs, eval/CI runs, human-in-the-loop review time, observability storage — is a function of CartPilot's own architecture and traffic, not of which company's model answers a given call. That's the key simplification this script relies on: swap the pricing table, hold everything else constant, compare totals.

Extending the calculator

python

# provider_tco_compare.py
from dataclasses import dataclass

# Illustrative per-million-token pricing — verify current rates before
# using this for real budgeting.
PROVIDER_PRICING = {
    "google": {
        "default": {"input": 0.30, "output": 2.50},   # gemini-3.5-flash
        "strong":  {"input": 1.25, "output": 10.00},  # gemini-2.5-pro
    },
    "openai": {
        "default": {"input": 0.15, "output": 0.60},   # gpt-4o-mini
        "strong":  {"input": 2.50, "output": 10.00},  # gpt-4o
    },
    "anthropic": {
        "default": {"input": 0.80, "output": 4.00},   # claude-haiku-4-5
        "strong":  {"input": 3.00, "output": 15.00},  # claude-sonnet-4-5
    },
    "ollama": {
        "default": {"input": 0.0, "output": 0.0},     # local — no per-token cost
        "strong":  {"input": 0.0, "output": 0.0},
    },
}

# A local Ollama setup isn't free — it's a different cost SHAPE: fixed
# monthly hardware/power instead of per-token. Set to $0 if you're on
# hardware you'd own regardless; set it to a real number if this
# deployment is the reason you bought or rented that hardware.
LOCAL_HARDWARE_MONTHLY_USD = 400.0


@dataclass
class AgentProfile:
    """Provider-independent usage shape — how often and how much an
    agent talks has nothing to do with which provider answers."""
    name: str
    tier: str  # "default" or "strong"
    calls_per_conversation: float
    avg_input_tokens_per_call: int
    avg_output_tokens_per_call: int

    def cost_per_conversation(self, provider: str) -> float:
        prices = PROVIDER_PRICING[provider][self.tier]
        input_cost = (self.avg_input_tokens_per_call * self.calls_per_conversation
                       / 1_000_000 * prices["input"])
        output_cost = (self.avg_output_tokens_per_call * self.calls_per_conversation
                        / 1_000_000 * prices["output"])
        return input_cost + output_cost

The rest of the fixed-cost machinery (FixedCosts, compute_tco_for_provider) is Part 16's TCOAssumptions/compute_tco renamed and parameterized by provider — one meaningful addition:

python

def compute_tco_for_provider(provider, conversations_per_month, agents, fixed):
    llm_cost_monthly = sum(a.cost_per_conversation(provider) for a in agents) * conversations_per_month
    # ... mcp_cost_monthly, eval_cost_monthly, hitl_cost_monthly, log_cost_monthly
    # unchanged from Part 16 ...

    hosting_cost = fixed.hosting_monthly_usd
    if provider == "ollama":
        # Swap the cloud LLM API line for a local hardware line.
        hosting_cost += LOCAL_HARDWARE_MONTHLY_USD

    total = llm_cost_monthly + hosting_cost + mcp_cost_monthly + eval_cost_monthly + hitl_cost_monthly + log_cost_monthly
    return {"provider": provider, "llm_api_cost": round(llm_cost_monthly, 2),
            "hosting_cost": round(hosting_cost, 2), "total_monthly_usd": round(total, 2),
            "cost_per_conversation_usd": round(total / conversations_per_month, 4)}

That if provider == "ollama" branch is the whole point of modeling this honestly: a local model's token cost is genuinely $0, but treating that as "Ollama is free" is a mistake — the compute has to run somewhere, and that somewhere costs money (hardware, power, or a rented GPU instance) whether or not you're billed per token for it.

Running it against CartPilot's real shape

bash

python provider_tco_compare.py

text

=== CartPilot Monthly TCO by Provider (50,000 conversations/mo) ===

Provider          LLM API      Hosting     Total/mo    Per convo
--------------------------------------------------------------
google       $    280.95 $    150.00 $    654.95 $    0.0131
openai       $    245.73 $    150.00 $    619.73 $    0.0124
anthropic    $    583.40 $    150.00 $    957.40 $    0.0191
ollama       $      0.00 $    550.00 $    774.00 $    0.0155

At this illustrative volume and with a $400/month hardware amortization assumption, Ollama is not the cheapest option — OpenAI and Gemini both undercut it, because 50,000 conversations/month isn't enough LLM-token spend to outweigh a fixed monthly hardware cost. That crossover point is exactly what this script is for: run it again at 500,000 conversations/month (just change conversations_per_month) and the ranking changes, because the fixed hardware cost gets amortized over far more usage while the cloud providers' bills scale linearly. There's a real break-even volume where local hosting starts winning — this script is how you find it for your own numbers instead of assuming "local is always cheaper" or "local is never worth it."

The caveat that matters most: cost isn't the only axis

The script prints this deliberately, and it's worth taking seriously rather than skimming past:

Only compare providers that clear the same bar on Part 12's eval suite. A cheaper provider that produces worse tool-call accuracy or worse structured-output reliability isn't actually cheaper once you account for the downstream cost — more refunds needing human review (Part 8), more support escalations, more shopper frustration. This script prices four cost centers explicitly; it does NOT automatically raise refunds_needing_approval_per_month or avg_approval_review_minutes when you swap in a weaker model, because it can't know that without you measuring it.

The correct workflow, combining everything this series has built:

Run Part 12's eval suite against each candidate provider (swap CARTPILOT_MODEL_PROVIDER from Part 15 between runs).
Only keep providers that pass your quality bar in the comparison at all — a provider that fails evals isn't a legitimate cheaper option, it's a different, worse product.
For providers that do pass, feed this script your real usage numbers — token counts from Part 10's usage_log running against each provider (token counts differ slightly by provider even for equivalent prompts, since tokenizers differ), not the illustrative defaults.
Also weigh latency and rate limits, which this script deliberately doesn't price — a provider that's technically cheaper per token but rate-limits you into dropped requests during a sale traffic spike has a cost this model can't capture in dollars.

Where this leaves the series

Part 15 gave CartPilot the ability to run on any model. Part 16 gave it a way to price itself as a whole system. This part closes the loop: the same portability that lets you swap providers for technical reasons (data residency, latency, a provider outage) also lets you swap them for economic reasons — and now you have a way to make that comparison honestly, hardware costs and quality bars included, instead of picking a provider once and never revisiting the decision as your usage volume changes.