How to Reduce AI API Costs: 7 Levers That Actually Move the Bill

Last updated
The short version

Most AI API bills are 2–10× bigger than they need to be. The biggest levers, in order:

  • Right-size the model — route easy work to a cheaper/mini model.
  • Turn on prompt caching for repeated prefixes (system prompts, RAG context).
  • Move bulk work to the Batch API for a ~50% discount.
  • Send fewer tokens — trim context, shorten prompts, use a compact output format.

The per-token sticker price is the least interesting number in your AI bill. What actually decides the total is how many tokens you send, on which model, how often you repeat the same context, and whether you ever batch or cache. Two teams paying the same per-token rate routinely differ 5× on the monthly invoice. This guide walks the seven levers that move the number — each tied to a calculator or live pricing so you can put real figures on it.

Want the bottom line for your traffic first? Price every model against your token counts in the AI API cost calculator, then come back and work the levers below.

1. Right-size the model

The single biggest lever. Flagship models can cost 10–30× more per output token than a capable mini/flash model — and most production calls (classification, extraction, routing, short answers) don’t need the flagship.

  • Tier your traffic: route easy requests to a cheap model, escalate only the hard ones.
  • Compare before you commit: the model directory has side-by-side pricing and a computed cost-per-workload for every cross-vendor pair.
Match the model to the task, not the demo

The model that wowed you in a demo is rarely the cheapest one that passes your evals. Run your eval set against a cheaper model before assuming you need the flagship.

2. Turn on prompt caching

If you send the same long prefix on many requests — a big system prompt, few-shot examples, tool definitions, or a fixed RAG document — prompt caching bills those repeated tokens at a fraction of the input rate (often a 75–90% discount on the cached portion).

This is free money for any app with a stable prefix. It has its own page with the live savings table and per-provider code: see prompt caching savings.

3. Move bulk work to the Batch API

For anything that doesn’t need a real-time answer — backfills, evals, embeddings jobs, nightly summaries — the Batch API typically runs at about half price with a 24-hour turnaround, and on separate, higher rate limits so it won’t starve your live traffic. If you’re running large synchronous loops for offline work, you’re overpaying.

4. Send fewer tokens

You pay per token, so the cheapest token is the one you don’t send. Where the bloat usually hides:

  • Unbounded conversation history. Trim with a sliding window or periodically summarize old turns — don’t replay the whole transcript every turn.
  • Over-stuffed RAG context. Lower top-k and chunk size; retrieve only what’s relevant.
  • Verbose system prompts. Tighten them; they’re sent on every call.
  • Bloated output format. Asking for verbose JSON wastes output tokens (the expensive kind). A compact format like TOON can cut structured-data tokens substantially — convert and measure with the JSON-to-TOON converter.
  • Right-size max_tokens. It’s reserved against your rate limit and caps spend; set it to what you actually need.

Measure any prompt before and after with the token counter — guessing is how bloat survives.

5. Cache and dedupe responses

Identical or near-identical requests? Cache the response in your own layer (Redis, a KV store) keyed on the normalized prompt. Common for FAQs, repeated classifications, and idempotent lookups. This is separate from prompt caching — it avoids the API call entirely.

6. Set max_tokens deliberately

Output tokens cost 3–5× more than input on most models, so runaway generations are where bills spike. Cap max_tokens to the real ceiling for each task, and prefer structured/short outputs where you can. (It also reserves rate-limit headroom — see the rate limits references.)

7. Monitor and budget

You can’t cut what you don’t measure.

  • Estimate up front with the cost calculator so your budget reflects real token math.
  • Set spend caps and alerts below your hard limit — a runaway loop hitting a budget cap shows up as an insufficient_quota error, which is a worse way to find out.
  • Track cost per request in your own metrics so a regression is visible before the invoice.

Put numbers on it

Rough order of impact for a typical chat/RAG app. Your mix varies — measure yours.
LeverTypical savingEffort
Right-size the model 30–80%Low–medium
Prompt caching Up to ~90% of repeated prefixLow
Batch API (offline work) ~50% on that workLow
Cut tokens / trim context 10–40%Medium
Response caching Varies (avoids the call)Medium
One router, cheapest model per call

If you want to route each request to the cheapest model that can handle it without maintaining several SDKs, a gateway helps. OpenRouter exposes many models behind one OpenAI-compatible API so you can switch or fall back per request. (Affiliate link.)

What to do next

  1. Estimate your current spend in the cost calculator to find the biggest line item.
  2. Right-size the model for your highest-volume call path — compare options in the model directory.
  3. Enable prompt caching if you have a stable prefix.
  4. Move offline work to the Batch API and trim tokens on your hottest prompts.

Frequently asked questions

What's the single biggest way to cut AI API costs?
Right-sizing the model. Flagship models cost many times more per token than capable mini/flash models, and most production calls don't need the flagship. Tier your traffic — cheap model by default, escalate only hard requests — and you can cut 30–80% before touching anything else.
How much does prompt caching actually save?
On the repeated portion of your prompt (system prompt, few-shot, fixed RAG context, tool defs), cached reads are typically billed at 10–25% of the standard input rate — a 75–90% discount on those tokens. It only helps when a long prefix is reused across requests. See the prompt caching guide for the live per-model savings table.
Is the Batch API worth it?
For anything non-real-time — evals, backfills, nightly jobs, embeddings — yes. It's usually about half the price with up to a 24-hour turnaround, and runs on separate rate limits so it won't compete with your live traffic.
Do output tokens cost more than input tokens?
Usually yes — output is commonly 3–5× the input rate. That's why runaway generations spike bills and why capping max_tokens and preferring concise outputs matters more than trimming input alone.
How do I know which lever to pull first?
Measure. Use the cost calculator to find your biggest line item (usually a high-volume call on an expensive model, or a huge repeated prefix). Pull the lever that targets that line — model right-sizing or prompt caching are almost always the top two.