ADR-011: Cost guard rails — account every call, cap every job, pause when money runs out

Who is this for

Developers, operators and the owner — read this before running anything that calls a paid API.

TL;DR

Each call to a provider is written to api_usage immediately, with tokens, stage and estimated cost.
A job stops when its estimated cost passes JOB_MAX_COST_USD (default 1.5).
HTTP 402 “no credits” fails the job at once, alerts the admin and switches on maintenance mode automatically.
EVENT_MODE=1 blocks paid developer tools on the production key.
Status: accepted.

Context

The load test on the server stopped at the eighth pack because the provider’s prepaid credits ran out. The pipeline treated the answer as a rate limit: each call waited through three backoff pauses, each job took about three minutes to fail, and only the first failure produced an alert. At an event that would mean every guest waiting minutes for a “we will help you manually” message while the cause — an empty balance — stayed invisible (LOAD_TEST_REPORT.md).

Separately, more than half of all development spend happened on a single day of experiments (COST_HISTORY.md), and the developer tools share the daily image quota with guests if they use the same key.

Decision

Accounting. bot/services/usage.py records every attempt — including failures, at zero cost — with stage, resolution and token counts. Prices come only from PRICE_* environment variables; an empty price contributes zero and the admin panel warns that spend is understated. The figure is an estimate from tokens, not the provider’s balance.
Per-job ceiling. Before each call the job’s accumulated cost is checked; over the ceiling the job fails without retry or fallback and an alert is sent. A manual Retry from the admin panel starts a fresh ceiling.
402 is not a rate limit. No backoff. If a fallback provider has a key it is tried; otherwise the job fails, the guest is told generation is temporarily unavailable and the photo is kept, the admin gets one alert per ten minutes, and new jobs are paused through maintenance mode. /resume lifts the pause after topping up.
Daily quota counter. Image requests are counted per Pacific-time day against IMAGE_RPD_LIMIT; at IMAGE_RPD_WARN_FRAC the admin panel shows a warning and one alert is sent per day.
Bounded retries everywhere. Job attempts, sheet attempts, backoff pauses, anchor retries and regenerations all have fixed limits; a job that keeps killing the process is failed at start instead of looping.
Event guard. With EVENT_MODE=1 the paid tools in tools/ refuse to start unless a separate TEST_GEMINI_API_KEY from a different provider project is set (bot/services/event_guard.py). In production the service refuses to start if the test key equals the production key.
Working rule. Tests use mocks only. A live call is made only to judge a picture by eye, with a budget named in advance and a hard stop.

Alternatives considered

Alternative	Why not
Rely on the provider’s billing dashboard	It lags by hours; the balance is debited at once.
Rely on a provider-side spend cap alone	It stops everything silently and gives no per-job protection.
Keep retrying on 402	Burns guest time and hides the cause.

Consequences

Prices must be kept up to date by hand in .env.
An operator has to top up the balance and send /resume; nothing resumes on its own.
Failed jobs from the pause window are retried manually from the admin panel while the selfie still exists.

Evidence

Commits 3f5a8af, 54ad838, 0390077, b890475, a20171f, 2fa8f08.
tests/test_billing.py, tests/test_costs.py, tests/test_event_guard.py.

Where to go next

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.