Rate limit 429

Who is this for

The engineer on call when packs slow down or fail with 429, or when the daily image counter alert arrives.

TL;DR

• Two different limits answer 429: a per-minute limit that clears in a minute or two, and a daily quota that clears only at midnight Pacific time.
• The per-minute limit is handled automatically: each call waits about 5, 15 and 30 seconds and retries on the same provider.
• The daily quota is not waited for. The call fails at once and goes to the fallback provider if one is configured.
• IMAGE_RPD_LIMIT only drives the counter and the alert. It does not limit anything; the provider does.
• When the daily quota is gone and there is no fallback, pause the station (/pause) until the reset or raise the tier.

Symptom

• Journal lines RATE_LIMIT <label>: 429, жду N c (повтор K/3) (“429, waiting N s, retry K of 3”).

• Packs take much longer than the usual 1–1.3 minutes, or fail with 429 / RESOURCE_EXHAUSTED in the error.

• Alert (once per Pacific-time day):

  ⚠️ [<EVENT_NAME>] ⚠️ Картинок за сутки Google (<YYYY-MM-DD>, тихоокеанское время): 700 из 1000 (70%). Квота сбрасывается в полночь по PT; при исчерпании новые паки упадут с 429 до сброса.

  Gloss: “Images in the provider’s day (Pacific time): 700 of 1000. The quota resets at midnight PT; when it runs out, new packs will fail with 429 until the reset.”

• Admin panel card Ошибки за 24 ч shows a growing first number (429s); card Картинок за сутки Google is near the limit and a warning banner appears.

Likely causes

1. Per-minute request limit of the provider tier, hit by several packs generating at once.
2. Daily image quota of the provider project exhausted (the error text contains PerDay or “per day”).
3. Something else is using the same provider project: a test script, a local bot, another product.
4. Not a rate limit at all: “out of money” is HTTP 402 and has its own alert → Billing 402.

Diagnose

# how many 429s recently, and are they waits or failures
journalctl -u sticker-bot --since -15min --no-pager -o cat | grep -c RATE_LIMIT
journalctl -u sticker-bot --since -15min --no-pager -o cat | grep -E 'RATE_LIMIT|PerDay|per day|RESOURCE_EXHAUSTED' | tail -20

# effect on packs: r429 = number of 429s, wait429 = seconds spent waiting
journalctl -u sticker-bot --since -30min --no-pager -o cat | grep STAGE_SUMMARY | tail -10

# has the daily-counter warning fired (the line shows "used of limit" and the Pacific date)
journalctl -u sticker-bot --since -24h --no-pager -o cat | grep IMAGE_RPD_WARN

Also open the admin panel: the card Картинок за сутки Google shows used / limit and the Pacific date.

What you see	Cause
RATE_LIMIT … жду … lines, jobs still finish, STAGE_SUMMARY shows r429 above 0	1 — per-minute limit, being absorbed
Error text with PerDay / “per day”; jobs fail quickly; counter near the limit	2 — daily quota
Counter far higher than packs today × 4.3	3 — someone else is using the project
Alert with 💳	4 → Billing 402

How the backoff works (ai_pipeline.py): a call that gets 429 waits 5 s, then 15 s, then 30 s (or the delay the provider suggests, capped at 60 s), each plus up to 20% random jitter, and retries on the same provider. After the third retry the call fails; the pipeline then tries the fallback provider, and the job itself has 2 attempts. A daily-quota answer skips the waits. The counter in the admin panel counts image requests except those answered 429 or 402.

Act

1 — per-minute limit

• Usually nothing. It clears within a minute or two.

• If it keeps happening, lower the parallelism and restart (the value 4 is an example, not a tested setting):

  E=/opt/ai-sticker-pack/bot/.env
  setenv() { grep -q "^$1=" $E && sed -i "s/^$1=.*/$1=$2/" $E || echo "$1=$2" >> $E; }
  setenv AI_MAX_PARALLEL_IMAGES 4
  bash /opt/ai-sticker-pack/deploy/update.sh --yes

  AI_MAX_PARALLEL_IMAGES (default 9) caps simultaneous image requests for the whole process. WORKER_SLOTS=1 is the stronger measure: one pack at a time.

2 — daily quota exhausted

Pick one:

• Raise the tier or quota in the AI provider console. Afterwards set IMAGE_RPD_LIMIT to the new limit so the counter and the alert stay meaningful, and restart.
• Switch to the fallback provider if a key is configured: setenv PRIMARY_AI_PROVIDER openai, setenv FALLBACK_AI_PROVIDER gemini, update.sh --yes.
• Wait for the reset. Pause the station so guests are not photographed for nothing: /pause in the bot, /resume after the reset. The reset is at midnight Pacific time: 08:00 UTC during US standard time, 07:00 UTC during US daylight time; add your UTC offset.

To stretch what is left of the quota, reduce image requests per pack: setenv CONSISTENCY_MAX_REGEN 0 (no pack-level regenerations) and update.sh --yes. Quality drops; undo it after the event.

3 — another consumer

Stop it: local bots, test scripts (tools/*.py). Set EVENT_MODE=1 on the server and in local .env files so paid tools refuse to run on the production key.

After any of these retry failed jobs from the admin panel: Generation failing.

Verify

journalctl -u sticker-bot --since -5min --no-pager -o cat | grep -c RATE_LIMIT        # 0 or falling
journalctl -u sticker-bot --since -5min --no-pager -o cat | grep STAGE_SUMMARY | tail -3

Expected: recent STAGE_SUMMARY lines with result=delivered and r429=0. One test pack arrives in normal time.

Prevent

• Before the event check the tier’s per-minute and per-day limits against expected packs × 4.3 image requests (Event playbook).
• Know the local reset time for the event date.
• Keep test packs on the event day to a minimum; they use the same daily quota.
• Use TEST_GEMINI_API_KEY in a separate provider project for all tests.
• Configure a fallback provider.

Where to go next

• Billing 402 — the money problem with a similar face.
• Generation failing — retrying the packs that failed meanwhile.
• Capacity — how many packs per hour and per day the limits allow.
• Runbooks — back to the symptom table.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.