Billing 402

Who is this for

The owner who can pay the AI provider, and the engineer on call who resumes the service afterwards.

TL;DR

• The alert starts with 💳 Кончились деньги на AI-ключе (“the AI key ran out of money”).
• The system pauses itself: guests get the maintenance text, the kiosk starts no sessions, the worker takes no jobs.
• Fix order: top up at the provider → /resume in the bot → retry the failed packs in the admin panel.
• Photos of the affected guests are kept for up to 72 hours, so their packs can be retried without them coming back.
• Jobs that were waiting in the queue fail after 15 minutes of pause. Retry those too.

Symptom

• Alert in the admin chat:

  ⚠️ [<EVENT_NAME>] 💳 Кончились деньги на AI-ключе (402, job N).
  <provider error text>
  
  Новые job на паузе: бот гостям — «техобслуживание», киоск не начинает сессии.
  1) Пополни баланс (AI Studio → Billing). 2) /resume в боте. 3) Упавшие паки — «Повторить» в админке (фото гостей хранятся до 72 ч).

  Gloss: “New jobs are paused: the bot tells guests ‘maintenance’, the kiosk starts no sessions. 1) Top up the balance. 2) /resume in the bot. 3) Failed packs — ‘Retry’ in the admin panel (guest photos are kept for up to 72 hours).”

• A second alert for the job itself: Job N FAILED … (без ретрая) with BillingExhausted in the error.

• The affected guest received Не получилось сделать твой набор: генерация временно недоступна… (“could not make your pack: generation is temporarily unavailable… your photo is saved, we will redo it”).

• Every other guest gets Сейчас идёт техобслуживание, загляни чуть позже 🛠.

• /health shows "maintenance":"auto"; the admin panel shows a maintenance banner.

Likely causes

1. The prepaid balance of the AI provider project is exhausted (Gemini answers HTTP 402, “prepayment credits are depleted”).
2. A spend cap on the provider project was reached.
3. With a fallback key configured: both providers answered “out of money” (OpenAI reports it as insufficient_quota). If only one did, the fallback provider carries on and there is no pause.

Diagnose

curl -s https://stickers.eventtalks.uz/health
journalctl -u sticker-bot --since -1h --no-pager -o cat | grep -E 'BILLING_EXHAUSTED|MAINTENANCE_ON|MAINTENANCE_OFF'
sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT value, updated_at FROM settings WHERE key='maintenance';"
sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db \
  "SELECT job_id, substr(error,1,120) FROM jobs WHERE status='failed' AND error LIKE '%BillingExhausted%' ORDER BY job_id DESC;"

Look for:

• BILLING_EXHAUSTED job N: … — стоп, job в failed, новые job на паузе (“stop, job failed, new jobs paused”);
• MAINTENANCE_ON source=auto reason=402 у AI-провайдера (job N);
• the settings row with "source": "auto".

Then confirm in the provider console (Billing) that the balance is zero or the cap is reached. The admin panel’s Баланс (оценка) card is only an estimate from token counts; it cannot see the real balance.

How the system behaved (so you know what guests saw): a “no money” answer is not treated as a rate limit. There are no 5/15/30 second waits and no second attempt. The job fails at once, the pause is written to the database, and both processes see it without a restart. The alert is sent once per pause (and at most once per 10 minutes).

Act

1. Owner — top up. AI provider console → Billing → add prepaid credit, and raise the spend cap if that was the limit. Size the top-up with the estimate in the Event playbook.

2. Engineer — resume. Send /resume to the bot in a private chat from an account in ADMIN_USER_IDS. Expected answer: Режим обслуживания снят — гости снова могут начать, очередь пошла. Упавшие из-за 402 паки — «Повторить» в админке. (“maintenance lifted — guests can start again, the queue is moving”).

   Without bot access:

   sqlite3 /opt/ai-sticker-pack/bot/data/app.db "DELETE FROM settings WHERE key='maintenance';"

3. Operator — retry failed packs. Admin panel → status filter Упал → Повторить джоб for every job that failed during the outage: those with BillingExhausted, and those with STICKERSET_DELIVERY_FAILED that sat in the queue past 15 minutes. Details: Generation failing.

4. Operator — record the balance. Admin panel → Баланс (оценка) → поправить по биллингу (“correct from billing”) → enter the real balance from the provider console.

If /resume answers Пауза из БД снята, но в bot/.env стоит MAINTENANCE_MODE=1 — режим остаётся (“the database pause is lifted, but MAINTENANCE_MODE=1 is set — the mode stays”), also set MAINTENANCE_MODE=0 and run bash /opt/ai-sticker-pack/deploy/update.sh --yes.

Do not /resume before the top-up is visible at the provider: the next job fails with 402 and the pause comes back.

Verify

curl -s https://stickers.eventtalks.uz/health        # "maintenance":null
journalctl -u sticker-bot --since -5min --no-pager -o cat | grep -E 'MAINTENANCE_OFF|режим обслуживания снят|STAGE_SUMMARY'

Expected: MAINTENANCE_OFF (settings), Воркер #N: режим обслуживания снят — беру job (“maintenance lifted — taking jobs”), then STAGE_SUMMARY … result=delivered for the retried jobs. Run one test pack from the tablet.

Prevent

• Top up before the event with the margin from the Event playbook and enable auto top-up if the provider offers it.
• Check the real balance on the morning of the event and enter it in the admin panel.
• Keep EVENT_MODE=1 and a separate TEST_GEMINI_API_KEY so test scripts cannot spend the guests’ balance.
• Configure a fallback provider key: with it, one provider running out of money does not stop the event.
• Fill in all PRICE_* settings so the spend estimate in the panel is not too low.

Where to go next

• Generation failing — retrying failed packs.
• Rate limit 429 — the quota problem that looks similar but is not about money.
• Cost — what a pack costs and why.
• Runbooks — back to the symptom table.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.