Generation failing

Who is this for

The engineer on call when the alert chat shows Job N FAILED, and the operator who has to retry packs from the admin panel.

TL;DR

• A job gets 2 attempts with a 10 second pause between them. Then it is failed, the guest is told, and an alert is sent.
• The error column of the job says why. Read it before doing anything.
• Retry from the admin panel: status filter Упал → Повторить джоб. It works only while the selfie file still exists.
• If retry is refused, grant one more pack with Разрешить ещё раз and photograph the guest again.
• “Out of money” and “daily quota” failures have their own runbooks; a retry before fixing them just fails again.

Symptom

• Alert: ⚠️ [<EVENT_NAME>] Job N FAILED (попыток: 2) (“attempts: 2”), sometimes with (без ретрая) (“no retry”).
• The guest received Что-то пошло не так с генерацией твоего набора… (“something went wrong with generating your pack… come to the station, we will help by hand”).
• Admin panel card Паки shows a growing third number (failed).

Likely causes

1. Provider errors: 5xx, timeouts (AI_TIMEOUT_SEC, default 180 s per sheet request), no fallback key.
2. Invalid sheets: the model drew a wrong layout in every attempt and the composite fallback is disabled or also failed.
3. Cost cap: the job reached JOB_MAX_COST_USD.
4. The selfie file is gone (deleted by TTL) when a retry starts.
5. The provider ran out of money → Billing 402.
6. Rate limit or daily quota → Rate limit 429.
7. The link was not delivered within 15 minutes → Pack not delivered.
8. The bot process was killed during generation twice (memory) → Disk and memory.

Diagnose

sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db \
  "SELECT job_id, attempts, substr(error,1,200) FROM jobs WHERE status='failed' ORDER BY job_id DESC LIMIT 10;"

journalctl -u sticker-bot --since -30min --no-pager -o cat \
  | grep -E 'упал|FAILED|JOB_COST_LIMIT|BILLING_EXHAUSTED|RATE_LIMIT|FIGURE_FALLBACK|STAGE_SUMMARY' | tail -40

The alert text itself contains the same error after Ошибка: (“error”).

Error text or log marker	Cause	Go to
GenerationError, 503, 500, DEADLINE_EXCEEDED, TimeoutError	1 — provider trouble	Act 1
Alert Fallback: провайдер <x> упал, лист сгенерирован через <y> (“provider x failed, sheet generated through y”)	1 — primary provider failing, fallback is carrying the load	Act 1
FigureSheetInvalid; info message ℹ️ … figure не удался, собирается composite (“figure failed, building composite”)	2 — layout problems. With the fallback on, the guest still gets a pack.	Act 2
JobCostExceeded, log JOB_COST_LIMIT	3 — cost cap	Act 3
PermanentJobError: нет файла фото … (удалён по TTL?) (“no photo file, deleted by TTL?”)	4 — selfie gone	Act 4
PermanentJobError: неизвестный template_id / нет файла шаблона (“unknown template” / “no template file”)	Template missing on the server after a deploy	Roll back or redeploy the templates
BillingExhausted, log BILLING_EXHAUSTED	5	Billing 402
429, RESOURCE_EXHAUSTED, many RATE_LIMIT lines	6	Rate limit 429
STICKERSET_DELIVERY_FAILED	7	Pack not delivered
[killed during processing: попытки исчерпаны]; alert процесс бота падал посреди его генерации (“the bot process crashed during generation”)	8	Disk and memory

Check the provider side too: the admin panel card Ошибки за 24 ч (“errors in 24 h”) shows 429 / other, and the provider’s status page shows outages.

Act

1 — provider errors

• A short outage passes by itself. Wait a few minutes, then retry the failed jobs (below).

• If the fallback alert repeats and a fallback key is configured, swap the providers:

  E=/opt/ai-sticker-pack/bot/.env
  setenv() { grep -q "^$1=" $E && sed -i "s/^$1=.*/$1=$2/" $E || echo "$1=$2" >> $E; }
  setenv PRIMARY_AI_PROVIDER openai
  setenv FALLBACK_AI_PROVIDER gemini
  bash /opt/ai-sticker-pack/deploy/update.sh --yes

• With no fallback key and a long outage, stop taking guests so they do not wait for nothing: /pause in the bot, /resume when the provider is back.

2 — invalid sheets

• Single cases need no action when the composite fallback delivered a pack.
• Many cases in a row: look at the saved sheets in /opt/ai-sticker-pack/bot/data/debug_sheets/job<N>_sheet*_try*.png (they contain guest faces; do not copy them off the server).
• Make sure FIGURE_FALLBACK_ENABLED=1. As a last resort use the rollback flags in Configuration.

3 — cost cap

The pack spent more than JOB_MAX_COST_USD (default $1.5), usually many regenerations on a difficult photo. In the admin panel the button with the dollar amount, по стадиям (“by stage”), shows where the money went. Retry once: a retry gets its own cap. If it fails again, grant one more pack and retake the photo with better light and the face straight to the camera.

4 — selfie gone

A retry is impossible. Admin panel → the guest → Разрешить ещё раз (“allow one more”), then the guest goes through the station again.

Retry a failed job (admin panel)

1. Open https://stickers.eventtalks.uz/admin.
2. Status filter → Упал (“failed”), or search by name, @username or Telegram id.
3. Press Повторить джоб (“retry job”) and confirm.

The job returns to the queue with attempts reset to zero. The 5 and 15 minute deadlines count from the moment of the retry. The guest receives the link in the bot as usual. The panel refuses the retry with a reason when:

Refusal	Meaning	Do
селфи уже удалено (TTL/после пака)… (“selfie already deleted”)	The photo file no longer exists	Разрешить ещё раз, new photo
лимит паков гостя исчерпан… (“guest’s pack limit is used up”)	The guest already has another pack counted	Разрешить ещё раз, then retry again
джоб не найден или не в статусе failed (“job not found or not failed”)	Someone already retried it	Refresh the page

Verify

journalctl -u sticker-bot --since -5min --no-pager -o cat | grep -E 'STAGE_SUMMARY|STICKERSET_CONFIRMED|FAILED'

Expected: a STAGE_SUMMARY … result=delivered line and STICKERSET_CONFIRMED job N for the retried job. In the admin panel the row turns to “ready” and shows the pack link.

Prevent

• Configure a fallback provider key before the event.
• Keep the station lighting even; most cost-cap and layout failures start with a bad photo.
• Watch STAGE_SUMMARY lines during the event; rising retries and r429 values warn before jobs start failing.
• Do not push code after the freeze: a missing template or module shows up as failed jobs.

Where to go next

• Billing 402 — the provider ran out of money.
• Rate limit 429 — minute limits and the daily quota.
• Generation flow — the stages a job goes through.
• Runbooks — back to the symptom table.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.