Monitoring and alerts

Who is this for

The engineer on call who has to understand an alert, and the operator who watches the admin panel during an event.

TL;DR

All alerts go to one Telegram chat, ADMIN_CHAT_ID, as messages that start with ⚠️ [<EVENT_NAME>] (problems) or ℹ️ [<EVENT_NAME>] (information).
Alerts are sent by the running services. A stopped bot, a stopped Caddy or a full disk produces no alert; add an external uptime check.
/health reports the API and the database, not the bot process. Check systemctl is-active sticker-bot separately.
Logs: journalctl -u sticker-bot, journalctl -u sticker-api, and the same content in /opt/ai-sticker-pack/logs/app.log and api.log.
The admin panel at /admin is the live dashboard: spend, balance estimate, queue, errors, daily image counter, per-guest actions.

Alerts

Source: bot/services/alerts.py and every caller of send_admin_alert.

Format. ⚠️ [<EVENT_NAME>] <text> or ℹ️ [<EVENT_NAME>] <text>, cut at 4096 characters. EVENT_NAME comes from bot/.env.

Destination. The chat in ADMIN_CHAT_ID (a private chat or a group). If it is empty, alerts are only written to the log as ADMIN_CHAT_ID не задан, алерт только в лог (“not set, alert only in the log”); in prod the bot refuses to start without it.

Cooldown. Alerts with a key are sent at most once per cooldown period. Suppressed repeats are counted, logged as alert [<key>] подавлен (cooldown), and the next alert that goes out ends with (+ ещё N таких же за последние M мин) (“plus N more like this in the last M minutes”). Cooldown state lives in process memory and resets on restart.

Failure. If Telegram rejects the alert, the error is logged as Не удалось отправить алерт в чат … (“could not send the alert”) and the job continues. An alert can never break a job.

#	Trigger	Text (stable prefix, verbatim)	Gloss	Cooldown	What to do
1	A job failed after all attempts, or with an error that is not retried	`Job N FAILED (попыток: K)` + optional `(без ретрая)`, then `Гость: …, шаблон: …, сессия: …`, `Ошибка: …`, and `Гостю отправлено «поможем вручную» — он может подойти к станции.`	“Job N failed (attempts: K) (no retry). Guest, template, session. Error. The guest was told ‘we will help by hand’ — they may come to the station.”	none: one per failed job	Generation failing
2	Same as 1, when the cause is “out of money”	Same first lines, ending `Гостю отправлено «генерация временно недоступна, фото сохранено» — после пополнения баланса «Повторить» в админке.`	“The guest was told ‘generation temporarily unavailable, photo saved’ — after topping up, press Retry in the admin panel.”	none	Billing 402
3	At bot start: a job was in `processing` when the process died and has no attempts left	`Job N FAILED: процесс бота падал посреди его генерации (попыток: K).`	“The bot process crashed in the middle of its generation.”	none	Disk and memory
4	The AI provider answered “out of money” and the automatic pause was switched on	`💳 Кончились деньги на AI-ключе (402, job N).` followed by the provider error and three numbered steps	“The AI key ran out of money. New jobs are paused. 1) Top up. 2) `/resume`. 3) Retry failed packs.”	key `billing_exhausted`, 10 min; also only when the pause is newly switched on	Billing 402
5	The primary provider failed for a sheet and the fallback provider produced it	`Fallback: провайдер <failed> упал, лист сгенерирован через <used> (job N).` then `Ошибка <failed>: …` and `Если повторяется — переключи PRIMARY_AI_PROVIDER=<used> в .env и перезапусти бота.`	“Provider X failed, the sheet was generated through Y. If it repeats, switch `PRIMARY_AI_PROVIDER` and restart the bot.”	key per provider pair, 10 min	Generation failing. Today this alert never fires: no fallback key is configured — see Known gaps, R1.
6	The `figure` render failed for a job and the pack is being built in `composite` mode instead	`ℹ️ … Job N: figure не удался, собирается composite (гость получит пак как обычно).` then `Причина: …` and the path pattern of the saved sheets	“Figure failed, building composite (the guest gets a pack as usual). Reason.”	none	Information. Act only if frequent: Generation failing
7	Image requests in the provider’s day reached `IMAGE_RPD_WARN_FRAC` × `IMAGE_RPD_LIMIT`	`⚠️ Картинок за сутки Google (<date>, тихоокеанское время): U из L (P%). Квота сбрасывается в полночь по PT; при исчерпании новые паки упадут с 429 до сброса.`	“Images in the provider’s day (Pacific time): U of L. The quota resets at midnight PT; when exhausted, new packs fail with 429 until the reset.”	once per Pacific-time day (stored in the database)	Rate limit 429
8	A job has had no link for 5 minutes since the photo	`5 мин без ссылки, гостю ушло «Дорабатываем детали».` then the job description: `job N, пак …, user …, этап: …, с «Генерирую…» прошло X мин`	“5 minutes without a link; the guest was sent ‘finishing the details’. Job, pack, user, stage, minutes since ‘generating’.”	once per job	Pack not delivered
9	A job has had no link for 15 minutes; it is now `failed`	`STICKERSET_DELIVERY_FAILED: ссылка не ушла за 15 мин.` then the job description and `Гостю отправлено «поможем вручную» — он может подойти к станции.`	“The link was not sent within 15 minutes. The guest was told ‘we will help by hand’.”	once per job	Pack not delivered
10	A lead failed to publish to the leads group 5 times or more	`Заявка #N не публикуется в группу заявок уже K раз: <error>. Заявка в БД, видна в /stickers_new; публикация повторяется каждые 15 минут.`	“Lead #N has failed to publish K times. It is in the database, visible in `/stickers_new`; publishing is retried every 15 minutes.”	key `leads_publish`, 30 min	Leads missing or fallback
11	More than `LEADS_HOURLY_LIMIT` leads arrived within an hour	`Заявки: больше N за час — лишние отклоняются (429). Похоже на флуд формы лендинга.`	“More than N leads per hour — the excess is rejected (429). Looks like a flood of the landing form.”	key `leads_hourly_limit`, 60 min	Leads missing or fallback

The stage names in alerts 8 and 9 are в очереди (“queued”), генерация (“generating”) and проверка ссылки на пак (“checking the pack link”). Alert 7 shows the warning sign twice because the text has its own. Alert 11 is sent by the API process; all others by the bot process.

Job alerts contain the guest’s numeric Telegram id. Treat the alert chat as private.

What is not alerted

Nothing in the repository watches these. You find them only by looking, or with an external check.

Condition	How to see it
`sticker-bot` stopped or crash-looping (also means: no alerts at all)	`systemctl is-active sticker-bot`
`sticker-api` or Caddy down	External HTTPS check on `/health`
Disk full, memory pressure, OOM kills	Disk and memory
Nightly backup failed	`FAIL` line in `/var/log/sticker-backup.log`; `systemctl status sticker-backup.service`
Maintenance mode switched on by `/pause` or by `MAINTENANCE_MODE=1`	`/health` field `maintenance`; admin panel banner
A second bot instance on the same token	`Conflict` in the bot journal
Wrong leads secret (`LEAD_REJECTED`), leads arriving through the fallback channel	API log; `⚠️ Запасной канал` messages in the leads group
Failed logins to the admin panel	`ADMIN_LOGIN_FAILED`, `ADMIN_LOGIN_BLOCKED` in the API log
TTL cleanup errors	`Ошибка TTL-очистки` in the bot log
Certificate renewal problems	`journalctl -u caddy`

`/health`

curl -s https://stickers.eventtalks.uz/health
curl -s localhost:8000/health          # on the server, bypassing Caddy

{"ok": true, "db": "ok", "active_jobs": 0, "maintenance": null, "version": "b715418", "render_mode": "figure", "app_env": "prod"}

Field	Values	How to read it
`ok`	`true` / `false`	`false` only together with HTTP `503`: the database did not answer
`db`	`"ok"` / `"error"`	`"error"` → disk full, file permissions or a damaged file: Disk and memory
`active_jobs`	number	Jobs in `queued`, `processing` or `delivering`. `0` means a restart interrupts nobody. A number that never falls means the worker is not taking jobs.
`maintenance`	`null`, `"env"`, `"admin"`, `"auto"`	Not `null` → guests are not served. See Configuration.
`version`	short commit hash, or `unknown`	From the `VERSION` file written by `push.ps1`. Confirms which code is deployed.
`render_mode`	`figure` / `composite`	The mode the API process read. The worker logs its own in `JOB_RENDER_MODE … (обработка, …)`.
`app_env`	`prod` / `dev`	Must be `prod` on an event server

The endpoint returns no secrets. It is served by the API process: it proves that the API and the database work, and says nothing about the bot process, Telegram, or the AI provider.

External uptime check

Recommendation — nothing like this exists in the repository. Set up any external HTTP monitor (a hosted uptime service, or a cron job on another machine) with two checks:

GET https://stickers.eventtalks.uz/health every minute; alert when the status is not 200 or the body lacks "ok":true. Optionally alert when "maintenance":null is missing during event hours.
A check that the bot process is alive, since /health cannot see it. The simplest form is a cron job on the server that calls the monitor’s heartbeat URL only when the bot is active:
Terminal window
```
* * * * * systemctl is-active --quiet sticker-bot && curl -fsS -m 10 -o /dev/null <heartbeat URL>
```

Send both to a channel other than the alert chat’s bot, so they still arrive when the bot is down.

Logs

What	Where	Rotation
Bot: polling, worker, delivery, cleanup, leads loop	`journalctl -u sticker-bot`; same lines in `/opt/ai-sticker-pack/logs/app.log`	File: 5 MB × 3 by the application, plus daily logrotate with `copytruncate`, 14 days, compressed. Journal: system default.
API: kiosk, admin panel, `/leads`	`journalctl -u sticker-api`; application lines in `/opt/ai-sticker-pack/logs/api.log`	Same as above
Caddy service: start, certificates	`journalctl -u caddy`	System default
Caddy access log for our site	`/var/log/caddy/stickers.log`	20 MiB per file, 14 files
Backups	`/var/log/sticker-backup.log`; `journalctl -u sticker-backup.service`	Not rotated; one line per run
Landing form	Vercel → project → Logs, lines starting with `[request]`	Vercel retention

Format of application lines: <date> <time> <LEVEL> <module>: <message>. Useful forms:

journalctl -u sticker-bot -f                                   # follow live
journalctl -u sticker-bot --since -30min --no-pager -o cat     # last 30 minutes, message only
grep -h 'STAGE_SUMMARY' /opt/ai-sticker-pack/logs/app.log*     # across rotated files

Logs contain numeric Telegram ids and session tokens, never photos, names from leads, phone numbers or secrets.

What to grep for

Marker	Process	Meaning
`STARTUP`	both	Start-up lines: render mode and its source, likeness and consistency modes, allowed updates, kiosk URL and version
`Конфигурация некорректна`	both	Start-up validation error; the process exits
`STARTUP отменён`	bot	Another `main.py` already runs on this machine (instance lock)
`Воркер очереди запущен, параллельных слотов: N`	bot	Worker started with N slots
`Вернул в очередь N job(ов)`	bot	Jobs found in `processing` at start were re-queued
`SHUTDOWN`	bot	Clean stop sequence; `SHUTDOWN: остановлен за N c` is the last line
`JOB_RENDER_MODE job=N mode=…`	both	Mode at creation (API) and at processing (bot); the bot line is the one that counts
`STAGE_SUMMARY job=N total=… … result=…`	bot	One line per job attempt: total time, per-stage times, retries, `r429`, result (for example `delivered`, `retry`, `failed`, `expired`, `delivery_failed`, `shutdown_requeued`)
`STAGE_TIMING job=N stage=…`	bot	Per-stage detail
`RATE_LIMIT`	bot	A provider call got `429` and is waiting to retry
`JOB_COST_LIMIT`	bot	A job hit `JOB_MAX_COST_USD`
`BILLING_EXHAUSTED`	bot	Provider out of money
`MAINTENANCE_ON` / `MAINTENANCE_OFF`	bot	Database pause switched on or off
`MAINTENANCE:`	both	A guest message or a kiosk request was turned away
`IMAGE_RPD_WARN`	bot	Daily image counter crossed the warning threshold
`FIGURE_FALLBACK_TO_COMPOSITE`	bot	Figure render failed; composite used
`упал (попытка`	bot	A job attempt failed, with the traceback after it
`DELAY_NOTICE`	bot	5-minute notice sent
`STICKERSET_CONFIRMED`	bot	Pack verified, link sent
`STICKERSET_DELIVERY_FAILED`	bot	15-minute deadline reached
`PACK_LIMIT`	API	A guest who already has a pack tried again
`kiosk:`	API	Session created, outfit chosen, photo queued, session dropped by the tablet (with reason)
`storage cleanup`	bot	TTL cleanup removed files
`Миграция БД`	both	Schema migration ran; the line names the automatic database copy
`ADMIN_LOGIN_FAILED` / `ADMIN_LOGIN_BLOCKED`	API	Wrong admin panel credentials; address blocked
`ADMIN <user> (<address>):`	API	An admin panel action: grant, reset, retry
`admin: /stats`, `admin: /admin_stats`, `admin: /resume`	bot	Admin commands in the bot
`LEAD_NEW`, `LEAD_DUPLICATE`, `LEAD_REPEAT`, `LEAD_REJECTED`, `LEAD_INVALID`, `LEAD_RATE_LIMITED`	API	Outcome of each `POST /leads`
`LEAD_PUBLISHED`, `LEAD_PUBLISH_FAILED`, `LEAD_PUBLISH_RETRY`	bot	Card publishing
`LEAD_STATUS`, `LEADS_CMD`, `LEAD_ERASED`, `LEAD_DELETED_TEST`, `LEAD_REMIND`, `LEAD_DIGEST`	bot	Actions in the leads group and scheduled housekeeping
`LEADS:`	bot	Leads module state at start: group membership, command menu, on or off
`Conflict`	bot	A second instance is polling the same token
`alert [`	bot	An alert was suppressed by its cooldown

Live view during an event:

journalctl -u sticker-bot -f | grep -E 'STAGE_SUMMARY|RATE_LIMIT|FAILED|JOB_COST|BILLING|ERROR'

Admin panel as a dashboard

https://stickers.eventtalks.uz/admin, HTTP Basic login with ADMIN_PANEL_USER and ADMIN_PANEL_PASSWORD. Press Обновить (“refresh”) to reload. Source: bot/admin/index.html, bot/services/admin_data.py.

Cards

Card	Shows	Watch for
`Потрачено всего` (“spent in total”)	Estimated AI spend since the database began	—
`Потрачено сегодня` (“spent today”)	Estimated spend since local midnight (`DISPLAY_UTC_OFFSET_HOURS`)	Compare with `packs × 0.42`
`Баланс (оценка)` (“balance, estimate”)	Entered real balance minus spend since it was entered; or `BUDGET_USD` minus all spend. Button `поправить по биллингу` (“correct from billing”) stores a real balance.	It is an estimate from token counts. The provider console is the truth.
`Паки` (“packs”)	`ready / in progress / failed`	The third number growing
`Очередь сейчас` (“queue now”)	Total active jobs, split into waiting, generating, link check	Waiting jobs that do not move
`Среднее время` (“average time”)	Photo → link, average over finished jobs	Well above 1–1.3 minutes
`Ошибки за 24 ч` (“errors in 24 h”)	`429 / other` failed provider calls	Either number climbing
`Картинок за сутки Google` (“images in the provider’s day”)	`used / limit`, percent, the Pacific date	Approaching the limit
`Режим` (“mode”)	Render mode and deployed version	Unexpected version

Banners

Banner	Meaning
`Режим обслуживания (<source>): <reason>…`	Maintenance mode is on; the text says how to lift it
`Запросов картинок за сутки Google: U из L — больше 70%…`	Daily image counter past the warning threshold
`Цены не заданы в .env (…) — расход и баланс занижены.` (“prices not set — spend and balance are too low”)	Fill the listed `PRICE_*` settings

Guest table and actions

Search by name, @username or Telegram id; filter by status: Готов (ready), Генерируется (generating), В очереди (queued), Упал (failed), Начал, без фото (started, no photo). Each row shows status, created and ready times, attempts, render mode, the pack link, the job cost with a per-stage breakdown and, when available, the stylised anchor portrait with a consistency summary.

Button	Effect
`Разрешить ещё раз` (“allow one more”)	Grants the guest one more pack
`Повторить джоб` (“retry job”)	Only on failed jobs: back to the queue with attempts reset; refused if the selfie is gone or the pack limit is used up
`Сбросить гостя` (“reset guest”)	Deletes the guest’s rows (user, sessions, jobs) after an automatic database copy. Destructive: see Backups and data lifecycle.

Every action is recorded in Журнал действий (audit) (“action log”) at the bottom of the page, with the user name and address. The panel never shows photos, file paths or the appearance description.

Bot commands for admins

Private chat with the bot only, and only for accounts in ADMIN_USER_IDS. For anyone else the bot does not react.

Command	Answer
`/stats`	`📊 Статистика`: users (total, consented, with a pack), sessions by status, jobs by status, average and maximum photo → pack time, age of the oldest queued job
`/admin_stats`	`📊 Сводка`: the same numbers as the admin panel cards in text form — maintenance state, spend total and today, balance estimate, packs, queue, 429s and other errors in 24 hours, daily image counter, render mode, panel link
`/pause`	Switches maintenance mode on (source `admin`)
`/resume`	Removes the database pause

In the leads group, for accounts in LEADS_ADMIN_IDS: /stickers_stats, /stickers_new, /stickers_lead <N>, /stickers_export, /stickers_forget <N>. See Bot commands.

Where to go next

Runbooks — what to do about each alert.
Observability — the design behind these signals and its gaps.
Event playbook — when to look at which signal during an event.
Bot commands — the full command reference.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.