Runbooks
Who is this for
Section titled “Who is this for”Whoever is on call when something breaks: the engineer with SSH access, and the operator at the station for the problems that need no server access.
- Find the symptom in the table below and open that runbook. Every runbook has the same six parts: Symptom → Likely causes → Diagnose → Act → Verify → Prevent.
- If you cannot tell what is wrong, run the five commands below first. They take under a minute and point at the right runbook.
- Stop guests before long repairs: send
/pauseto the bot. Lift it with/resume. - A restart is almost always safe:
bash /opt/ai-sticker-pack/deploy/update.sh --yes. Unfinished jobs return to the queue. - Failed packs can be retried from the admin panel for as long as the selfie exists (up to 72 hours).
How to use the runbooks
Section titled “How to use the runbooks”- Symptom tells you whether you are on the right page.
- Likely causes are ordered from most to least likely.
- Diagnose gives commands and the exact line to look for. Russian log lines are quoted verbatim because you will grep for them.
- Act fixes the cause you found. Do only the block that matches your diagnosis.
- Verify proves the fix. Do not skip it during an event.
- Prevent is for after the incident.
Commands run on the server as root (ssh root@<server>) unless a step says “workstation”, “tablet” or “admin panel”.
Paths and unit names are explained in Deploy and rollback.
First five commands for any incident
Section titled “First five commands for any incident”curl -s https://stickers.eventtalks.uz/healthsystemctl is-active sticker-bot sticker-api caddyjournalctl -u sticker-bot -n 50 --no-pager -o catjournalctl -u sticker-api -n 30 --no-pager -o catfree -m; df -h /| Command | Healthy | If not |
|---|---|---|
curl … /health |
{"ok":true,"db":"ok","active_jobs":N,"maintenance":null,…} |
No answer or 502 → Kiosk 502. "maintenance" not null → guests are paused, see Bot silent or Billing 402. |
systemctl is-active … |
active three times |
Any other word → Services stopped on a shared droplet |
| Bot journal | Recent STARTUP lines, job lines, no tracebacks |
Конфигурация некорректна → fix bot/.env, see Configuration. Conflict → Bot silent. FAILED → Generation failing. |
| API journal | Request lines, STARTUP |
Конфигурация некорректна → fix bot/.env |
free -m; df -h / |
Free memory and disk | → Disk and memory |
/health is answered by the API process. A healthy /health does not prove that the bot process is alive; that is
what the second command is for.
Symptom → runbook
Section titled “Symptom → runbook”| Symptom | Runbook |
|---|---|
| Guest scans the QR code, the bot does not react | Bot silent |
Bot answers every guest Сейчас идёт техобслуживание… (“maintenance in progress”) |
Bot silent, then Billing 402 if the source is auto |
Kiosk page or admin panel shows 502, a certificate error or does not load |
Kiosk 502 |
Alert Job N FAILED, guests get “something went wrong” |
Generation failing |
Alert 💳 Кончились деньги на AI-ключе (“the AI key ran out of money”) |
Billing 402 |
Alert about the daily image counter, RATE_LIMIT lines, packs slow or failing with 429 |
Rate limit 429 |
| One guest says the pack did not arrive | Pack not delivered |
Alert 5 мин без ссылки or STICKERSET_DELIVERY_FAILED |
Pack not delivered |
| Tablet frozen, black camera, “no connection”, stuck on one screen | Station stuck |
Lead cards do not arrive, arrive with ⚠️ Запасной канал, or buttons answer Заявки сейчас не ведутся… |
Leads missing or fallback |
Packs became slow, the service restarts by itself, No space left on device |
Disk and memory |
| Everything worked, then stopped after someone worked on the server | Services stopped on a shared droplet |
| A person asks to delete their data (lead or guest) | Data deletion request |
| Broken right after a deploy | Rollback |
Severity guide
Section titled “Severity guide”| Level | Meaning | Examples | Response |
|---|---|---|---|
| 1 | No guest can get a pack | Bot down, API down, provider out of money, daily quota exhausted | Act now; /pause if the fix takes more than a few minutes; tell the operator on site what to say to guests |
| 2 | Some guests are affected | Single failed jobs, slow packs, one tablet stuck | Fix between guests; retry failed jobs from the admin panel |
| 3 | Guests unaffected | Leads arriving through the fallback channel, backup failed, disk filling | Fix the same day |
Where to go next
Section titled “Where to go next”- Monitoring and alerts — every alert text and what it means.
- Event playbook — checks that prevent most of these incidents.
- Deploy and rollback — restart, update and rollback procedures.
- Reliability — why the system recovers the way it does.
Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.