Event playbook

Who is this for

The team that runs an event with AI Sticker Pack: the person who owns the money and accounts, the person at the tablet, and the engineer who can log in to the server.

TL;DR

• Freeze code before the rehearsal. After the freeze only bot/.env flags and rollbacks go to the server; code only for an incident.
• Money and quota stop events, not bugs: top up the prepaid AI balance and check the daily image quota before guests arrive.
• Take a database backup and a server snapshot after the final configuration.
• On the morning of the event /health must show "maintenance": null and one test pack must arrive from the real tablet.
• No second copy of the bot may run anywhere with the same token. Stop local python bot\main.py.
• After the event: export statistics and cost, verify that photos are deleted within the TTL, set EVENT_MODE=0.

Roles

Role	Who	Needs
Owner	Owns accounts, money and the go/no-go decision	AI provider console, hosting, Vercel, BotFather
Operator on site	Stands at the station, helps guests	Admin panel password, the tablet, a phone with the bot
Engineer on call	Reachable for the whole event	SSH to <server>, the repository checkout, alert chat membership
Client/venue	Provides the space	Power, internet, branding assets

One person can hold several roles. Somebody must be in the alert chat (ADMIN_CHAT_ID) with notifications on.

Planning numbers

All from docs/LOAD_TEST_REPORT.md (7 packs in a row, one station, 2 worker slots, 2 vCPU / 4 GB server) unless stated otherwise.

Quantity	Value
Photo → pack link	Median 56.4 s, maximum 74.8 s in the test; the report expects about 1–1.3 minutes with real Telegram calls
Cost per pack	$0.417 average at the prices of the test date
Image requests per pack	4.29 average
Peak process memory	1420 MB
Hard cost cap per job	JOB_MAX_COST_USD, default 1.5 (bot/config.py)
Daily image quota	IMAGE_RPD_LIMIT, default 1000 requests per Pacific-time day; alert at IMAGE_RPD_WARN_FRAC = 70%

Estimate — budget. Expected generation spend = expected packs × $0.42. The report recommends a prepaid balance of at least $50 for about 70 guests (expected spend about $29–30), which is roughly 1.7 × the expected spend, to cover test packs and retries. Formula: prepaid ≥ packs × 0.42 × 1.7. The balance ran out in the middle of that load test, which is why this check is on every list below.

Estimate — quota. Image requests = expected packs × 4.3. With a limit of 1000 that is about 230 packs per Pacific-time day (1000 / 4.29), and the 70% alert fires at about 160 packs (700 / 4.29). Test packs made on the production key the same day count too.

Quota reset time. The provider resets the daily quota at midnight Pacific time (usage.pacific_day_start_utc). Conversion rule:

• US standard time (first Sunday of November to second Sunday of March): reset at 08:00 UTC.
• US daylight time (the rest of the year): reset at 07:00 UTC.
• Local reset time = that UTC hour + your UTC offset. Example: at UTC+5 the reset is at 13:00 in winter and 12:00 in summer.

Write the local reset time for your event date on the station cheat sheet.

One week before

#	Check	Command or screen	Expected	Owner
1	Server exists and is deployed	curl -s https://stickers.eventtalks.uz/health	"ok":true, version is the intended commit	Engineer
2	Production profile	Compare server bot/.env with Configuration	APP_ENV=prod, STICKER_RENDER_MODE=figure, WORKER_SLOTS=2, ALLOW_MULTIPLE_PACKS=0, prices set	Engineer
3	AI billing	AI provider console → Billing	Prepaid balance per the estimate above; auto top-up on if available; spend cap not below the estimate	Owner
4	Quota tier	AI provider console → rate limits	Daily image limit matches IMAGE_RPD_LIMIT; enough for packs × 4.3	Owner
5	Fallback provider decision	grep -c '^OPENAI_API_KEY=.\+' /opt/ai-sticker-pack/bot/.env	1 if a fallback key is set. 0 means a long outage of the primary provider fails packs; accept that knowingly.	Owner
6	Branding assets received	Logos, theme choice, outfit previews	Files in hand, see Venue and branding	Client/venue, Owner
7	Logos in the kiosk footer	Commit the logo files into bot/kiosk/static/ under the file names listed in KIOSK_LOGOS in bot/api.py, then push.ps1 -Restart	curl -s https://stickers.eventtalks.uz/branding lists them. A missing file just leaves the slot empty.	Engineer
8	Theme	KIOSK_THEME=dark or light in bot/.env, update.sh --yes	Tablet shows the chosen theme. /kiosk/light always gives the light one; ?theme=light or ?theme=dark overrides.	Engineer
9	Screen fit	Open https://stickers.eventtalks.uz/kiosk/?preview=<W>x<H> on a laptop (CSS pixels of the tablet)	Layout fits; opening the page creates no session	Engineer
10	Alerts reach a human	Send /stats to the bot from an account in ADMIN_USER_IDS	The bot answers. Alerts themselves arrive only on failures.	Engineer
11	Venue basics agreed	Written confirmation	Power socket at the station, venue Wi-Fi details, permission for a mobile hotspot as backup	Client/venue
12	Landing form	Submit a test request on pack.eventtalks.uz	A card with status buttons in the leads group	Engineer
13	Neighbouring project	Tell the owner of the neighbouring project on the same droplet the event date	No deploys or Caddy restarts on their side during the event window	Owner

Code freeze

Set a freeze date before the rehearsal. From that date:

• allowed on the server: flags in bot/.env followed by update.sh, and rollbacks;
• a code push only for an incident, and only with a test pack afterwards;
• note the frozen commit: cat /opt/ai-sticker-pack/VERSION.

Owner: Engineer. The frozen commit is also the rollback target, see Deploy and rollback.

Rehearsal

Run it with the real tablet, ideally on the venue network. All commands are on the server (ssh root@<server>).

#	Check	Command or screen	Expected	Owner
1	Version	curl -s https://stickers.eventtalks.uz/health	"ok":true, version = frozen commit	Engineer
2	Services	systemctl is-active sticker-bot sticker-api caddy	three lines active	Engineer
3	Caddy	caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile; ls /etc/caddy/sites/	Valid configuration; stickers.caddy is listed	Engineer
4	Limits	systemctl show sticker-bot -p MemoryHigh -p MemoryMax -p CPUWeight	Values of 1.45G, 1.6G in bytes and 50	Engineer
5	Swap and memory	swapon --show; free -m	/swapfile 2G; available memory comfortably above the 1420 MB pack peak with an empty queue	Engineer
6	Disk	df -h /	Under half used	Engineer
7	EVENT_MODE	grep '^EVENT_MODE=' /opt/ai-sticker-pack/bot/.env	EVENT_MODE=1. Set it with setenv EVENT_MODE 1; no restart needed. Do the same in local .env files that hold the production key.	Engineer
8	Money	AI provider console → Billing; then admin panel → Баланс (оценка) (“balance, estimate”) → поправить по биллингу (“correct from billing”)	Balance per the estimate; the panel shows the entered value. The panel cannot see the real balance, only an estimate of spend.	Owner
9	Quota	Admin panel → card Картинок за сутки Google (“images per provider day”)	Far from the limit	Engineer
10	Tablet setup	See Tablet setup	Kiosk mode, camera allowed, charging	Operator
11	Test pack	Full guest path from the tablet: Start → QR → bot → Согласен (“I agree”) → outfit → photo	Pack link arrives in about 1–1.3 minutes	Operator
12	Second test pack on the same account	Admin panel → find yourself → Разрешить ещё раз (“allow one more”) → repeat the path	Second pack arrives	Operator
13	Staff reset gesture	Hold the top-left corner of the kiosk screen for 3 seconds	Kiosk returns to the start screen	Operator
14	Network loss drill	Switch the tablet from Wi-Fi to the hotspot mid-session	Kiosk shows нет связи с сервером (“no connection to the server”) and recovers	Operator
15	Local bot stopped	On every laptop that has the token: no running python bot\main.py	Journal has no TelegramConflictError: journalctl -u sticker-bot --since -30min | grep -c Conflict → 0	Engineer
16	Database backup	bash /opt/ai-sticker-pack/deploy/backup.sh	Line OK app-<time>.db.gz … integrity=ok in /var/log/sticker-backup.log	Engineer
17	Snapshot	DigitalOcean → Droplets → the droplet → Snapshots → Take snapshot	Snapshot listed, dated after the final configuration	Owner
18	Rollback drill (optional)	setenv CONSISTENCY_MODE 0, update.sh --yes, then back to 1	Both restarts end with a healthy /health	Engineer

One day before

#	Check	Command or screen	Expected	Owner
1	Nothing changed since rehearsal	cat /opt/ai-sticker-pack/VERSION	Frozen commit	Engineer
2	Maintenance state is intended	curl -s https://stickers.eventtalks.uz/health	"maintenance": null, or a deliberate value you will lift in the morning	Engineer
3	Money again	AI provider console → Billing	Still at the estimate after rehearsal spend	Owner
4	Quota reset time known	Apply the conversion rule above for the event date	Local reset time written on the cheat sheet	Engineer
5	Backup timer	systemctl list-timers sticker-backup.timer	Next run listed (03:30 UTC)	Engineer
6	Firewall	ufw status	Only 22, 80, 443	Engineer
7	Hardware packed	Tablet, charger, stand, power strip, phone for the hotspot, printed cheat sheet	All present	Operator
8	Contacts	Phone numbers of Engineer on call and Owner on the cheat sheet	Reachable for the whole event	Owner
9	Runbooks at hand	Runbooks open on the engineer’s phone and laptop	Loads without the venue network	Engineer

Tablet setup

Item	How	Expected
URL	https://stickers.eventtalks.uz/kiosk/ (HTTPS is mandatory for the camera)	Start screen
Kiosk mode	Browser in full-screen kiosk mode or the OS screen-pinning feature, so guests cannot leave the page	No address bar, no way out by swipe
Camera permission	Allow the camera for the site permanently during the first session	No permission prompt for the next guest
Camera fallback	If the browser asks again for every guest or the camera opens slowly, use https://stickers.eventtalks.uz/kiosk/?camera=always	Camera stays open from page load
Screen sleep	Disable auto-lock. The page also requests a screen wake lock.	Screen stays on
Power	On the charger for the whole event	Battery not draining
Network	Venue Wi-Fi plus a mobile hotspot saved as a second network	Tablet reconnects on its own
Sound and notifications	Do-not-disturb on	Nothing covers the page

Timeouts built into the kiosk: an unscanned QR code expires after 90 s; a guest who scanned but stalled in the bot is dropped after 3 minutes; the outfit screen asks Ты здесь? (“are you here?”) after KIOSK_IDLE_SEC (90 s) and resets after KIOSK_IDLE_PROMPT_SEC (15 s).

Event day

Morning, before guests

#	Check	Command or screen	Expected	Owner
1	Health and maintenance	curl -s https://stickers.eventtalks.uz/health	"ok":true, "maintenance": null, "active_jobs":0	Engineer
2	Services	systemctl is-active sticker-bot sticker-api caddy	three lines active	Engineer
3	Memory and disk	free -m; df -h /	As at rehearsal	Engineer
4	Queue empty	sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT status, COUNT(*) FROM jobs GROUP BY status;"	No queued, processing or delivering rows	Engineer
5	Night backup	tail -1 /var/log/sticker-backup.log	OK … integrity=ok from last night	Engineer
6	Clean logs	journalctl -u sticker-bot --since -1h | grep -E 'ERROR|Traceback'	Empty	Engineer
7	Money and quota	AI provider console; admin panel cards	Balance per estimate; image counter low	Owner
8	Test pack from the venue	Full guest path on the real tablet and network (grant yourself one more pack first)	Link arrives	Operator
9	Tablet	Kiosk mode, camera allowed, charging, Wi-Fi up, hotspot ready	Ready	Operator
10	Local bot stopped	No local python bot\main.py anywhere	No conflict lines in the journal	Engineer
11	Alert chat	Notifications on for the chat behind ADMIN_CHAT_ID	A human sees alerts within a minute	Engineer

During the event

Engineer on call keeps two things open:

• the alert chat;

• a live log:

  journalctl -u sticker-bot -f | grep -E 'STAGE_SUMMARY|RATE_LIMIT|FAILED|JOB_COST|BILLING|ERROR'

Operator on site keeps the admin panel open on a phone: https://stickers.eventtalks.uz/admin.

Situation	Action	Owner
Guest says the pack did not arrive	Pack not delivered	Operator
Tablet frozen, camera or network problem	Station stuck	Operator
Alert Job N FAILED	Generation failing	Engineer
Alert 💳 Кончились деньги на AI-ключе	Billing 402	Owner tops up, Engineer resumes
Alert about the daily image counter, or many RATE_LIMIT lines	Rate limit 429	Engineer
Bot does not answer	Bot silent	Engineer
Kiosk page does not open	Kiosk 502	Engineer
A guest must get a second pack (bad photo, VIP)	Admin panel → the guest → Разрешить ещё раз	Operator
Planned pause (break, moving the station)	/pause in the bot, later /resume	Engineer

Check every hour (Engineer): admin panel cards Очередь сейчас (“queue now”), Ошибки за 24 ч (“errors in 24 h”), Картинок за сутки Google, and free -m on the server.

Closing

#	Check	Command or screen	Expected	Owner
1	Queue drained	curl -s https://stickers.eventtalks.uz/health	"active_jobs":0	Engineer
2	Failed jobs handled	Admin panel → status filter Упал (“failed”)	Each one retried with Повторить джоб (“retry job”) or consciously left	Operator
3	Station closed to guests	/pause in the bot if the tablet stays on, or simply power the tablet off	Bot answers guests with the maintenance text	Engineer
4	Quick numbers for the client	/stats and /admin_stats in the bot	Totals noted	Owner

After the event

#	Check	Command or screen	Expected	Owner
1	Statistics	/stats in the bot; admin panel; sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT status, COUNT(*) FROM jobs WHERE created_at >= '<YYYY-MM-DD>' GROUP BY status;"	Counts of done and failed	Engineer
2	Stage timings	grep -h '^<YYYY-MM-DD>.*STAGE_SUMMARY' /opt/ai-sticker-pack/logs/app.log*	One line per job attempt	Engineer
3	Cost report (on the workstation; the out folder must exist)	.venv\Scripts\python.exe tools\cost_report.py --ssh root@<server> --days 2 --csv out\event_cost.csv	Table per Pacific-time day; compare with the provider’s billing	Owner
4	Database backup	bash /opt/ai-sticker-pack/deploy/backup.sh	OK … integrity=ok	Engineer
5	EVENT_MODE off	setenv EVENT_MODE 0 on the server and in local .env files	grep '^EVENT_MODE=' /opt/ai-sticker-pack/bot/.env → EVENT_MODE=0	Engineer
6	Flags back	Undo any rollback flag set during the event, update.sh --yes	STARTUP lines show the normal profile	Engineer
7	Maintenance decision	Leave /pause on, or set MAINTENANCE_MODE=1, if the station must not serve anyone until the next event	/health shows the intended value	Owner
8	Photo TTL, first check	find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f -mmin +4320 | wc -l	0 (no file older than 72 hours)	Engineer
9	Photo TTL, final check, 72 hours after the last guest	find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f | wc -l; grep -h "storage cleanup" /opt/ai-sticker-pack/logs/app.log* | tail -3	0 files; cleanup lines present. The bot service must stay running for these 72 hours: cleanup runs inside it.	Engineer
10	Data handling	Follow Backups and data lifecycle	Leads exported if needed; old database copies removed	Owner
11	Money	AI provider console: turn auto top-up off if the next event is far away	No surprise charges	Owner
12	Notes	What went wrong, what to change next time	Added to Known gaps or the roadmap	Engineer

Where to go next

• Runbooks — every “situation” row above in detail.
• Deploy and rollback — the procedures behind freeze, snapshot and rollback.
• Monitoring and alerts — what each alert and admin card means.
• Capacity — where the planning numbers come from and how they scale.
• Venue and branding — what to ask the client and the venue for.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.