Observability

Who is this for

Operators who watch an event live, and developers who need to find one job, one lead or one failure in the logs.

TL;DR

There is no metrics stack. Observability is: structured log markers you can grep, a public /health, the admin panel summary, and alerts sent to a Telegram chat.
Bot logs go to logs/app.log, API logs to logs/api.log, both also to journald. Files rotate at 5 MB inside the application and daily through logrotate, kept 14 days.
Each job attempt ends with exactly one STAGE_SUMMARY line that holds the time of every stage.
Every AI provider call, including failures, becomes a row in the api_usage table with tokens and an estimated cost.
Log markers are English upper-case tokens; the rest of each line is Russian. Grep for the marker.

Log files and rotation

Source	File	Also in	In-app rotation	System rotation
Bot, worker, delivery, cleanup, leads loop	`logs/app.log`	`journalctl -u sticker-bot`	5 MB × 3 backups (`RotatingFileHandler`)	logrotate: daily, 14 files, `copytruncate`, compressed
Kiosk API, admin panel, leads endpoint	`logs/api.log`	`journalctl -u sticker-api` (uvicorn’s own access lines appear only here)	5 MB × 3 backups	Same logrotate rule
HTTPS front	`/var/log/caddy/stickers.log`	`journalctl -u caddy`	20 MiB × 14 files (Caddy)	—
Database backup	`/var/log/sticker-backup.log`	journald	—	—
Landing server action	Console lines `[request] <request_id> → <route>`	Hosting platform logs	—	Platform policy
Kiosk page	Browser console, prefix `[kiosk]`	—	—	—

Format of the Python logs: %(asctime)s %(levelname)s %(name)s: %(message)s. Timestamps are in the server’s local time zone. Sources: main.py setup_logging, api.py _attach_file_log, deploy/logrotate/sticker-pack.

Log markers

Process lifecycle

Marker	Meaning
`STARTUP`	One line per fact at start: render mode and where it was read from, likeness and consistency flags, allowed update types, kiosk and health URLs with the code version. `STARTUP отменён` means another bot process holds the instance lock
`SHUTDOWN`	Signal received, a task ended unexpectedly, jobs interrupted and requeued, total stop time
`MAINTENANCE`	A guest request or an API call was answered with the maintenance text
`MAINTENANCE_ON` / `MAINTENANCE_OFF`	The database pause was written (with `source` and `reason`) or removed
`LEADS:`	Leads module start-up facts: target chat, number of admins, whether the bot is a group admin, command menu status, loop errors

Session and kiosk (API log)

Marker	Meaning
`kiosk: новая сессия`	Session created
`kiosk: сессия … брошена планшетом (<reason>)`	Session expired by the tablet, with the reason code
`kiosk: сессия … -> job … в очереди`	Photo accepted, job queued
`PACK_LIMIT`	A guest hit the pack limit; state is `ready` or `in_progress`
`JOB_RENDER_MODE`	Render mode seen by the API at creation, and by the worker at processing. The worker’s line is the one that counts
`ADMIN_LOGIN_FAILED` / `ADMIN_LOGIN_BLOCKED`	Wrong admin credentials from an IP; the IP was locked out
`ADMIN <user> (<ip>): <action>`	An admin action: `grant_extra_pack`, `reset_user`, `retry_job`

Generation (bot log)

Marker	Meaning
`STAGE_TIMING`	One line per stage: `job`, `stage`, `sec`, and for AI stages `calls`, `r429`, `wait429`, `in_bytes`, `provider`
`STAGE_SUMMARY`	One line per job attempt: `total`, per-stage seconds, `retries`, `r429`, optional `slot_wait`, `result`, `attempt`
`APPEARANCE_CARD` / `APPEARANCE_RETRY` / `APPEARANCE_FAILED`	Passport built (or taken from cache), a retry, or given up
`FACE_CROP`, `FACE_CROP_SKIPPED`, `FACE_BOX_MISSING`	Face crop made, skipped, or the passport had no face box
`ART_STYLE`, `SHEET_LAYOUT`, `LIKENESS_MODE`, `HARD_LOCKS`, `TEMPLATE_DEFAULT`	Which style, layout, likeness mode and lock text a job used; a job without an outfit fell back to the default
`ANCHOR`, `ANCHOR_VERIFY`, `ANCHOR_SKIN`, `CONSISTENCY_ANCHOR_FAILED`	Anchor portrait generated or cached, verification result, skin correction, anchor failure (legacy path used)
`FIGURE_CUT`, `FIGURE_SPLIT`, `FIGURE_BOTTOM_CUT`, `SHEET_MISSING`	Result of cutting a sheet, neighbours that had to be split, close-ups touching the bottom edge, empty cells
`EDGE_TORSO_OK` / `EDGE_TORSO_NO`, `EDGE_PADDED`, `EDGE_CUT_LEFT`	Edge contact accepted, rejected, hidden by padding, or left visible
`FLAT_CUT`, `FLAT_CUT_REGEN`, `FLAT_CUT_LEFT`	A straight vertical cut on an arm was found, the regenerated figure had it too, or it was left as is
`SKIN_NORM`	Skin colour distance before and after normalisation, per sticker
`QA_IDENTITY_RESULT`, `QA_IDENTITY_MISMATCH`, `QA_IDENTITY недоступен`	Per-sheet identity check result, figures that still failed, QA unavailable
`CONSISTENCY_QA`, `CONSISTENCY_REGEN`, `CONSISTENCY_REGEN_TIMEOUT`, `CONSISTENCY_REGEN_FAILED`, `CONSISTENCY_REGEN_REJECTED`	Pack-level QA outliers and what happened to each regeneration
`LIKENESS_TARGET`, `LIKENESS_POST`, `LIKENESS_QA`, `LIKENESS_REGEN`, `LIKENESS_REGEN_REJECTED`	The same for classic and hybrid modes
`FIGURE_FALLBACK_TO_COMPOSITE`	No valid figure sheet; the job is being rebuilt as composite
`RATE_LIMIT`	A 429; the line states the wait and the retry number
`IMAGES_SLOT_WAIT`, `CUT_SLOT_WAIT`	A call waited more than 1 s for a parallelism slot
`BILLING_EXHAUSTED`	A provider answered “no money”; job failed, pause switched on
`JOB_COST_LIMIT`	The job reached `JOB_MAX_COST_USD`
`IMAGE_RPD_WARN`	Image requests today passed the warning share of the daily quota
`Провайдер … упал`	A provider failed for one call; the chain moved on

Delivery

Marker	Meaning
`Пак … создан`	`createNewStickerSet` succeeded
`STICKERSET_CONFIRMED`	Both checks passed and the link was sent; the line lists poll counts and seconds
`DELAY_NOTICE`	5 minutes without a link; the guest got the “still working” text
`STICKERSET_DELIVERY_FAILED`	15 minutes without a link; job failed
`storage cleanup:`	Number of files removed by the TTL pass

Leads

Marker	Meaning
`LEAD_NEW`, `LEAD_DUPLICATE`, `LEAD_REPEAT`	Outcome of `POST /leads`, with lead number, `request_id`, language
`LEAD_REJECTED`, `LEAD_INVALID`, `LEAD_RATE_LIMITED`	Wrong secret, failed validation (field names only), a limit hit
`LEAD_PUBLISHED`, `LEAD_PUBLISH_FAILED`, `LEAD_PUBLISH_RETRY`	Card posted, a failed attempt, Telegram asked to wait
`LEAD_REPEAT_NOTIFIED`, `LEAD_REPEAT_NOTIFY_FAILED`, `LEAD_REPEAT_CARD`	Repeat ping sent, failed, or the card could not be rewritten
`LEAD_STATUS`, `LEAD_STATUS_SAME`, `LEAD_STATUS_DENIED`, `LEAD_STATUS_BAD_BUTTON`, `LEAD_CARD_EDIT_FAILED`	Status button outcomes
`LEAD_PICK`, `LEAD_PICK_DENIED`, `LEAD_PICK_BAD_BUTTON`, `LEAD_BUTTON_ORPHAN`	Picker buttons; a button nobody handled
`LEADS_CMD`	A `/stickers_*` command and who ran it
`LEAD_REMIND`, `LEAD_DIGEST`	Reminder sent (or not sent), digest sent or skipped
`LEAD_ERASED`, `LEAD_ERASE`, `LEAD_DELETED_TEST`	Anonymised by request or by retention, card could not be rewritten, test lead deleted

Useful one-liners

grep STAGE_SUMMARY logs/app.log                       # one line per job attempt
grep -E "job[ =]<job_id>\b" logs/app.log              # everything about one job
grep -E "RATE_LIMIT|BILLING_EXHAUSTED|JOB_COST_LIMIT" logs/app.log
grep -E "STICKERSET_|DELAY_NOTICE" logs/app.log
grep -E "LEAD_|LEADS" logs/app.log logs/api.log

Reading `STAGE_SUMMARY`

The sample below is the illustrative line from the module’s own docstring, extended with the newer keys. It is not a measurement.

STAGE_SUMMARY job=<id> total=83.1 queue=1.2 passport=6.0 anchor=- gen1=18.2 cut1=3.4 qa1=5.3 gen2=18.0 cut2=3.8
              qa2=4.1 gen3=- cut3=- skin=- stickers=5.6 cqa=- regen=- pack=3.1 deliver=2.5 retries=0 r429=0
              result=delivered attempt=1

Field	Meaning
`total`	Seconds from `jobs.created_at` (photo accepted) to this line
`queue`	Seconds from creation until a slot took the job. On a second attempt it includes the first attempt
`passport`, `anchor`, `gen1..3`, `cut1..3`, `qa1..2`, `skin`, `stickers`, `cqa`, `regen`, `pack`, `deliver`	Seconds per stage; `-` means the stage did not run. Parallel sheets make the sum larger than `total`
`retries`	Sheet regenerations inside this attempt
`r429`	Number of 429 answers
`slot_wait`	Seconds spent waiting for an image slot, shown only when not zero
`result`	`delivered`, `already_sent`, `delivery_failed`, `stopped_<status>`, `retry`, `failed`, `expired`, `shutdown_requeued`

These lines carry no names, user ids or file paths (services/stage_timing.py).

`/health`

GET https://stickers.eventtalks.uz/health is public and unauthenticated.

Field	Value
`ok`	`true`, or `false` with HTTP `503` when the database query failed
`db`	`ok` or `error`
`active_jobs`	Number of jobs in `queued`, `processing`, `delivering`
`maintenance`	`null`, or the source of the pause: `env`, `auto`, `admin`
`version`	Content of the `VERSION` file written at deploy (short commit hash); `unknown` if absent
`render_mode`	`STICKER_RENDER_MODE` as the API process sees it
`app_env`	`prod` or `dev`

/health proves that the API process and the database answer. It says nothing about the bot process. To check the bot, look at systemctl status sticker-bot, at fresh lines in logs/app.log, or send /stats to the bot.

Admin panel and bot commands

GET /admin/api/summary (and /admin_stats in the bot, which prints a short text version of the same data) returns, per services/admin_data.py:

Group	Fields
Maintenance	Source, reason and start time of the pause, or nothing
Spend	Estimated total spend, spend today (local day by `DISPLAY_UTC_OFFSET_HOURS`), the per-job ceiling, list of price variables that are not set
Balance	Estimated remaining balance: a manually entered billing figure minus spend since it was entered, or `BUDGET_USD` minus all spend. Always labelled as an estimate
Daily image quota	Image requests in the current Pacific-time day against `IMAGE_RPD_LIMIT`, with a warning flag
Packs	Ready, in progress, failed; queue split into `queued`, `processing`, `delivering`; average seconds from photo to link
Errors	In the last 24 hours: count of 429 answers and count of other failed AI calls
Build	Render mode and code version

The panel also lists guests and jobs with status, duration, attempts, render mode, error text and cost per job (/admin/api/guests), cost per stage for one job (/admin/api/jobs/{id}/cost), and the last 100 admin actions (/admin/api/audit).

/stats in the bot (admins only) prints counts of users, sessions by status and jobs by status, the average and maximum time from photo to pack, and the age of the oldest queued job. A growing “oldest queued” value means the queue is stuck.

Alerts overview

Alerts are Telegram messages to ADMIN_CHAT_ID, sent by services/alerts.py. Each starts with ⚠️ [<EVENT_NAME>] (or ℹ️ for information). Without ADMIN_CHAT_ID they are only logged; in production the bot refuses to start without it.

Alert	Trigger	Throttle
Job failed	A job ran out of attempts or failed permanently	None
Job failed after process kills	Found at start-up	None
Provider fallback	The primary provider failed and the fallback produced the sheet	10 minutes per provider pair; suppressed repeats are counted
Figure fallback (information)	A figure job was rebuilt as composite	None
Billing exhausted	First 402 that switches the pause on	Once per pause; 10-minute key
Daily image quota	Usage passed `IMAGE_RPD_WARN_FRAC`	Once per Pacific-time day
Delay notice	5 minutes without a link	Once per job
Delivery failed	15 minutes without a link	Once per job
Leads flood	More than `LEADS_HOURLY_LIMIT` requests in an hour	1 hour
Lead card not publishing	5 or more failed attempts for one lead	30 minutes

Nothing alerts when a whole process is down: a stopped bot cannot send a message. Thresholds, wording and responses are on Monitoring and alerts.

`api_usage` accounting

services/usage.py records every provider call the moment it returns, successful or not.

Column	Content
`job_id`, `ts`	Which job, when (UTC)
`kind`	`image`, `vision` or `qa`
`stage`	`passport`, `anchor`, `anchor_verify`, `sheet`, `regen`, `qa_pack`, `qa_regen`, `qa_sheet`, `heads`
`provider`, `model`, `resolution`	For example `1K` or `2K` for images
`prompt_tokens`, `output_tokens`, `image_tokens`, `thoughts_tokens`, `images`	As reported by the provider
`ok`, `http_status`, `error`	Failures and 429 answers are stored with `ok = 0`
`est_cost_usd`	Estimate from the `PRICE_*` variables at the time of the call; `0` for failed calls

Rules worth knowing:

Prices come only from bot/.env. A missing price contributes 0 and the panel shows a warning, so the spend is understated rather than invented.
A call cancelled by a time budget or by shutdown is still recorded (as failed), because the request was probably billed.
The figure is an estimate from tokens, not the provider’s invoice. The panel repeats this note next to the balance.
Rows are not deleted when a guest is reset.

`tools/cost_report.py`

tools/cost_report.py is a read-only report over api_usage.

python tools/cost_report.py                      # local database
python tools/cost_report.py --db <path-to-app.db>
python tools/cost_report.py --ssh root@<server>  # reads the server database read-only over SSH
python tools/cost_report.py --days 7 --csv <file>

It prints two Markdown tables: per day (jobs, calls, image requests, 429 count, estimated cost, cost per job) and per day × stage × model × resolution. Days are Pacific-time days, because that is how the image provider counts its daily quota.

What is not observable

No host metrics (CPU, memory, disk) are collected by the product. update.sh prints the units’ memory peak after a restart; otherwise use system tools.
No request tracing and no error aggregation service.
No uptime probe is defined in the repositories. /health is ready to be probed by an external service.
The kiosk does not report its own errors to the server; they stay in the tablet’s browser console.

Where to go next

Monitoring and alerts — what to watch during an event and how to react.
Runbooks — per-symptom procedures that use these markers.
Performance — measured stage timings.
Cost — numbers derived from api_usage.
API reference — /health and the admin endpoints.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.