Reliability
Who is this for
Section titled “Who is this for”Engineers who need to predict what happens when something breaks, and operators who want to know which failures heal by themselves.
- State lives in SQLite, not in memory. Queued jobs, delivery deadlines, unpublished lead cards and the maintenance pause all survive a restart.
- Every status change is a conditional
UPDATE(WHERE status = …) or aBEGIN IMMEDIATEtransaction, so two actors cannot both win. - Retries exist at four levels: provider call (429 backoff), sheet, single figure, and job. Each level has a hard cap.
- A 402 from the AI provider pauses the whole guest flow automatically until an admin sends
/resume. - The leads module and the guest flow share one bot process but cannot break each other through normal errors.
- Single points of failure: one server, one bot process, one SQLite file, one AI key when no fallback key is set, and Telegram itself.
Outbox pattern
Section titled “Outbox pattern”Two places use “write the intent to the database first, let a loop deliver it”.
| Outbox | Written by | Delivered by | Marker of “pending” |
|---|---|---|---|
| Lead cards | API process, POST /leads |
Bot process, leads_bot.publish_due, every 2 s |
leads.published_at IS NULL and next_publish_at reached |
| Repeat notifications | API process | Bot process, leads_bot.notify_repeats |
lead_repeats.notified_at IS NULL for a published lead |
| Generation jobs | API process, photo upload | Bot process, worker slots, every 1 s | jobs.status = 'queued' |
| Pack link | Worker (delivering) |
Delivery task, restarted by the sweeper | jobs.status = 'delivering' and link_sent_at IS NULL |
Consequence: the API can accept photos and leads while the bot is down. They are processed when the bot starts. The 15-minute delivery deadline still applies to jobs, so a guest whose job waited through a long outage gets the failure text rather than silence.
Retry and backoff tables
Section titled “Retry and backoff tables”Generation
Section titled “Generation”| What | Attempts | Pause | Then |
|---|---|---|---|
| Job | 2 (MAX_ATTEMPTS) |
10 s before requeue | failed, admin alert, guest message |
| Figure sheet | 3 (FIGURE_SHEET_ATTEMPTS) |
none | Composite fallback if enabled, otherwise the job attempt fails |
| Head sheet (composite) | 2 (SHEET_ATTEMPTS) |
none | Best cells from both attempts are used |
| Appearance passport | 2 (CARD_ATTEMPTS) |
none | Generation continues without a passport |
| Anchor portrait | 1 retry, only on a verified mismatch | none | First anchor is used; if no anchor at all, the legacy path runs |
| Single-figure regeneration | 1 per figure, at most 4 per pack | time budget 20 s or 35 s | Original sticker stays |
| Provider call on 429 | 3 retries | 5 s, 15 s, 30 s (or the provider’s hint, up to 60 s), plus up to 20 % jitter | Next provider in the chain |
| Provider call on a daily-quota 429 | 0 | — | Next provider at once |
| Provider call on 402 | 0 | — | Next provider; if none succeeds, the job stops and the pause switches on |
| Provider chain | Primary, then fallback | — | GenerationError → job retry |
Sources: worker.py,
services/ai_pipeline.py.
Delivery and messages
Section titled “Delivery and messages”| What | Attempts | Pause |
|---|---|---|
| Pack readiness poll | Until the 15-minute deadline | 2 s for the first 90 s after pack creation, then 10 s |
| Message to the guest from the delivery module | 3 | 2 s |
| “Generating…” message from the API | 1 | — |
| Admin alert | 1; failure is only logged | Repeating alerts with the same key are suppressed for 10 minutes and counted |
Lead card publishing
Section titled “Lead card publishing”| Consecutive failures for one lead | Next attempt after |
|---|---|
| 1 | 5 s |
| 2 | 15 s |
| 3 | 60 s |
| 4 | 5 min |
| 5 or more | 15 min; admin alert at most every 30 minutes |
| Telegram flood control | The retry_after Telegram returns; the rest of the batch waits |
Repeat notifications follow the same delays and stop after 5 failures. Source:
leads_bot.py (BACKOFF_SEC,
LATER_SEC, ALERT_AFTER).
Landing
Section titled “Landing”| What | Timeout | On failure |
|---|---|---|
POST /leads |
5 s | Fallback to direct Telegram message |
Fallback sendMessage |
10 s | Visitor sees an error; the form keeps its content |
| Visitor retry | Same requestId |
Server answers 200 with the same lead number |
| What | Attempts | Pause |
|---|---|---|
| Outfit list load | Until it succeeds | 1 s doubling up to 10 s |
| Photo upload | 3 | 1 s, 2 s, 3 s |
| Full capture | 2 per session | — |
| Status poll | Forever while a session exists | 1 s; a network error only shows a badge |
Auto-pause on 402
Section titled “Auto-pause on 402”When no provider can serve a request and at least one answered “no money” (HTTP 402 or a billing marker), the worker:
- fails the job without a retry (
BILLING_EXHAUSTEDin the log); - writes a
maintenancerow to thesettingstable (maintenance.pause, sourceauto); - sends one admin alert (cooldown key
billing_exhausted); - tells the guest that generation is temporarily unavailable and that the photo is saved.
Because the pause is a database row, both processes see it without a restart. Before this rule existed a 402 was
treated as a rate limit; the load-test report in the product repository records that each guest then waited about 3
to 4 minutes before failing (docs/LOAD_TEST_REPORT.md).
Maintenance mode sources
Section titled “Maintenance mode sources”| Source | How it is switched on | How it is switched off |
|---|---|---|
env |
MAINTENANCE_MODE=1 in bot/.env, then restart |
Set it to 0, then restart. /resume cannot clear it |
auto |
Automatically on a 402 | /resume in the bot, from a user in ADMIN_USER_IDS |
admin |
/pause in the bot |
/resume |
While it is on (services/maintenance.py):
- the bot answers every guest message and button with the maintenance text; admin commands other than
/startstill work; POST /session, outfit choice and photo upload answer503with codemaintenance;- worker slots do not claim new jobs; jobs already running finish;
- queued jobs wait, and their 5- and 15-minute deadlines keep running;
- the leads group is unaffected.
If the database cannot be read, the middleware lets the guest through: normal service is preferred over a silent bot.
Graceful shutdown and requeue
Section titled “Graceful shutdown and requeue”main.shutdown in main.py runs on
SIGINT or SIGTERM, or when any background task ends.
| Step | Limit | Effect |
|---|---|---|
| 1. Stop polling | 5 s (POLLING_STOP_TIMEOUT_SEC) |
No new updates, so no new guests |
| 2. Let running jobs finish | 15 s (SHUTDOWN_GRACE_SEC) |
Slots take no new jobs. After the grace period running jobs are cancelled |
| 3. Requeue cancelled jobs | — | processing → queued, note [requeued on shutdown]. The first 2 such requeues of a job are free (SHUTDOWN_FREE_REQUEUES); later ones consume an attempt, so repeated restarts cannot loop a paid generation forever |
| 4. Cancel loops | — | Sweeper, cleanup, leads loop. Their state is in the database |
| 5. Cancel delivery checks | — | Jobs stay delivering; the sweeper resumes them after start |
| 6. Close the bot session | — | Last, so finishing jobs can still message the guest |
| 7. Exit | 3 s for worker threads (EXECUTOR_EXIT_WAIT_SEC) |
AI calls run in daemon threads and are abandoned |
systemd sends SIGINT and waits 30 s (TimeoutStopSec) before killing the process, which is above the sum of the
limits. The behaviour is covered by
tests/test_shutdown.py.
After a hard kill (out-of-memory, SIGKILL) the next start runs two queries before the slots begin:
db.fail_stale_over_limit: aprocessingjob whose attempts would now reach the cap becomesfailed, with an alert and the failure text for the guest;db.requeue_stale_processing: otherprocessingjobs return toqueuedwith one attempt consumed.
Every AI call is written to api_usage as it happens, so the cost ceiling still sees the spend of an attempt that
was killed.
Isolation between the guest flow and leads
Section titled “Isolation between the guest flow and leads”| Risk | Guard |
|---|---|
| A leads error stops the bot | In main.serve the end of any task stops everything. The leads loop therefore catches every exception per iteration and per housekeeping step and never returns |
| A group message triggers the guest flow | Guest, admin and maintenance routers filter on private chats |
| A guest button is taken by the leads router | The leads router filters on the leads chat id and on the stickers_ callback prefixes |
| The maintenance pause blocks lead handling | The maintenance middleware passes every non-private event |
| Leads are switched off but old cards still have buttons | A fallback router answers orphan buttons |
| The whole module misbehaves | LEADS_ENABLED=0 removes the router, the loop and the endpoint |
The wiring is tested in
tests/test_leads_wiring.py.
Deadlines persisted in the database
Section titled “Deadlines persisted in the database”The 5-minute notice and the 15-minute failure are computed from jobs.created_at and guarded by
delay_notice_sent_at and link_sent_at. They do not depend on timers in memory. A bot that was down for 20
minutes fails overdue jobs on its first sweep and does not send the “still working” notice to them first. Details:
Pack delivery.
The daily digest uses the same idea: the last sent day is a settings row, so a restart cannot send it twice.
Single-instance lock
Section titled “Single-instance lock”instance_lock.py takes an
exclusive OS lock on data/main.lock at start. A second main.py on the same machine logs
STARTUP отменён (“start cancelled”) and exits with code 1. The lock is released by the operating system when the
process ends for any reason, so a crash cannot leave it stuck. Two bots would otherwise compete for Telegram updates
and each worker would render jobs with its own configuration.
SQLite under two processes
Section titled “SQLite under two processes”Database in db/db.py opens the file
in WAL mode with a 5-second busy timeout. Loops that hit database is locked log the error and continue on the next
tick. Schema migration takes a backup, rehearses on a copy, and applies inside BEGIN IMMEDIATE; the two processes
may start in any order.
Single points of failure
Section titled “Single points of failure”| Component | If it fails | Mitigation that exists | What is missing |
|---|---|---|---|
| The server | Kiosk, bot, admin panel and leads API are all down. The landing still accepts requests through its fallback | Daily database backup; optional provider snapshot; Restart=always on both units |
No second server, no automatic failover |
| Bot process | No updates, no generation, no delivery, no lead cards | systemd restarts it after 3 s; state is in the database | Single instance by design |
| API process | Kiosk shows a connection error; the landing falls back | systemd restart | Single uvicorn worker (the in-memory limiters depend on that) |
| SQLite file | Everything stops | WAL, integrity-checked daily backups | Backups are on the same server |
| AI provider | No packs | Fallback provider if a second key is set; backoff on 429 | With one key there is no fallback at all |
| AI account balance | No packs | Auto-pause, alert, retry button | Top-up is manual |
| Telegram | No bot, no packs, no lead cards | Landing fallback also depends on Telegram | None |
| Venue network or the tablet | That station stops | Upload retries; staff reset gesture | One tablet per station |
| Shared server neighbour | Memory or CPU pressure | MemoryHigh / MemoryMax / CPUWeight on the units; 2 GB swap |
Same kernel, same disk |
Where to go next
Section titled “Where to go next”- Observability — how each of these failures shows up in logs and alerts.
- Generation and Pack delivery — the flows these rules protect.
- State machines — every allowed transition.
- Runbooks — what to do when a failure does not heal by itself.
- Capacity — where the single server runs out.
Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.