Generation, QC and provider fallback

Who is this for

Developers who change the generation pipeline, and operators who need to read a failed job and decide what to do.

TL;DR

A worker slot claims the oldest queued job atomically, builds 12 stickers, creates the Telegram pack and hands the job to delivery. The kiosk never waits for this.
There are five pipeline variants. Which one runs is decided by STICKER_RENDER_MODE, LIKENESS_MODE and CONSISTENCY_MODE, read from bot/.env when the bot starts.
Default per code: composite (AI draws heads, code puts them on a body template). Default per bot/.env.example: figure with CONSISTENCY_MODE=1, SHEET_LAYOUT=3x4, LIKENESS_MODE empty.
A job has 2 attempts (MAX_ATTEMPTS). A sheet has 2 or 3 attempts depending on the mode. A 429 is retried on the same provider after 5, 15 and 30 seconds.
A 402 (“no money on the AI key”) stops the job at once, pauses all new jobs (maintenance mode) and alerts the admin chat.
JOB_MAX_COST_USD (default 1.5) is checked before every AI call. Reaching it fails the job without a retry.

Which path runs

flowchart TD
    J["Job claimed"] --> P["Appearance passport (vision call)"]
    P --> M{"STICKER_RENDER_MODE"}
    M -- composite --> C["Composite: 2 sheets of 6 heads"]
    M -- figure --> L{"LIKENESS_MODE"}
    L -- "classic or hybrid" --> CL["Selfie-only path: classic / hybrid"]
    L -- empty --> CM{"CONSISTENCY_MODE"}
    CM -- "1" --> AN["Anchor portrait"]
    AN -- ok --> CO["Consistency path: parallel sheets"]
    AN -- failed --> LE["Legacy figure path: 2 sheets in sequence"]
    CM -- "0" --> LE
    CL -- "no valid sheet" --> FB{"FIGURE_FALLBACK_ENABLED"}
    CO -- "no valid sheet" --> FB
    LE -- "no valid sheet" --> FB
    FB -- "1" --> C
    FB -- "0" --> X["Job attempt fails"]

The decision code is build_stickers and build_figure_stickers in worker.py. The mode a running process uses is printed at start: STARTUP render_mode=… and STARTUP likeness=… consistency_mode=… sheet_layout=…, and per job: JOB_RENDER_MODE job=<id> mode=….

Setting	Default in `config.py`	Value in `bot/.env.example`
`STICKER_RENDER_MODE`	`composite` when the key is absent; an empty value is a start-up error	`figure`
`CONSISTENCY_MODE`	`1`	`1`
`LIKENESS_MODE`	empty	empty
`SHEET_LAYOUT`	`3x4`	`3x4`
`FIGURE_SHEET_ASPECT`	`2:3`	`3:2`
`FIGURE_SHEET_ATTEMPTS`	`3`	`3`
`FIGURE_FALLBACK_ENABLED`	`1`	`1`
`ART_STYLE`	`cute_cartoon`	`cute_cartoon`
`QA_IDENTITY_ENABLED`	`true`	`true`

Sequence (consistency path, the `.env.example` default)

sequenceDiagram
    autonumber
    participant W as Worker
    participant D as SQLite
    participant AI as AI provider
    participant T as Telegram
    W->>D: claim oldest queued job, status processing
    W->>AI: Vision call, appearance passport
    W->>D: cache passport in jobs.appearance_json
    W->>AI: Anchor portrait, 1K, 1 to 1
    opt ANCHOR_VERIFY
        W->>AI: Vision check anchor against selfie, 10 s limit
        W->>AI: One anchor retry when a mismatch is reported
    end
    Note over W: Shift anchor skin toward the selfie estimate
    par each sheet
        W->>AI: Sheet with anchor as image 1, 2K
        Note over W: Cut and chroma key, up to FIGURE_SHEET_ATTEMPTS
    end
    Note over W: Normalise skin tone to anchor, export 512 px
    W->>AI: Pack QA, 12 stickers against anchor
    opt outliers, edge cuts, missing figures
        W->>AI: Single-figure regenerations in parallel, 1K
        W->>AI: QA of the regenerated figures
    end
    W->>T: createNewStickerSet with 12 stickers
    W->>D: status delivering, selfie file deleted

Steps, tied to code

Claim. _worker_loop polls every second (POLL_INTERVAL_SEC). It skips the queue while maintenance mode is on. db.claim_next_job is one conditional UPDATE … RETURNING, so two slots never take the same job.
Cost baseline. process_one_job reads what the job already spent since jobs.created_at and opens a usage collector with the JOB_MAX_COST_USD limit. Every AI call is written to api_usage immediately.
Appearance passport. appearance.build_card asks the vision model for a structured description of the guest (skin tone hex, hair, facial hair, age band, glasses, face box). 2 attempts (CARD_ATTEMPTS). If both fail the job continues without a passport (APPEARANCE_FAILED). The passport is cached in jobs.appearance_json, so a job retry does not pay again.
Face crop (figure modes). appearance.face_crop cuts the face from the selfie using the passport’s face box and stores it in the session folder.
Anchor (consistency path). make_anchor generates a neutral front-facing portrait in the target style and outfit. consistency.verify_anchor compares hair length, hair colour, facial hair and glasses with the selfie (ANCHOR_VERIFY_TIMEOUT_SEC, default 10 s); a mismatch causes exactly one anchor retry. The anchor’s skin is then shifted toward the passport estimate (ANCHOR_SKIN_STRENGTH 0.6, capped by SKIN_NORM_MAX_DE 4). A job retry reuses anchor.png from the session folder.
Sheets. With SHEET_LAYOUT=3x4 there are 3 sheets of 4 figures (2×2 grid, 1:1); with 2x6, 2 sheets of 6 (3×2 grid). In the consistency path all sheets run in parallel with the anchor as the main reference and a HARD LOCKS text block in the prompt.
Cut and chroma key. Each sheet is drawn on bright green (#00FF00). sticker_builder.figures_from_sheet finds the figures, splits touching neighbours and keys out the background. Cutting is limited to CUT_MAX_PARALLEL sheets at once per process.
Sheet validity. A sheet without the expected grid of figures burns one sheet attempt and is regenerated. A valid sheet is taken at once. In the consistency, classic and hybrid paths, a figure touching the sheet edge or a missing figure does not regenerate the sheet; it is queued for single-figure regeneration.
Skin normalisation. Each sticker whose mean skin colour is further than CONSISTENCY_SKIN_DE_MAX (6 ΔE) from the anchor is shifted by CONSISTENCY_SKIN_STRENGTH (0.85), never by more than SKIN_NORM_MAX_DE (4 ΔE).
Export. finish_figure adds the white die-cut outline; export_for_telegram writes 512 px PNG files of at most 512 KB.
Pack QA. consistency.check_pack shows the vision model the anchor and contact sheets of the 12 stickers and asks for outliers (hair length, hair colour, skin tone, face shape, facial hair). If QA is unavailable the pack is accepted as is.
Single-figure regeneration. _regen_outliers regenerates QA outliers, edge cuts, flat side cuts and missing figures, each once, in parallel, at 1K. Limits are in the table below. A regenerated sticker replaces the original only if a second QA pass does not reject it.
Pack and hand-off. The worker re-reads the job status (the delivery sweeper may have failed it on the 15-minute deadline), creates the pack, deletes the selfie file, sets delivering and spawns the delivery check. See Pack delivery.

How the variants differ

Variant	Face source	Sheets	Identity QA	Skin handling
Consistency (`CONSISTENCY_MODE=1`)	Anchor portrait first, selfie second	All in parallel	One pack-level QA against the anchor	Normalised to the anchor
Classic (`LIKENESS_MODE=classic`)	Selfie and face crop only, no anchor	Sheet 1 first, the rest in parallel with sheet 1 as reference	Per sheet against the selfie; failures become single-figure regenerations	Not touched
Hybrid (`LIKENESS_MODE=hybrid`)	As classic, plus a `COLOR LOCK` prompt block	As classic	Pack-level QA against the selfie	Skin and hair moved to targets (median of the 12 figures, pulled toward the selfie by `HYBRID_SELFIE_PULL`), green spill removed
Legacy figure (`CONSISTENCY_MODE=0`, or anchor failed)	Selfie and face crop	2 sheets of 6 in sequence; sheet 2 gets sheet 1 as reference	Per sheet; at most one sheet regeneration for QA or edge defects	Not touched
Composite (`STICKER_RENDER_MODE=composite`, or figure fallback)	Selfie	2 sheets of 6 heads in sequence	Per sheet; `SHEET_ATTEMPTS` = 2	Not touched; head is composed on a body template with Pillow

LIKENESS_MODE takes priority over CONSISTENCY_MODE. Layout, cutting, edge rules, missing-figure handling, single-figure regeneration, cost accounting, the cost ceiling and 402 handling are shared by the three newer figure paths.

Retries, timeouts and limits

Level	Limit	Source
Job attempts	2 (`MAX_ATTEMPTS`), 10 s pause before requeue (`RETRY_DELAY_SEC`)	`worker.py`
Figure sheet attempts	3 (`FIGURE_SHEET_ATTEMPTS`); an invalid sheet burns one	`config.py`
Head sheet attempts (composite)	2 (`SHEET_ATTEMPTS`)	`worker.py`
Passport attempts	2 (`CARD_ATTEMPTS`)	`services/appearance.py`
Anchor retries	1, only on a verified mismatch	`worker.py`
Single-figure regenerations	At most `CONSISTENCY_MAX_REGEN` (4) per pack; missing figures are outside this cap	`worker.py`
Regeneration time budget	Edge or flat cut: `EDGE_REGEN_TIMEOUT_SEC` (35 s). QA outlier: `CONSISTENCY_REGEN_BUDGET_SEC` (20 s). Missing figure: no budget	`worker.py`
Image request timeout	`AI_TIMEOUT_SEC` (180 s), plus a 15 s outer guard	`services/ai_pipeline.py`
Vision request timeout	45 s (`VISION_TIMEOUT_SEC`), plus a 10 s outer guard	`services/ai_pipeline.py`
Parallel image requests	`AI_MAX_PARALLEL_IMAGES` (9) per process, all worker slots together	`services/limits.py`
Delivery deadline	15 minutes from job creation, applies during generation too	`delivery.py`

If a missing figure cannot be drawn, the pack would be incomplete, so the worker raises GenerationError and the job goes to its normal retry. A regeneration that times out or fails for any other figure leaves the original sticker in place and writes a log line (CONSISTENCY_REGEN_TIMEOUT, CONSISTENCY_REGEN_FAILED, EDGE_CUT_LEFT, FLAT_CUT_LEFT).

Provider chain

generate_head_sheet and vision_json in ai_pipeline.py share one rule set.

sequenceDiagram
    participant W as Worker
    participant P1 as Primary provider
    participant P2 as Fallback provider
    participant Adm as Admin chat
    W->>P1: Request
    alt 429 rate limit
        P1-->>W: 429
        Note over W: Wait 5 s, then 15 s, then 30 s, or the hinted delay up to 60 s, plus jitter
        W->>P1: Retry, up to 3 times
    else daily quota or other error
        P1-->>W: Error
    end
    opt primary did not succeed
        W->>P2: Same request
        P2-->>W: Image
        W->>Adm: Fallback alert, at most once per 10 minutes per provider pair
    end
    opt no provider succeeded
        alt any provider answered 402 or a billing marker
            Note over W: BillingExhausted, job failed, auto-pause
        else
            Note over W: GenerationError, job attempt fails
        end
    end

Order: PRIMARY_AI_PROVIDER (default gemini), then FALLBACK_AI_PROVIDER (default openai). A provider without an API key is skipped. With only one key the fallback does not exist; the start-up check warns about it.
Default models: gemini-3.1-flash-image and gpt-image-2 for images, gemini-3.5-flash-lite and gpt-5-mini for vision. GEMINI_IMAGE_MODEL, OPENAI_IMAGE_MODEL, GEMINI_VISION_MODEL and OPENAI_VISION_MODEL override them.
429 backoff (with_rate_limit_backoff): waits RATE_LIMIT_DELAYS = 5, 15, 30 s, or the provider’s own retryDelay hint capped at 60 s, plus up to 20 % jitter. Log marker RATE_LIMIT. A daily-quota error (text contains PerDay) is not waited for.
402 / billing (is_billing_exhausted): HTTP 402, or an error text containing prepayment credits are depleted, payment required, insufficient_quota or billing_hard_limit_reached. No waiting; the chain moves on at once.
JobCostExceeded and BillingExhausted are JobStop exceptions. They never trigger the fallback and never trigger a job retry.

Job-level outcomes

Outcome	What the worker does	Guest message
Success	Deletes the selfie, `processing → delivering`, spawns the delivery check	Link after the pack is verified
Ordinary error, attempt 1	Waits 10 s, `processing → queued`, `attempts = 1`	None
Ordinary error, attempt 2	`failed`, admin alert `Job <id> FAILED`	`Что-то пошло не так с генерацией твоего набора…` (“Something went wrong… we will help manually”)
`PermanentJobError` (photo file missing, unknown outfit id)	`failed` without a retry	Same
`JobCostExceeded`	Log `JOB_COST_LIMIT`, `failed` without a retry	Same
`BillingExhausted`	Log `BILLING_EXHAUSTED`, `failed` without a retry, maintenance pause, admin alert once	`Не получилось сделать твой набор: генерация временно недоступна…` (“Generation temporarily unavailable, your photo is saved”)
Figure sheet never valid	`FIGURE_FALLBACK_TO_COMPOSITE`: same job rebuilt as composite, calm admin notice	None
Bot stopped mid-job	Job returns to `queued`; the first two interruptions do not consume an attempt	None
Process killed mid-job	On next start: back to `queued` with one attempt consumed, or `failed` if attempts are exhausted	Failure text only in the second case

After a 402 an admin tops up the balance, sends /resume to the bot, and presses Retry in the admin panel for the failed jobs. Retry works while the selfie file still exists (see Data erasure).

Cost ceiling

usage.check_budget runs before every provider call. It adds what earlier attempts of the job spent (from api_usage, since jobs.created_at) to the calls of the current attempt. When the sum reaches JOB_MAX_COST_USD the next call is refused. Calls already in flight are not cancelled, so the real spend can exceed the ceiling by the cost of those calls. An admin retry resets created_at, so the retried job gets a fresh ceiling.

After each job the worker also compares the number of image requests in the current Pacific-time day with IMAGE_RPD_LIMIT (default 1000). At IMAGE_RPD_WARN_FRAC (0.7) it sends one warning per day.

Where to go next

Pack delivery — the next stage of the same job.
Generation pipeline component — modules and prompts.
Reliability — all retry tables side by side.
Runbook: generation failing, billing 402, rate limit 429.
Cost — what a pack costs per stage.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.