Performance
Who is this for
Section titled “Who is this for”Engineers who size a server or tune concurrency for an event, and front-end developers who must keep the landing page fast.
- Photo to link: median 56.4 s, mean 61.9 s, maximum 74.8 s over 7 packs in the server load test
(
docs/LOAD_TEST_REPORT.md). The report expects roughly 1.5 s more with the real Telegram API. - Memory: process resident memory peaked at 1420 MB with 2 worker slots and 2 parallel cuts on a 2 vCPU / 4 GB
server (
docs/LOAD_TEST_REPORT.md). One pack alone peaked at 1.27 GB when three sheets were cut at once (HANDOFF.md). - Not measured: three packs generated at the same moment with the current pipeline on the server. Treat any number above one station as an estimate.
- Three knobs set the load:
WORKER_SLOTS,AI_MAX_PARALLEL_IMAGES,CUT_MAX_PARALLEL. The systemd units cap memory and lower CPU priority because the server is shared. - Landing: the repository holds rules, not numeric budgets. The only hard thresholds in code are in
scripts/audit.mjs(for example layout shift must be zero). Lighthouse and JS weight scripts report numbers but do not fail on them.
Product: measured timings
Section titled “Product: measured timings”Server load test
Section titled “Server load test”Source: docs/LOAD_TEST_REPORT.md in the product repository. Conditions: 2 vCPU / 4 GB server with 2 GB swap;
tools/load_test.py running as a separate process with its own database and a fake Telegram; 2 worker slots;
AI_MAX_PARALLEL_IMAGES=9; CUT_MAX_PARALLEL=2; figure mode with CONSISTENCY_MODE=1 and SHEET_LAYOUT=3x4; one
test photo; packs submitted 60–90 s apart. Seven packs completed before the test stopped on a billing error.
| Metric | Value |
|---|---|
| Queue wait | mean 0.4 s, maximum 1.0 s |
| Photo accepted → “delivered” | median 56.4 s, mean 61.9 s, maximum 74.8 s |
| Anchor portrait | 9.2–13.0 s |
| One sheet | 13.6–23.2 s typical; 35.2 s once, when a sheet was regenerated |
| Single-figure regeneration, when it happened | 14.3–14.5 s |
| Wait for an image or cut slot | at most 2.2 s |
| 429 answers | 0 |
| Image requests per pack | 4.29 on average |
The report adds that the real Telegram API costs about 1.5 s more than the fake one (pack creation 1.0–1.9 s on the production bot, link check 2.0–2.1 s), and concludes that a guest waits about 1–1.3 minutes from photo to link.
Pipeline generations compared
Section titled “Pipeline generations compared”Source: docs/COST_HISTORY.md, table “price and time of a pack by stage”. Time is from job pick-up to a finished
pack, without delivery. Only completed packs.
| Pipeline stage | Where | Packs | Time, mean / median |
|---|---|---|---|
| Composite (AI head on a body template) | Laptop | 8 | 81 s / 86 s |
| First figure version with passport and per-sheet QA | Laptop | 4 | 104 s / 92 s |
| Figure, 2 sheets in sequence, per-sheet QA | Production server | 26 | 99 s / 92 s |
Consistency, layout 2x6 |
Laptop | 13 | 118 s / 124 s |
Consistency, layout 3x4 |
Laptop | 7 | 110 s / 100 s |
Consistency, layout 3x4, OpenCV cutting |
Server | 10 | 60 s / 56 s |
The same document notes that the laptop rows were measured under CPU throttling and parallel runs, so their times are inflated; the last row is the comparable one.
Individual measurements
Section titled “Individual measurements”Source: HANDOFF.md.
| Measurement | Value | Conditions |
|---|---|---|
| One pack, no pack creation | 66.0 s, 49.0 s, 48.5 s | Server, layout 3x4, one pack at a time |
| Cutting one sheet while three are cut together | 2.4–4.2 s | Same run |
| Export of 12 stickers | 4.0–4.6 s | Same run |
| CPU | 29 % average; both cores at 100 % for about 10 s per pack (cutting and export) | Same run |
| Three packs at once, 9 sheets in flight | about 94 s each, 0 × 429 | Stress run; layout 3x4 |
| Cut + outline + export of one 2048 px sheet after the move to OpenCV | 6.1 s → 2.0 s | Laptop |
| Passport | about 4 s | — |
| Per-sheet QA | about 5 s per sheet | — |
| Sheet regeneration | adds 20–40 s | — |
HANDOFF.md names the two sources of variance: a sheet with the wrong number of figures (a full sheet
regeneration) and figures cut by the sheet edge (single-figure regeneration with a budget of up to 35 s).
What dominates the time
Section titled “What dominates the time”From the tables above: the provider’s image generation is the long pole (one anchor plus parallel sheets, roughly 10–13 s and 14–23 s). CPU work on the server (cutting and export) is a few seconds per pack. Queue wait is negligible for one station.
Product: memory
Section titled “Product: memory”| Fact | Value | Source |
|---|---|---|
| Peak resident memory of the generating process, packs arriving one after another | 1420 MB | docs/LOAD_TEST_REPORT.md |
| Peak for a single pack with three sheets cut simultaneously | 1.27 GB | HANDOFF.md |
| Memory per sheet being cut | about 0.4 GB | Comment in bot/.env.example and worker.py |
| Lowest available memory during the load test | 1694 MB of 3915 MB, swap unused | docs/LOAD_TEST_REPORT.md |
| Idle production bot process during the load test | 206 MB | docs/LOAD_TEST_REPORT.md |
| API process peak (journald “memory peak”) | 165–192 MB | HANDOFF.md |
| Load average during the load test | up to 1.53 on 2 vCPU | docs/LOAD_TEST_REPORT.md |
HANDOFF.md records that the margin between the measured peak (1.42 GB) and the bot’s MemoryHigh (1.45 G) is
about 30 MB, and that two packs inside the cutting stage at the same moment were not measured. Above MemoryHigh the
kernel slows the process down and reclaims its memory; it does not kill it.
Product: concurrency knobs
Section titled “Product: concurrency knobs”| Variable | Default in code | In .env.example |
What it limits | Raise it when | Cost of raising |
|---|---|---|---|---|---|
WORKER_SLOTS |
2 | 2 | Jobs generated in parallel by one bot process | Queue wait grows (queue= in STAGE_SUMMARY) |
Memory per concurrent pack, more parallel provider calls |
AI_MAX_PARALLEL_IMAGES |
9 | 9 | Image requests in flight per process, all slots together | slot_wait= appears in STAGE_SUMMARY and there are no 429s |
More 429 answers if the provider’s per-minute quota is lower |
CUT_MAX_PARALLEL |
number of vCPUs | empty (= vCPUs) | Sheets being cut at once per process | Never above the vCPU count | About 0.4 GB per extra sheet; cutting is CPU-bound |
AI_TIMEOUT_SEC |
180 | 180 | One image request | — | A hung call holds a slot longer |
CONSISTENCY_MAX_REGEN |
4 | 4 | Single-figure regenerations per pack | Quality matters more than time | Time and cost per pack |
CONSISTENCY_REGEN_BUDGET_SEC / EDGE_REGEN_TIMEOUT_SEC |
20 / 35 | 20 / 35 | Extra wait per pack for regenerations | — | Directly adds to the worst-case time |
The legacy name WORKER_CONCURRENCY is still read when WORKER_SLOTS is not set. Slots are asyncio tasks in one
process, not separate processes; heavy image work runs in threads.
Why 9 parallel images: one pack in layout 3x4 sends 3 sheets at once, and the stress run of three simultaneous
packs (9 sheets in flight) produced no 429 (config.py comment, HANDOFF.md).
Estimate — packs per hour for one bot process. Formula: WORKER_SLOTS × 3600 / seconds per pack. With 2 slots
and the load-test mean of 61.9 s: 2 × 3600 / 61.9 ≈ 116 packs per hour. This is an upper bound on paper: it ignores
the provider’s per-minute and daily quotas, and the memory of two packs cutting at once has not been measured. See
Capacity.
Product: systemd limits
Section titled “Product: systemd limits”From
sticker-bot.service
and
sticker-api.service.
| Setting | Bot | API | Effect |
|---|---|---|---|
MemoryHigh |
1.45G | 1.2G | Above it the kernel throttles the process and reclaims memory |
MemoryMax |
1.6G | 1.6G | Hard RAM cap; beyond it the process uses swap, then is killed |
CPUWeight |
50 | 50 | Half the default weight of 100, so the co-tenant wins a CPU contest |
| Swap | 2 GB swap file, vm.swappiness=10 |
Safety net only (deploy/swap.sh) |
update.sh prints the effective values and the memory peak of both units after every restart.
Product: smaller design choices that matter
Section titled “Product: smaller design choices that matter”| Choice | Effect | Where |
|---|---|---|
| Sheets generated in parallel with an anchor reference | Removed one sequential generation | worker.build_figure_stickers_consistent |
Layout 3x4 instead of 2x6 |
Fewer figures at the sheet edge, fewer regenerations | catalog.SHEET_LAYOUTS |
| Edge defects fixed by single 1K figures instead of whole 2K sheets | Saves a sheet regeneration | worker._regen_outliers |
| Vision inputs downscaled to 1024 px | QA call about 5 s instead of about 15 s (module comment) | ai_pipeline.VISION_MAX_SIDE |
| Request images sent as original bytes when possible | Avoids re-encoding a JPEG as a larger PNG | ai_pipeline._gemini_image_part |
| One cached provider client per process | Saves about 0.5 s per call (module comment) | ai_pipeline._gemini_client |
| Skin normalisation and export per figure in threads | Parallel instead of sequential over 12 stickers | worker.build_figure_stickers_consistent |
| Queue poll every 1 s | Average pick-up delay of half a second | worker.POLL_INTERVAL_SEC |
| Delivery check outside the worker slot | The slot is free as soon as the pack exists | delivery.spawn |
| Kiosk outfit previews versioned by content hash | Cached for a year, loaded before the guest reaches the screen | api.template_preview, kiosk.js |
Landing: rules
Section titled “Landing: rules”The landing’s CLAUDE.md lists the rules that keep the mobile Lighthouse score. They are conventions enforced by
review and partly by npm run audit, not numeric budgets.
| Rule | Reason given in the repository |
|---|---|
| The first screen does not wait for JavaScript: the headline animates with CSS | Waiting for hydration cost about 6 s of LCP on mobile |
| Client components receive only their slice of the dictionary | Everything passed to a client component is serialised into the HTML |
Mock-ups below the first screen mount on approach (LazyMount) with a same-size placeholder |
Less work before first paint, no layout shift |
| The tablet mock renders only the current screen | Fewer DOM nodes |
Validation uses zod/mini and is imported only on submit |
The full build was a 106 KB gzip chunk on the first screen |
| No animation library on the page; motion is CSS | Keeps the bundle small |
Fonts use swap, only two are preloaded, with size-adjusted fallbacks |
First frame does not wait for a font; no line-count change on swap |
Image widths restricted in next.config.ts |
srcset attributes were half of the HTML; the /ru document went from 295 KB to 180 KB |
Do not use experimental.inlineCss or content-visibility: auto on sections |
The first doubled the document size (31 → 71 KB gzip); the second broke anchor scrolling |
| Text inside mock-ups is at least 12 px | Lighthouse “legible font sizes” |
Recorded state in the same file: first-load JavaScript of about 172 KB gzip; simulated mobile LCP typically 3.5–3.6 s with FCP 1.1 s; a Performance score above 90 is accepted. The file names the next step toward an LCP of 2.5 s or less: less client JavaScript on the first screen.
Landing: what the scripts measure
Section titled “Landing: what the scripts measure”All three need a running production build (npm run build && npm start); SHOTS_BASE changes the address.
| Script | Measures | Thresholds in code |
|---|---|---|
scripts/lighthouse.mjs |
Lighthouse 12 through Playwright’s Chromium: /ru, /uz, /en on mobile and /ru on desktop. Prints category scores, FCP, LCP, total blocking time, cumulative layout shift, speed index, and every non-performance audit below a perfect score. Writes HTML and JSON reports to shots/lighthouse |
None on scores. A run is killed after 120 s |
scripts/js-weight.mjs |
For each locale at a 390 × 844 viewport: gzip size (level 9) of all same-origin scripts on first load, and again after scrolling to the bottom | None; it prints numbers |
scripts/audit.mjs |
A long list of layout, accessibility and motion checks in a real browser at 360, 768, 1280 and 1920 px | Yes; any finding makes the script fail |
Performance-related thresholds inside audit.mjs:
| Check | Threshold |
|---|---|
| Cumulative layout shift over load and a full scroll | must not exceed 0.001 |
| Headline line count with web fonts blocked versus loaded | must be equal |
| Headline length | at most 4 lines at 360 and 768 px, 3 lines at 1280 and 1920 px |
| Subtitle length | at most 4 lines at 360 and 768 px, 2 lines at 1280 and 1920 px |
| Animations on a resting first screen, below the fold, in a hidden tab, under reduced motion | none may be running |
| Text inside mock-ups at 1280 px | at least 12 px |
| Horizontal overflow | 0 px |
| Hero call-to-action at 360 px | within the first 780 px |
Every check script runs under a watchdog
(scripts/lib/watchdog.mjs):
a step without progress for a minute fails with the step name, so a hang becomes an error instead of a wait.
Where to go next
Section titled “Where to go next”- Capacity and Scenarios — what these numbers mean for an event size.
- Cost — the money side of the same measurements.
- Landing quality tooling — how to run the landing scripts.
- Observability — how to read
STAGE_SUMMARYon a live server. - Runbook: disk and memory — what to do when the limits bite.
Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.