Current capacity and bottlenecks
Who is this for
Section titled “Who is this for”The owner and the engineer who has to say “yes” or “no” to an event of a given size.
- Measured: one stream of guests, one pack every 60–90 seconds, on a 2 vCPU / 4 GB server: median 56 s from photo to delivery, queue wait under 1 s, no swap, no rate-limit answers.
- Not measured: several packs generating at the same time on the server. The burst part of the load test never ran.
- The first limit you hit is not the server — it is the provider’s daily image quota: about 230 packs per day at the configured limit of 1 000 image requests.
- The second is memory: one pack peaks at about 1.4 GB inside a unit capped at 1.6 GB.
- A queue cannot grow without bound: any job older than 15 minutes is failed by design.
What was measured
Section titled “What was measured”Source for this section:
docs/LOAD_TEST_REPORT.md.
The run used the production pipeline settings (figure mode, consistency mode, layout 3x4, one outfit, one photo)
in a separate process with its own database and a fake Telegram.
| Metric | Value |
|---|---|
| Server | 2 vCPU, 4 GB RAM, 2 GB swap, shared with another project |
| Packs completed | 7 in a stream (planned 10 + a burst of 3; stopped when provider credits ran out) |
| Cadence | one new pack every 60–90 s |
| Queue wait | average 0.4 s, maximum 1.0 s |
| Photo → delivery | median 56.4 s, mean 61.9 s, maximum 74.8 s |
| Real Telegram overhead (from production) | pack creation 1.0–1.9 s, link check about 2 s |
| Guest-visible wait | about 1–1.3 minutes |
| Memory | test process RSS up to 1 420 MB; available memory never below 1 694 MB of 3 915 MB; swap 0 |
| Load average | up to 1.53 on 2 vCPU |
| Rate-limit answers (429) | 0 |
| Image requests per pack | 4.29 (anchor + 3 sheets + regenerations) |
Stage timings for the same packs: anchor 9–13 s, each sheet 14–23 s (one outlier 35 s), a regeneration about 14 s.
An earlier single-pack measurement on the same server
(HANDOFF.md, consistency section,
item 15) gave 48–66 s per pack, with both cores at 100 % for about 10 s per pack (cutting and export) and an average
CPU load of 29 %.
Capacity of one station
Section titled “Capacity of one station”The repository contains no measurement of how long a guest spends at the tablet. The load test assumed one guest every 60–90 seconds.
Estimate. Station throughput = 3 600 s ÷ seconds per guest:
| Seconds per guest at the tablet | Guests per hour |
|---|---|
| 60 | 60 |
| 90 | 40 |
| 120 | 30 |
Measure the real figure at the rehearsal: time ten consecutive guests from Start to the “done” screen.
Several tablets can open the same kiosk address at once: sessions are independent and the photo-to-job step is atomic. All tablets share one queue, one brand theme and one bot.
Capacity of one server
Section titled “Capacity of one server”| Mode | Throughput | Basis |
|---|---|---|
| One pack at a time | about 60 packs per hour | Measured: about 60 s per pack, queue wait under 1 s at one pack per 60–90 s |
Two packs in parallel (WORKER_SLOTS=2, the default) |
up to about 120 packs per hour | Estimate: 2 slots × 3 600 ÷ 60 s. Not measured on the server; memory of two packs cutting at once is unknown |
| More than two | unknown | Not measured; exceeds the unit’s memory limit as configured |
Bottlenecks, in the order you hit them
Section titled “Bottlenecks, in the order you hit them”1. Provider daily image quota
Section titled “1. Provider daily image quota”IMAGE_RPD_LIMIT defaults to 1 000 image requests per day (the provider’s entry tier, per
bot/.env.example). The day resets
at midnight Pacific time.
Estimate. Packs per day = limit ÷ image requests per pack = 1 000 ÷ 4.3 ≈ 230 packs. The warning alert fires at 70 %, about 160 packs. Rate-limited and “no credits” answers are not counted.
The limit is a property of the provider account, not of the code. Raising it means a higher tier or more provider
projects. The counter in the admin panel only reports; set IMAGE_RPD_LIMIT to the real quota.
2. Provider prepaid balance
Section titled “2. Provider prepaid balance”A pack costs about $0.42 (Cost). An empty balance stops everything: the job fails, the system
pauses itself and waits for /resume. See ADR-011.
3. Memory per pack
Section titled “3. Memory per pack”Cutting three 2K sheets at once uses full-resolution float arrays. Peak process memory for one pack was measured at
1.27–1.42 GB. The bot unit has MemoryHigh=1.45G and MemoryMax=1.6G:
- above
MemoryHighthe kernel throttles the process and reclaims memory — packs get slower, they do not fail; - above
MemoryMaxthe process uses swap, and is killed only if that is not enough.
CUT_MAX_PARALLEL (default: number of vCPUs) bounds how many sheets are cut at once across all jobs, so two packs
do not double the cutting memory, but the second pack’s data is still held. This case is not measured.
4. The 15-minute deadline
Section titled “4. The 15-minute deadline”A job that has not delivered its link 15 minutes after the photo is failed and the guest is told staff will help
(bot/delivery.py). The deadline
covers queue time too.
Estimate. Maximum useful backlog ≈ throughput × 14 minutes: about 14 jobs at one pack per minute, about 28 at two. A burst larger than that fails its tail. A group of 30 people photographed in ten minutes at three tablets would already be at the edge on today’s server.
5. CPU
Section titled “5. CPU”Two vCPUs are saturated for about 10 s per pack. CPU weight is set below the neighbouring project’s, so under contention this product gets the smaller share.
6. SQLite and the single machine
Section titled “6. SQLite and the single machine”One writer at a time, one file, two processes on one host. Not a limit at the measured load. It becomes one when workers move to other machines — see ADR-002.
7. Provider request concurrency
Section titled “7. Provider request concurrency”AI_MAX_PARALLEL_IMAGES (default 9) caps simultaneous image requests per process. A laptop stress run of three
packs at once — nine sheets in flight — produced no rate-limit answers. The provider’s per-minute limits for higher
concurrency are not recorded in the repository.
8. Telegram limits on pack creation
Section titled “8. Telegram limits on pack creation”Creating a pack is one createNewStickerSet call that uploads all 12 stickers, from one bot. Telegram’s
rate limits for this method at volume are not measured or documented in the repository. The code retries name
conflicts but has no special handling for flood limits during pack creation. Treat this as an unknown that must be
tested before any event with more than a couple of packs per minute.
9. The shared droplet
Section titled “9. The shared droplet”A second project lives on the same server with a database of its own. It competes for memory and CPU, and a mistake in its web-server configuration can take HTTPS down for both — see Runbook: services stopped on the shared droplet.
10. No working fallback provider
Section titled “10. No working fallback provider”The fallback provider has no key, so a provider outage is an outage — see Known gaps.
Summary
Section titled “Summary”| Question | Answer today |
|---|---|
| Guests per day on one provider account at entry tier | about 230 (estimate) |
| Packs per hour, proven | about 60 |
| Packs per hour, probable but unproven | about 120 |
| Largest burst that still delivers | about 14–28 queued jobs (estimate) |
| Stations on one server | several, limited by the numbers above |
Where to go next
Section titled “Where to go next”- Scaling scenarios — what to change for 70, 500, 2 000 and 10 000 guests.
- Cost — money per pack and per event.
- Performance — timings in more detail.
Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.