Disk and memory

Who is this for

The engineer on call when packs become slow, the bot restarts by itself, or the server reports no space.

TL;DR

Slow packs with no errors usually mean the bot is being throttled at MemoryHigh. Check memory.events.
A bot that restarts mid-pack was killed for memory. A job killed twice goes to failed and needs a manual retry.
Limits can be raised at runtime without a restart: systemctl set-property --runtime sticker-bot MemoryHigh=1.55G.
A full disk breaks the database first. Free space, then restart both services.
Guest photos are removed by the bot’s own TTL cleanup. Do not delete files under bot/data/sessions by hand during an event.

Symptom

Packs take well over 2 minutes while the provider is fine (no RATE_LIMIT lines).
sticker-bot shows a recent start time although nobody restarted it; jobs show [requeued after restart].
Alert: Job N FAILED: процесс бота падал посреди его генерации (“the bot process crashed during its generation”).
Errors containing No space left on device, database or disk is full, or /health answering 503 with "db":"error".
The night backup log has a FAIL line.

Likely causes

The bot’s memory peak exceeds MemoryHigh (1.45 GB): the kernel throttles it and swaps its memory out.
The bot hit MemoryMax (1.6 GB) and swap could not absorb it: the kernel killed it.
The whole machine is short of memory: the neighbouring project on the same droplet plus this one.
Swap is missing or off.
The disk is full: journal, Caddy logs, database copies, debug sheets, or the neighbour’s data.

Reference numbers: peak process memory of 1420 MB in the load test, with WORKER_SLOTS=2 and CUT_MAX_PARALLEL=2, on a 2 vCPU / 4 GB server with 2 GB swap (docs/LOAD_TEST_REPORT.md). Three packs generating at the same moment were not measured.

Diagnose

Memory

free -m
swapon --show
systemctl show sticker-bot -p MemoryHigh -p MemoryMax -p MemoryPeak -p MemorySwapPeak -p ActiveEnterTimestamp
cat /sys/fs/cgroup/system.slice/sticker-bot.service/memory.events
journalctl -k --since -2h | grep -i -E 'oom|killed process'
ps -eo pid,user,rss,cmd --sort=-rss | head -8
journalctl -u sticker-bot --since -30min --no-pager -o cat | grep STAGE_SUMMARY | tail -10

What you see	Meaning	Cause
`memory.events`: `high` grows between two reads	The bot is being throttled at `MemoryHigh`	1
`memory.events`: `oom_kill` above 0	The bot was killed at its own `MemoryMax`	2
Kernel log has `oom` / `Killed process`, but `oom_kill` in our cgroup is 0	The machine as a whole ran out; the victim may be either project	3
`swapon --show` prints nothing	No swap	4
`free -m`: little `available`, swap `used` near total	Machine-wide pressure	3
`STAGE_SUMMARY`: `cut…` and `stickers` stage times grew, `gen…` did not	Local CPU or memory is the bottleneck, not the provider	1 or 3
`ps`: a process of another user at the top	The neighbour is the large consumer	3

Disk

df -h /
du -sh /opt/ai-sticker-pack/bot/data/* /opt/ai-sticker-pack/logs /var/backups/sticker-pack /var/log/caddy 2>/dev/null
journalctl --disk-usage
ls -lh /opt/ai-sticker-pack/bot/data/app.db*
du -xh / --max-depth=2 2>/dev/null | sort -h | tail -15

What normally grows:

Path	Content	Bounded by
`bot/data/sessions/`, `bot/data/debug_sheets/`	Selfies, generated sheets (several MB each)	TTL cleanup, every hour, files older than `PHOTO_TTL_HOURS`
`bot/data/app.db.bak-*`	Automatic database copies before a migration, and before each guest reset	Nothing. They are never rotated.
`logs/.log`	Application logs	5 MB × 3 by the application, plus daily logrotate keeping 14 days
`/var/backups/sticker-pack/`	Compressed nightly database backups	Last 14
`/var/log/caddy/stickers.log`	Access log	20 MiB × 14
systemd journal	Service output	System default

Act

1 — throttling at MemoryHigh (packs slow, high counter growing)

Raise the threshold at runtime. No restart; lasts until the next reboot:

systemctl set-property --runtime sticker-bot MemoryHigh=1.55G

Keep it below MemoryMax (1.6 GB). Or lower the peak instead, which needs a restart:

E=/opt/ai-sticker-pack/bot/.env
setenv() { grep -q "^$1=" $E && sed -i "s/^$1=.*/$1=$2/" $E || echo "$1=$2" >> $E; }
setenv CUT_MAX_PARALLEL 1
bash /opt/ai-sticker-pack/deploy/update.sh --yes

CUT_MAX_PARALLEL=1 lowers the peak by about 0.4 GB and makes a pack about 3 seconds longer (docs/EVENT_RUNBOOK.md). WORKER_SLOTS=1 is the stronger measure: one pack at a time.

2 — killed at MemoryMax (oom_kill above 0)

systemctl set-property --runtime sticker-bot MemoryMax=2G

Do this only if free -m shows the machine has the room. Then lower the peak as in block 1. Retry the jobs that went to failed: Generation failing.

3 — the whole machine is short

Lower our own footprint: WORKER_SLOTS=1, CUT_MAX_PARALLEL=1.
Talk to the owner of the neighbouring project before touching their processes.
If the event is running and the machine cannot cope, resize the droplet in the hosting panel. That needs a power-off; /pause first and wait for "active_jobs":0.

4 — no swap

bash /opt/ai-sticker-pack/deploy/swap.sh

Expected: swap: /swapfile 2G, swappiness=10.

5 — disk full

Free space in this order, safest first:

journalctl --vacuum-size=200M
apt-get clean
ls -lt /opt/ai-sticker-pack/bot/data/app.db.bak-*        # old automatic copies; keep the newest one or two
rm /opt/ai-sticker-pack/bot/data/app.db.bak-<old time>*
find /var/log/caddy -name 'stickers*.log*' -mtime +7 -delete

Then restart, because a process that hit a full disk may hold a broken database connection:

bash /opt/ai-sticker-pack/deploy/update.sh --yes
sqlite3 /opt/ai-sticker-pack/bot/data/app.db 'PRAGMA integrity_check;'

Do not delete app.db, app.db-wal, app.db-shm, main.lock, or anything under bot/data/sessions for jobs that are still queued, processing or failed (a retry needs the selfie). After the event, expired guest files go by themselves: Backups and data lifecycle.

To make a runtime limit permanent, change deploy/systemd/sticker-bot.service in the repository and deploy; update.sh installs changed units. Runtime properties disappear on reboot.

Verify

cat /sys/fs/cgroup/system.slice/sticker-bot.service/memory.events    # high / oom_kill stop growing
systemctl show sticker-bot -p MemoryHigh -p MemoryMax
free -m; df -h /
curl -s localhost:8000/health
journalctl -u sticker-bot --since -10min --no-pager -o cat | grep STAGE_SUMMARY | tail -3

Expected: "ok":true,"db":"ok", recent packs back to about 1–1.3 minutes in STAGE_SUMMARY total=….

Prevent

Check free -m, df -h /, swapon --show and the unit limits at rehearsal and on the morning of the event.
Keep WORKER_SLOTS=2 on a 4 GB server; the limits in the unit files were sized for it.
Remove old app.db.bak-* copies after each event (they also contain guest data).
Agree memory budgets with the neighbouring project, or move one of the two projects to its own server.
Record MemoryPeak after each event; if it approaches MemoryHigh, raise the limit in the unit file before the next one.

Where to go next

Services stopped on a shared droplet — when the neighbour is the cause.
Backups and data lifecycle — what may be deleted and when.
Capacity — sizing the server for more stations.
Runbooks — back to the symptom table.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.