Services stopped on a shared droplet

Who is this for

The engineer on call when the product stopped working although nobody deployed it, on a server that also hosts the neighbouring project on the same droplet.

TL;DR

Check five things: the three services are active, the three services are enabled, Caddy validates, port 8000 answers, the backup timer is listed.
Restart order: Caddy (after caddy validate) → sticker-api → sticker-bot. Enable everything so it survives a reboot.
Most work resumes by itself: queued jobs, interrupted jobs, link checks, unpublished leads, missed cleanup.
Jobs older than 15 minutes fail on the first sweep after the start. Retry them from the admin panel.
Leads sent while the API was down arrived through the fallback channel and are not in the database.

Symptom

/health does not answer, the kiosk shows an error, the bot is silent, or all of these at once.
No alert arrived. Alerts are sent by the bot process, so a stopped bot cannot report its own stop.
It started after work on the server: a deploy of the other project, a package upgrade, a reboot, a Caddy change.

Likely causes

Someone ran systemctl stop or systemctl disable on our units (a wrong unit name, a wildcard, a cleanup script).
The server was rebooted and our units were not enabled.
Caddy was restarted with an invalid site file of the other project, so it serves nothing.
/etc/caddy/Caddyfile was overwritten and lost the import /etc/caddy/sites/*.caddy line, or our site file was removed.
A port conflict: another process took port 8000.
The firewall was reset and 80 or 443 is closed.
A package upgrade changed Python or removed something the venv needs.

Diagnose

systemctl is-active  sticker-bot sticker-api caddy sticker-backup.timer
systemctl is-enabled sticker-bot sticker-api caddy sticker-backup.timer
systemctl list-units --failed --no-pager
uptime
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
cat /etc/caddy/Caddyfile; ls -l /etc/caddy/sites/
curl -s localhost:8000/health
ss -tlnp | grep -E ':(80|443|8000)\s'
ufw status

What you see	Cause
`inactive` for our services, short `uptime`	2 — reboot without enabled units
`inactive`, long `uptime`	1 — stopped by hand
`disabled` in the second command	1 or 2 — will not survive the next reboot either
`caddy` is `failed`; `caddy validate` names another site file	3
`Caddyfile` has no import line, or `sites/stickers.caddy` is missing	4
`sticker-api` restarts in a loop with `address already in use` in its journal; `ss` shows another program on 8000	5
`ufw status` lacks 80 or 443	6
`ModuleNotFoundError` or an interpreter error in the journal	7

When and how it stopped:

journalctl -u sticker-bot -u sticker-api -u caddy --since -6h --no-pager | grep -E 'Stopping|Stopped|Started|Failed|SHUTDOWN' | tail -30
last -n 10
journalctl --since -6h --no-pager | grep -E 'systemctl|apt|dpkg' | tail -20

SHUTDOWN: получен SIGINT — останавливаюсь (“received SIGINT — stopping”) followed by SHUTDOWN: остановлен за N c means a clean stop by systemd. No SHUTDOWN lines before the gap means the process was killed.

How long was it down, and what is waiting:

sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT status, COUNT(*), MIN(created_at) FROM jobs WHERE status IN ('queued','processing','delivering') GROUP BY status;"
sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT COUNT(*) FROM leads WHERE published_at IS NULL;"

Act

Restart from the outside in.

Step 1 — Caddy

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile

Valid → systemctl enable --now caddy (or systemctl restart caddy if it is running but not serving).
Error in the other project’s file → move it aside and tell its owner: mv /etc/caddy/sites/<their-file>.caddy /root/, validate again, systemctl restart caddy.
Import line or our site file missing → bash /opt/ai-sticker-pack/deploy/caddy.sh. If it refuses because the Caddyfile is foreign, add import /etc/caddy/sites/*.caddy to /etc/caddy/Caddyfile by hand and run it again.

Step 2 — API

systemctl enable --now sticker-api
curl -s localhost:8000/health

If port 8000 is taken (cause 5), find the owner with ss -tlnp | grep ':8000' and have it moved. Port 8000 on the loopback interface belongs to this product; the other project has its own port.

Step 3 — bot

systemctl enable --now sticker-bot
journalctl -u sticker-bot -n 40 --no-pager -o cat

Step 4 — backup timer and firewall

systemctl enable --now sticker-backup.timer
ufw allow 80/tcp; ufw allow 443/tcp

Shortcut when Caddy is fine and only our services were down: bash /opt/ai-sticker-pack/deploy/update.sh --yes restarts both, reinstalls our Caddy site if it differs, and checks /health. It does not enable disabled units; run the enable commands as well.

Cause 7 (broken Python environment): reinstall dependencies, then restart.

sudo -u sticker /opt/ai-sticker-pack/venv/bin/pip install -q -r /opt/ai-sticker-pack/bot/requirements.txt
bash /opt/ai-sticker-pack/deploy/update.sh --yes

If the venv itself is broken after a Python upgrade, remove /opt/ai-sticker-pack/venv and upload again with push.ps1 -Restart from the workstation; it recreates the venv.

What resumes automatically

Item	After the start
Jobs in `queued`	Taken by the worker in order
Jobs in `processing`, clean stop	Already back in `queued` (marker `[requeued on shutdown]`), attempt not charged; generation starts over
Jobs in `processing`, process killed	Returned to `queued` with the attempt charged (`[requeued after restart]`); if that was the last attempt → `failed`, alert, guest notified
Jobs in `delivering`	The sweeper restarts the link check within 20 seconds
Any active job older than 15 minutes since the photo	Fails on the first sweep: the guest gets the “we will help by hand” message and you get `STICKERSET_DELIVERY_FAILED` alerts. Jobs between 5 and 15 minutes old trigger the one-time delay notice.
Leads stored by the API while the bot was down	Published within seconds
Leads sent while the API was down	Not stored. The landing sent them to the group directly with the `⚠️ Запасной канал` marker (if its fallback bot is configured).
Lead reminders and the daily digest	Reminders go out at the next working-hours pass. A missed digest is sent only if the bot is back before 12:00 in the leads time zone; otherwise that day is skipped.
Photo TTL cleanup	Did not run while the bot was down. It runs immediately at start and then hourly.
Nightly database backup	Independent of our services. With `Persistent=true` a run missed during power-off happens after boot.
Maintenance pause (`admin` or `auto`)	Still on: it is stored in the database
Alert cooldowns, admin-login and leads rate-limit counters	Reset: they live in process memory

What needs you

Retry the jobs that failed because of the outage: admin panel → status filter Упал → Повторить джоб. Works while the selfie exists (up to 72 hours from the photo).
Process ⚠️ Запасной канал messages by hand: Leads missing or fallback.
Tell the operator on site what happened and what to say to guests who waited.

Verify

systemctl is-active  sticker-bot sticker-api caddy sticker-backup.timer     # active ×4
systemctl is-enabled sticker-bot sticker-api caddy sticker-backup.timer     # enabled ×4
curl -s https://stickers.eventtalks.uz/health                               # {"ok":true,"db":"ok",…}
journalctl -u sticker-bot -n 40 --no-pager -o cat | grep -E 'STARTUP|Воркер очереди запущен|Вернул в очередь|LEADS:'
systemctl list-timers sticker-backup.timer

Then one guest pass from the tablet and one /stats to the bot.

Prevent

Write down the shared-server rules and give them to everyone with root access:
- unit names starting with sticker- belong to this product; never use wildcards with systemctl stop;
- each project edits only its own file in /etc/caddy/sites/; nobody replaces /etc/caddy/Caddyfile;
- caddy validate first, then systemctl reload caddy; never restart with an unvalidated configuration;
- port 8000 on the loopback interface is taken; only 22, 80 and 443 are open in the firewall;
- no deploys, upgrades or reboots during an announced event window.
Keep all four units enabled; check systemctl is-enabled at rehearsal.
Add an external uptime check, because a stopped bot sends no alert: Monitoring and alerts.
Take a snapshot before the other project does major work.
For events that matter, move this product to its own server: Scenarios.

Where to go next

Kiosk 502 — the Caddy and API part in more detail.
Disk and memory — when the neighbour’s load, not its commands, is the problem.
Deploy and rollback — the Caddy layout and what caddy.sh touches.
Runbooks — back to the symptom table.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.