Kiosk 502

Who is this for

The engineer on call when the tablet shows an error page instead of the kiosk, or https://stickers.eventtalks.uz does not open.

TL;DR

• 502 means Caddy is up and the API behind it is not. Anything else (timeout, connection refused, certificate error) points at Caddy, DNS or the firewall.
• Test the API directly on the server: curl -s localhost:8000/health. If that works, the problem is in front of it.
• On a shared server the usual Caddy failure is a neighbour’s broken site file followed by a Caddy restart.
• bash /opt/ai-sticker-pack/deploy/caddy.sh restores our own site file from the deployed repository copy.
• The bot keeps working while the kiosk is down: packs already in generation are delivered.

Symptom

• The tablet shows 502 Bad Gateway, a blank page, a browser certificate warning or “site cannot be reached”.
• curl -s https://stickers.eventtalks.uz/health returns nothing, an error or an HTML error page.
• The kiosk shows нет связи с сервером (“no connection to the server”) on every screen. If only one tablet is affected, start with Station stuck.

Likely causes

1. sticker-api is stopped or crash-looping (bad bot/.env, most often an empty ADMIN_PANEL_PASSWORD in prod).
2. Caddy is not running because a site file in /etc/caddy/sites/ is invalid and Caddy was restarted.
3. Our site file is missing or damaged.
4. The certificate could not be issued or renewed (DNS wrong, port 80 or 443 blocked, rate limit).
5. DNS does not point at the server.
6. The firewall blocks 80 or 443.

Diagnose

systemctl is-active sticker-api caddy
curl -s localhost:8000/health
curl -s -o /dev/null -w '%{http_code}\n' https://stickers.eventtalks.uz/health

Local /health	Public URL	Go to
No answer	502	Cause 1
OK	000 (connection failed)	Causes 2, 5, 6
OK	Certificate error	Cause 4
OK	404 or another site’s content	Cause 3

# 1: why the API is down
systemctl status sticker-api --no-pager | head -12
journalctl -u sticker-api -n 40 --no-pager -o cat

# 2 and 3: Caddy
systemctl status caddy --no-pager | head -8
journalctl -u caddy -n 40 --no-pager
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
cat /etc/caddy/Caddyfile
ls -l /etc/caddy/sites/

# 4: certificate
journalctl -u caddy --since -6h --no-pager | grep -i -E 'certificate|acme|obtain|renew|error' | tail -20

# 5: DNS (compare with the server's public address in the hosting panel)
getent hosts stickers.eventtalks.uz

# 6: firewall and listeners
ufw status
ss -tlnp | grep -E ':(80|443|8000)\s'

What to look for:

• API journal: Конфигурация некорректна: … (“configuration is invalid”) names the setting. A Python traceback right after a deploy means a code problem: roll back.
• caddy validate prints the file and line of the error. Note whether it is stickers.caddy or another file.
• /etc/caddy/Caddyfile must contain the line import /etc/caddy/sites/*.caddy, and /etc/caddy/sites/stickers.caddy must exist.
• ss must show Caddy on 80 and 443 and Python on port 8000.

Act

1 — API down

nano /opt/ai-sticker-pack/bot/.env          # only if the journal named a setting
systemctl enable --now sticker-api
systemctl restart sticker-api

If it broke right after a code update: Rollback to the previous commit.

2 — Caddy broken by another site file

On a reload with an invalid configuration Caddy keeps the old configuration and keeps serving. On a restart it does not start at all. So a neighbour’s bad file plus a restart takes our site down too.

mv /etc/caddy/sites/<their-file>.caddy /root/      # move the broken file aside, do not delete it
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
systemctl restart caddy

Tell the owner of the neighbouring project at once: their site is now down and their file is in /root/.

3 — our site file missing or damaged, or the import line is gone

bash /opt/ai-sticker-pack/deploy/caddy.sh

The script reinstalls /etc/caddy/sites/stickers.caddy, validates and reloads. If it stops with a message that /etc/caddy/Caddyfile is a foreign configuration without the import line, add the line import /etc/caddy/sites/*.caddy to that file by hand and run the script again. It never overwrites a Caddyfile it does not recognise.

4 — certificate

Fix the underlying cause (DNS, ports 80 and 443 open), then systemctl reload caddy. Caddy retries on its own. Do not restart it in a loop: repeated failed attempts run into the certificate authority’s rate limits.

5 — DNS

Correct the A record at the DNS provider. Nothing on the server needs to change.

6 — firewall

ufw allow 80/tcp
ufw allow 443/tcp

Verify

caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile   # Valid configuration
systemctl is-active sticker-api caddy                               # active, active
curl -s https://stickers.eventtalks.uz/health                       # {"ok":true,"db":"ok",…}

Reload the kiosk page on the tablet and start one session up to the QR screen.

Prevent

• Agree with the neighbouring project: always caddy validate before touching Caddy, and systemctl reload caddy, never restart.
• Each project edits only its own file in /etc/caddy/sites/.
• Check caddy validate and ls /etc/caddy/sites/ in the rehearsal checklist.
• Add an external HTTPS check on /health: Monitoring and alerts.

Where to go next

• Services stopped on a shared droplet — the wider version of this incident.
• Deploy and rollback — how the Caddy layout works.
• Station stuck — when only the tablet is affected.
• Runbooks — back to the symptom table.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.