Deploy and rollback

Who is this for

An operator or engineer who has to put the product on a server, update it, or undo an update. You need a Windows machine with the ai-sticker-pack checkout (the upload script is PowerShell) and root SSH access to the server.

TL;DR

• Order on a new server: deploy/setup.sh → deploy/push.ps1 → create bot/.env by hand → systemctl enable --now sticker-api sticker-bot → check /health.
• Routine update: commit, then powershell -ExecutionPolicy Bypass -File deploy\push.ps1 -Restart.
• Only committed files are uploaded (git archive HEAD). bot/.env, the database, photos and logs never travel with the code.
• Fastest rollback is a flag in bot/.env plus bash /opt/ai-sticker-pack/deploy/update.sh --yes. The next step is the previous commit via a detached checkout and push.ps1 -Restart.
• A restart is safe for jobs in flight: unfinished jobs return to the queue, link checks resume. The 5 and 15 minute delivery deadlines keep running.
• The landing page is a separate Vercel project. Its rollback is “redeploy a previous deployment” or “remove LEADS_SECRET and redeploy”.

What runs where

Thing	Location
Code	/opt/ai-sticker-pack (owner sticker, a system user without a login shell)
Python environment	/opt/ai-sticker-pack/venv
Configuration and secrets	/opt/ai-sticker-pack/bot/.env (owner sticker, mode 600)
Database, selfies, debug sheets, instance lock	/opt/ai-sticker-pack/bot/data/ (mode 750)
Application logs	/opt/ai-sticker-pack/logs/app.log (bot), /opt/ai-sticker-pack/logs/api.log (API)
Deployed commit	/opt/ai-sticker-pack/VERSION (short hash, shown by /health)
Database backups	/var/backups/sticker-pack/
Caddy site file	/etc/caddy/sites/stickers.caddy
Public endpoints	https://stickers.eventtalks.uz/kiosk/, /admin, /health, /leads

The API listens on port 8000 on the loopback interface only. Caddy terminates HTTPS on ports 80 and 443 and proxies to it. HTTPS is mandatory: the tablet camera (getUserMedia) does not work over plain HTTP.

New installation: the whole order on one page

#	Step	Where	Done when
1	Choose a server size for the event — Scaling scenarios (pilot of about 70 guests: 2 vCPU / 4 GB; about 500 guests: dedicated, 4 vCPU / 8 GB; about 800: 8 vCPU / 16 GB — all estimates except the pilot)	DigitalOcean	Server reachable over SSH with a key
2	DNS: an A record for the API domain pointing at the server	DNS provider	The name resolves to the server
3	Create the bot in BotFather; create an AI provider project and key	Telegram, provider console	Token and key in hand (never in git)
4	Steps 1–5 below: setup.sh, push.ps1 -Server root@<server>, bot/.env (minimum list), start, verify	Server	/health answers ok: true over HTTPS
5	Leads module (step 6 below): secret, salt, group, admins; bot added to the group as admin	Server, Telegram	A test lead appears as a card
6	Landing on Vercel (below): import, Node 22, env vars with the same LEADS_SECRET, domain via the CNAME Vercel shows	Vercel, DNS provider	The form sends a test request that becomes a card
7	Branding of the kiosk, theme, logos; commit and push	Workstation	The kiosk shows the event look
8	Full rehearsal from the real tablet — Event playbook	Venue	A pack arrives end to end
9	Snapshot of the droplet	DigitalOcean	Snapshot listed

From zero on a new server

Prerequisites

Item	Value	Source
OS	Ubuntu 24.04 (ships Python 3.12; bot/requirements.txt is compiled for Linux + Python 3.12)	deploy/README.md
Size used in the load test	2 vCPU / 4 GB RAM plus 2 GB swap	docs/LOAD_TEST_REPORT.md
DNS	An A record for the API domain (stickers.eventtalks.uz) pointing at the server. Create it before setup.sh, otherwise Caddy cannot obtain a certificate.	deploy/README.md
SSH	Key-based root login: ssh root@<server> works without a password	deploy/README.md
Workstation	Windows with the OpenSSH client (ssh, scp), tar and git	deploy/push.ps1
Accounts	Telegram bot from BotFather, AI provider key, see Secrets	—

Caution

push.ps1 has a default -Server value that points at the original pilot server. On any other server always pass -Server root@<server> explicitly.

Step 1. setup.sh (once, idempotent)

From the repository root on the workstation:

scp -r deploy root@<server>:/root/sticker-deploy
ssh root@<server> "bash /root/sticker-deploy/setup.sh"

The script must run as root from a folder that contains caddy/stickers.caddy and systemd/. It does the following, in this order (deploy/setup.sh):

1. apt-get update and upgrade, then installs python3-venv, python3-pip, sqlite3, ufw, fail2ban, rsync, curl, gnupg and the keyring packages.
2. Adds the official Caddy apt repository and installs caddy.
3. Creates the system user sticker (no login shell) and the directories /opt/ai-sticker-pack, bot, logs (mode 755) and bot/data (mode 750), all owned by sticker.
4. Creates the Python venv and upgrades pip. If the code is already uploaded, it installs bot/requirements.txt; otherwise push.ps1 does that later.
5. Installs the four systemd units into /etc/systemd/system/, copies backup.sh to /usr/local/sbin/sticker-backup.sh and update.sh to /usr/local/sbin/sticker-update.sh, runs daemon-reload, and enables and starts sticker-backup.timer.
6. Creates /var/log/caddy, enables caddy, and runs deploy/caddy.sh (see Caddy on a shared droplet).
7. Installs the logrotate rule /etc/logrotate.d/sticker-pack.
8. Configures ufw: deny incoming by default, allow 22, 80 and 443.
9. Configures fail2ban for sshd (5 retries within 10 minutes → 1 hour ban).
10. Runs deploy/swap.sh: a 2 GB /swapfile, an /etc/fstab entry, vm.swappiness=10.

It does not touch bot/.env and does not start sticker-bot or sticker-api. After this step Caddy already has a certificate and answers 502 because the API is not running yet. That is expected.

Step 2. Upload the code

powershell -ExecutionPolicy Bypass -File deploy\push.ps1 -Server root@<server>

Expected last lines: pip: OK, deployed: <commit>, Python 3.12.x, then a line that says the services were not restarted.

Step 3. Create bot/.env by hand

The file is never in the repository. Fill a copy of bot/.env.example locally, outside the checkout, then upload it and fix line endings, owner and mode:

scp C:\secure\sticker.env root@<server>:/opt/ai-sticker-pack/bot/.env
ssh root@<server> "sed -i 's/\r$//' /opt/ai-sticker-pack/bot/.env && chown sticker:sticker /opt/ai-sticker-pack/bot/.env && chmod 600 /opt/ai-sticker-pack/bot/.env"

• The sed removes Windows line endings (CRLF). Without it systemd keeps a carriage return at the end of every value, including the bot token.
• Comments go on their own lines only. systemd EnvironmentFile does not understand KEY=value # comment.
• Which values matter: see Configuration. Where each secret comes from: see Secrets.

Step 4. Start and enable the services

ssh root@<server>
systemctl enable --now sticker-api sticker-bot
systemctl status sticker-api sticker-bot --no-pager

With APP_ENV=prod the bot refuses to start without TELEGRAM_BOT_TOKEN, TELEGRAM_BOT_USERNAME, an AI key and ADMIN_CHAT_ID; the API refuses to start without ADMIN_PANEL_PASSWORD. The reason is in the journal as Конфигурация некорректна: … (“configuration is invalid”).

Step 5. Verify

Check	Command or screen	Expected
API, locally	curl -s localhost:8000/health	{"ok":true,"db":"ok","active_jobs":0,"maintenance":null,"version":"<commit>","render_mode":"figure","app_env":"prod"}
API through Caddy	curl -s https://stickers.eventtalks.uz/health	The same JSON
Kiosk page	Open https://stickers.eventtalks.uz/kiosk/ on the tablet	Start screen; the browser asks for camera permission during the first session
Admin panel	Open https://stickers.eventtalks.uz/admin	Browser asks for login; without credentials the answer is 401
Bot start-up lines	journalctl -u sticker-bot -n 40 --no-pager -o cat | grep -E 'STARTUP|Воркер'	STARTUP render_mode=figure (источник: /opt/ai-sticker-pack/bot/.env …), STARTUP likeness=… consistency_mode=… sheet_layout=…, STARTUP allowed_updates=[…], Воркер очереди запущен, параллельных слотов: 2
API start-up lines	journalctl -u sticker-api -n 20 --no-pager -o cat | grep STARTUP	STARTUP киоск: https://stickers.eventtalks.uz/kiosk/ health: …/health version=<commit>
Firewall	ufw status	Only 22, 80, 443
Backup timer	systemctl list-timers sticker-backup.timer	A next run is listed
Swap	swapon --show	/swapfile with size 2G

Then run one full guest pass from the tablet (Start → QR → bot → consent → outfit → photo → pack link arrives).

Step 6. Enable the leads module (optional)

Without LEADS_SECRET the module is off: POST /leads answers 404 and the bot does not handle leads. To enable it, add these lines to the server bot/.env (values: see Secrets):

LEADS_ENABLED=1
LEADS_SECRET=<generated, at least 24 characters>
LEADS_IP_SALT=<generated>
LEADS_CHAT_ID=<LEADS_CHAT_ID>
LEADS_ADMIN_IDS=<telegram_user_id>,<telegram_user_id>

1. Add the bot to the leads group and make it a group administrator. People listed in LEADS_ADMIN_IDS must be group administrators too, otherwise they do not see the command menu (commands and buttons still work).

2. Restart: bash /opt/ai-sticker-pack/deploy/update.sh --yes.

3. Check the journal: journalctl -u sticker-bot -n 60 --no-pager -o cat | grep LEADS. Expected: LEADS: бот в группе … — administrator (“bot is in the group — administrator”), LEADS: меню команд для админов группы … установлено (“command menu installed”), LEADS: публикация в чат … (“publishing to chat”).

4. Send a test lead from the server. Put ТЕСТ in the name so it can be deleted completely afterwards:

   S=$(grep '^LEADS_SECRET=' /opt/ai-sticker-pack/bot/.env | cut -d= -f2-)
   curl -s -X POST https://stickers.eventtalks.uz/leads \
     -H "Content-Type: application/json" -H "X-Stickers-Leads-Secret: $S" \
     -d "{\"request_id\":\"$(cat /proc/sys/kernel/random/uuid)\",\"name\":\"ТЕСТ Guest Name\",\"telegram\":\"@example_user\",\"phone\":\"+<digits>\",\"locale\":\"en\"}"

   Expected: {"ok":true,"lead":<N>,"duplicate":false} with HTTP 201, and a card in the group within a few seconds. Remove it in the group with /stickers_forget <N> тест.

5. Only then set LEADS_ENDPOINT and LEADS_SECRET in Vercel (see Landing on Vercel).

Swap

setup.sh already calls swap.sh. To run it separately, or with another size:

bash /opt/ai-sticker-pack/deploy/swap.sh
SWAP_SIZE=4G bash /opt/ai-sticker-pack/deploy/swap.sh   # only applies when /swapfile does not exist yet

Expected output: swap: /swapfile 2G, swappiness=10.

Adapting to a different domain

The domain is written in these places. Change all of them in one commit:

File or setting	What to change
deploy/caddy/stickers.caddy	The site address on the first block line
deploy/update.sh	DOMAIN= — used for the HTTPS health check after a restart
deploy/caddy.sh	The literal domain in main_kind (only matters when migrating an old single-site Caddyfile)
Server bot/.env	BASE_URL — used in logs and in the admin panel link of /admin_stats; must start with https://
Vercel project of the landing	LEADS_ENDPOINT = https://<new domain>/leads, then redeploy
Tests	tests/test_deploy.py asserts the domain in the Caddy file and in .env.example
DNS	A new A record for the new domain

The kiosk itself calls the API with relative paths, so it follows the domain automatically.

Routine update

One command

git commit -am "..."
powershell -ExecutionPolicy Bypass -File deploy\push.ps1 -Restart

-Restart runs update.sh --yes on the server after the upload, so it restarts without asking even when jobs are in flight. To be asked first, upload without -Restart and run update.sh yourself:

powershell -ExecutionPolicy Bypass -File deploy\push.ps1
ssh root@<server> "bash /opt/ai-sticker-pack/deploy/update.sh"

What push.ps1 checks and refuses

Flag	Effect
(none)	Upload only. Services are not restarted.
-Restart	Upload, then update.sh --yes on the server.
-AllowDirty	Proceed with uncommitted changes in the working copy. HEAD is uploaded, not your edits.
-DryRun	Build and check everything, send nothing.
-Server root@<server>	Target server.
-AppDir <path>	Target directory, default /opt/ai-sticker-pack.

The script stops with ОШИБКА: … (“error”) and a non-zero exit code when:

• the working copy has uncommitted changes and -AllowDirty is not given;
• the archive contains anything forbidden: .env, venv/ or .venv/, bot/data/, data/, logs/, sessions/, debug_sheets/, preview_*.png, figure_sheet*, anything with .bak, __pycache__/, .git/;
• any non-PowerShell file under deploy/ has a UTF-8 BOM or a carriage return (they break bash, systemd and Caddy);
• scp fails, the user sticker does not exist on the server (run setup.sh first), or pip install fails. In the last case the services are not touched.

On the server it unpacks the archive over the existing tree, writes VERSION, fixes ownership, keeps bot/.env at mode 600 and installs dependencies. Files deleted from the repository stay on the server; they do not affect the running code.

What update.sh does

deploy/update.sh runs as root:

1. Prints the deployed version. Converts CRLF in bot/.env to LF if needed and re-applies owner sticker and mode 600.
2. pip install -r bot/requirements.txt.
3. Counts jobs in queued, processing and delivering. If there are any and --yes was not given, it asks Перезапускать сейчас? [y/N] (“restart now?”). Any answer other than y cancels with отменено, сервисы не трогал (“cancelled, services untouched”).
4. Installs any systemd unit that differs from deploy/systemd/ and runs daemon-reload.
5. systemctl restart sticker-api, then systemctl restart sticker-bot.
6. Runs deploy/caddy.sh. A Caddy failure does not cancel the update; the old Caddy configuration keeps serving.
7. Polls the local /health for up to 15 seconds, then checks https://stickers.eventtalks.uz/health.
8. Prints unit state, MemoryHigh, MemoryMax, CPUWeight, MemoryPeak, the last journal lines and the last STARTUP line.

Exit code is 0 only when the local /health answered.

Note

update.sh refreshes unit files but not the copy of the backup script in /usr/local/sbin/sticker-backup.sh. After a change to deploy/backup.sh run install -m 750 /opt/ai-sticker-pack/deploy/backup.sh /usr/local/sbin/sticker-backup.sh. Always call update.sh by its path under /opt/ai-sticker-pack/deploy/; the copy in /usr/local/sbin/ dates from setup.sh.

What happens to jobs in flight

Source: bot/main.py, bot/worker.py, bot/delivery.py.

Job status at restart	What happens
queued	Stays queued. Picked up after start.
processing, finishes within 15 s	Completes normally.
processing, does not finish within 15 s	Interrupted and returned to queued with the marker [requeued on shutdown]. Generation starts over. The attempt is not charged for the first two such interruptions of a job; from the third one it is.
processing, process was killed (OOM, SIGKILL, power)	At the next start it returns to queued with [requeued after restart] and the attempt is charged. If that exhausts the 2 attempts, the job goes to failed, the guest gets the failure message and an alert is sent.
delivering	The link check stops and the delivery sweeper restarts it within 20 s after start.

The delivery deadlines are counted from the moment the photo was accepted and survive restarts: 5 minutes without a link → one delay notice to the guest; 15 minutes → the job fails. A restarted generation costs money again. When in doubt, wait for an empty queue.

systemd units

Source: sticker-bot.service, sticker-api.service, sticker-backup.service, sticker-backup.timer.

Unit	User	Command	Memory	CPU	Restart	Stop
sticker-bot.service	sticker	venv/bin/python main.py in /opt/ai-sticker-pack/bot — Telegram polling, queue worker, delivery sweeper, TTL cleanup, leads loop. Exactly one instance.	MemoryHigh=1.45G, MemoryMax=1.6G	CPUWeight=50	Restart=always, RestartSec=3	KillSignal=SIGINT, TimeoutStopSec=30
sticker-api.service	sticker	venv/bin/python -m uvicorn api:app on port 8000, loopback only, --proxy-headers — kiosk API, admin panel, /leads, /health	MemoryHigh=1.2G, MemoryMax=1.6G	CPUWeight=50	Restart=always, RestartSec=3	TimeoutStopSec=15
sticker-backup.service	root	/usr/local/sbin/sticker-backup.sh (oneshot, Nice=10, idle I/O class)	—	—	—	—
sticker-backup.timer	—	OnCalendar=*-*-* 03:30:00 UTC, RandomizedDelaySec=5min, Persistent=true	—	—	—	—

Both services read EnvironmentFile=/opt/ai-sticker-pack/bot/.env and run hardened: NoNewPrivileges, ProtectSystem=full, ProtectHome, PrivateTmp, the code tree read-only, and write access only to bot/data and logs. Above MemoryHigh the kernel throttles the process and pushes its memory to swap; MemoryMax is the RAM ceiling. CPUWeight=50 gives a neighbour with the default weight of 100 twice the CPU share under contention.

Useful commands:

systemctl is-active sticker-bot sticker-api caddy
systemctl show sticker-bot -p MemoryHigh -p MemoryMax -p CPUWeight -p MemoryPeak
systemctl restart sticker-api sticker-bot

Caddy on a shared droplet

The server may host a second project (in these pages: “the neighbouring project on the same droplet”). Both share one Caddy, so the layout is:

• /etc/caddy/Caddyfile contains only import /etc/caddy/sites/*.caddy (global options, if any, go in a { } block above that line);
• every project owns one file in /etc/caddy/sites/; ours is /etc/caddy/sites/stickers.caddy, installed from deploy/caddy/stickers.caddy.

Only deploy/caddy.sh installs our file. setup.sh and update.sh call it.

Situation	What caddy.sh does
Caddyfile already has the import line	Does not touch Caddyfile, whatever else is in it.
Caddyfile is missing, empty, the stock package file, or the old single-site file for our domain	Copies it to Caddyfile.bak-<time> and writes the single import line. Happens once.
Any other Caddyfile without the import line	Stops with an error and changes nothing. Add the import line by hand, then run it again.
Other files in /etc/caddy/sites/	Never touched.
Our site file differs from the repository	Installs it, runs caddy validate, then systemctl reload caddy.
caddy validate fails (the error may be in a neighbour’s file)	Restores our previous files and does not reload. The site keeps running on the old configuration.
Nothing changed	No validate and no reload, so it never applies a neighbour’s unfinished edit.

To change our site: edit deploy/caddy/stickers.caddy, commit, push.ps1, then update.sh (or bash /opt/ai-sticker-pack/deploy/caddy.sh). Our site block sets a 20 MB request body limit, security headers, reverse_proxy to port 8000 on the loopback interface and an access log at /var/log/caddy/stickers.log.

Landing on Vercel

The landing page (pack.eventtalks.uz) lives in the repository ai-stickers-landing and is deployed by Vercel. Source: landing README.md.

1. Import: Vercel → New Project → Import Git Repository → ai-stickers-landing. Framework preset Next.js, root directory is the repository root, default build command.

2. Node version: Settings → General → Node.js Version 22.x (package.json requires >=22).

3. Environment variables (Settings → Environment Variables, Production):

   Name	Scope	Purpose
   LEADS_ENDPOINT	server only	https://stickers.eventtalks.uz/leads
   LEADS_SECRET	server only	Same value as LEADS_SECRET in the server bot/.env
   TELEGRAM_BOT_TOKEN	server only	Bot for the direct fallback route
   TELEGRAM_CHAT_ID	server only	Leads group for the fallback route
   NEXT_PUBLIC_SITE_URL	build and client	https://pack.eventtalks.uz, no trailing slash

   Never prefix a secret with NEXT_PUBLIC_.

4. Domain: Settings → Domains → Add pack.eventtalks.uz. Vercel shows the DNS record to create: a CNAME for a subdomain. Add exactly the record Vercel shows at the DNS provider and wait for the green check. HTTPS is issued automatically.

5. Redeploy after changing NEXT_PUBLIC_SITE_URL: Deployments → … → Redeploy. The value is baked into the build (canonical links, hreflang, sitemap, OG images), so saving the variable is not enough.

6. Post-deploy checks:

   • https://pack.eventtalks.uz/ redirects to /ru or /en by browser language;
   • /robots.txt and /sitemap.xml contain the real domain, not localhost;
   • /ru/opengraph-image renders;
   • a test request from the form arrives in the leads group as a card with status buttons. A message that starts with ⚠️ Запасной канал (“fallback channel”) means the leads server did not accept it: see Leads missing or fallback.

Rollback

Pick the first row that matches. Verify with the check in the last column.

Symptom	Fastest safe rollback	Verify
A recently enabled behaviour misbehaves (faces drift, too many regenerations, skin patches, leads module)	Feature flag	STARTUP line and one test pack
Service does not start or misbehaves right after a code update	Previous commit	/health shows the old version
Database is damaged, or data was deleted by mistake	Restore the database	PRAGMA integrity_check returns ok
The whole server is broken (packages, disk, configuration)	DigitalOcean snapshot	Services active, /health ok
The landing is broken after a deploy	Previous Vercel deployment	The page loads, the form works
The form must stop depending on the leads server	Remove LEADS_SECRET on Vercel	A test request arrives as a plain message
Our Caddy site file is broken	bash /opt/ai-sticker-pack/deploy/caddy.sh restores it from the deployed repository copy	curl -s https://stickers.eventtalks.uz/health

Rollback 1: feature flag

On the server. The helper adds the line when it is missing:

E=/opt/ai-sticker-pack/bot/.env
setenv() { grep -q "^$1=" $E && sed -i "s/^$1=.*/$1=$2/" $E || echo "$1=$2" >> $E; }
setenv CONSISTENCY_MODE 0
bash /opt/ai-sticker-pack/deploy/update.sh --yes

The list of flags and what each one costs is in Configuration. Verify: journalctl -u sticker-bot -n 40 --no-pager -o cat | grep STARTUP.

Rollback 2: previous commit

On the workstation, with a clean working copy:

git log --oneline -8
git switch --detach <commit>
powershell -ExecutionPolicy Bypass -File deploy\push.ps1 -Restart
git switch main

Verify: curl -s https://stickers.eventtalks.uz/health shows "version":"<commit>".

Limits of this rollback:

• The upload unpacks over the existing tree, so files that exist only in the newer commit stay on the server.
• The database is not rolled back. Schema migrations only add columns (migrate_schema in bot/db/db.py), so older code normally runs against the newer schema. This is not tested for every pair of commits; run a test pack.

Rollback 3: restore the database

ls -lt /var/backups/sticker-pack/ | head
systemctl stop sticker-bot sticker-api
cd /opt/ai-sticker-pack/bot/data
cp -p app.db /root/app.db.before-restore-$(date +%Y%m%d-%H%M%S)
rm -f app.db-wal app.db-shm
gunzip -c /var/backups/sticker-pack/app-<time>.db.gz > app.db
chown sticker:sticker app.db
sqlite3 app.db 'PRAGMA integrity_check;'
systemctl start sticker-api sticker-bot
curl -s localhost:8000/health

Expected: ok from the integrity check and "db":"ok" from /health. Everything written after the backup is lost: jobs, packs issued, leads. Details and other copies of the database: Backups and data lifecycle.

Rollback 4: DigitalOcean snapshot

1. Take a snapshot before an event or a large update: DigitalOcean → Droplets → the droplet → Snapshots → Take snapshot. For a consistent database stop the services for the minute it takes to start: systemctl stop sticker-bot sticker-api, then start them again.
2. Restore: Droplets → the droplet → Backups/Snapshots → Restore. The droplet is overwritten with the snapshot state. The public address stays the same, DNS does not change.
3. After the restore: systemctl status sticker-api sticker-bot caddy --no-pager, then curl -s https://stickers.eventtalks.uz/health.

A snapshot restores the whole machine, including the neighbouring project on the same droplet. Agree with its owner first.

Rollback 5: landing

• Previous deployment: Vercel → Deployments → pick the last good deployment → … → Redeploy (or Promote to Production). This is the Vercel dashboard flow, not something stored in the repository. With the Vercel CLI the equivalent is vercel rollback <deployment-url> (standard platform command). Verify: the production domain serves the chosen deployment (Deployments list shows it as Current), /ru, /uz, /en load, and one test request marked «ТЕСТ» reaches the group as a card, then remove it with /stickers_forget <n> тест.
• Revert the form to direct send: Vercel → Settings → Environment Variables → remove LEADS_SECRET → Redeploy. With either LEADS_ENDPOINT or LEADS_SECRET empty, the form skips the leads server and sends the request straight to the Telegram group with TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID. Such requests are not in the database and have no status buttons.
• Server-side switch: setenv LEADS_ENABLED 0 and update.sh --yes. /leads answers 404 and the landing falls back to direct send on its own, with the ⚠️ Запасной канал marker.

Where to go next

• Configuration — which settings matter and which flags roll features back.
• Secrets — how to create and rotate every secret used above.
• Event playbook — the checklist that uses these procedures before an event.
• Runbooks — what to do when a deploy leaves something broken.
• Deploy and rollback flow — the same process as a diagram.

---

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.