Skip to content
AI Sticker Pack Docs

Deploy and rollback flow

Who is this for

Section titled “Who is this for”

Engineers who ship a change or have to undo one. This page explains the mechanics; the operator checklist is on Deploy and rollback (operations).

TL;DR

Section titled “TL;DR”
  • The product has no CI/CD. A Windows script, deploy/push.ps1, uploads an archive of the committed HEAD; a server script, deploy/update.sh, restarts the services and checks /health.
  • push.ps1 never restarts anything unless you pass -Restart. update.sh asks for confirmation when jobs are in progress, unless you pass --yes.
  • Secrets are not part of a deploy. bot/.env lives only on the server and is never in the archive.
  • The landing page deploys from Git through the hosting platform (Vercel); there is no deploy script in its repository.
  • Rollback, cheapest first: environment flag → previous commit → database backup → server snapshot. For the landing: redeploy or roll back the previous deployment in the platform.

Product deploy sequence

Section titled “Product deploy sequence”
sequenceDiagram
    autonumber
    participant Dev as Developer PC
    participant Srv as Server shell
    participant Sd as systemd
    participant Cd as Caddy
    participant Api as API health
    Dev->>Dev: git status must be clean
    Dev->>Dev: git archive HEAD
    Dev->>Dev: scan archive for forbidden paths, BOM and CR
    Dev->>Srv: scp archive
    Dev->>Srv: remote script, unpack over app dir
    Srv->>Srv: write VERSION, fix owner, chmod 600 on env file
    Srv->>Srv: pip install requirements
    Srv-->>Dev: pip OK and deployed commit
    Dev->>Srv: update.sh
    Srv->>Srv: pip install again, count active jobs
    opt jobs in progress and the yes flag was not given
        Srv-->>Dev: ask for confirmation
    end
    Srv->>Sd: install changed unit files, daemon-reload
    Srv->>Sd: restart sticker-api, then sticker-bot
    Srv->>Cd: caddy.sh, validate then reload only if our site file changed
    loop up to 15 times, 1 s apart
        Srv->>Api: GET /health on the loopback interface
    end
    Srv->>Api: GET /health over HTTPS
    Srv-->>Dev: unit status, memory limits, last log lines

Step 1 — deploy/push.ps1 (on Windows)

Section titled “Step 1 — deploy/push.ps1 (on Windows)”

Source: deploy/push.ps1.

  1. Clean tree. git status --porcelain must be empty. -AllowDirty overrides this; even then only HEAD is sent, not the uncommitted edits.
  2. Archive. git archive --format=tar.gz HEAD. Only tracked files can be in it.
  3. Forbidden-path scan. The archive listing is matched against a deny list: .env, venv/ and .venv/, bot/data/, data/, logs/, sessions/, debug_sheets/, preview_*.png, figure_sheet*, anything containing .bak, __pycache__/, .git/. Any hit aborts the upload.
  4. Line-ending scan. Every non-PowerShell file under deploy/ is checked for a UTF-8 BOM and for carriage returns, because either breaks bash, systemd and Caddy on Linux.
  5. Upload. scp to the server’s temporary directory.
  6. Remote script. Sent as base64 so the console encoding cannot damage it. It:
    • requires the sticker system user to exist (created by deploy/setup.sh);
    • unpacks the archive over the application directory. Files deleted from the repository stay on the server;
    • re-checks deploy/ for CRLF;
    • writes the short commit hash to VERSION (this is what /health reports);
    • creates bot/data (mode 750) and logs, gives everything to sticker, sets bot/.env to mode 600;
    • creates the virtualenv if missing and runs pip install -r bot/requirements.txt. A pip failure makes push.ps1 exit non-zero and no service is touched.
  7. Optional restart. With -Restart it runs update.sh --yes over SSH.

-DryRun performs steps 1–4 and builds the remote script, then stops without contacting the server.

Step 2 — deploy/update.sh (on the server, as root)

Section titled “Step 2 — deploy/update.sh (on the server, as root)”

Source: deploy/update.sh.

  1. Prints the deployed version.
  2. Converts CRLF to LF in bot/.env if needed and re-applies owner and mode 600. systemd would otherwise keep a carriage return at the end of each value.
  3. Runs pip install.
  4. Counts jobs in queued, processing and delivering with a read-only sqlite3 query. If there are any and --yes was not given, it asks Перезапускать сейчас? [y/N] (“Restart now?”). Any answer but yes exits without touching the services.
  5. Compares the four unit files in deploy/systemd/ with the installed ones; installs the changed ones and runs daemon-reload.
  6. Restarts sticker-api, then sticker-bot.
  7. Runs deploy/caddy.sh. A failure here is reported but does not fail the update: the site keeps running on the previous Caddy configuration.
  8. Polls GET /health on the loopback interface up to 15 times, one second apart, then once over https://stickers.eventtalks.uz/health.
  9. Prints unit status, the effective MemoryHigh / MemoryMax / CPUWeight / MemoryPeak, and the last log lines of both services.
  10. Exits non-zero if the local health check never answered.

What a restart does to jobs in flight

Section titled “What a restart does to jobs in flight”
Job state at restart Result
queued Untouched; picked up after start
processing, finishes within the 15 s grace period Completes normally
processing, interrupted by the graceful stop Returned to queued. The first two such interruptions of a job do not consume an attempt; later ones do. Generation starts again from the beginning, except the cached passport and anchor
processing, process killed (no graceful stop) On start: queued with one attempt consumed, or failed if that was the last attempt
delivering The sweeper resumes the link check within 20 s of start

The bot unit sends SIGINT and waits up to 30 s (TimeoutStopSec) before systemd kills the process. Details are on Reliability.

What caddy.sh guarantees on a shared server

Section titled “What caddy.sh guarantees on a shared server”

The main Caddyfile contains only an import of per-project site files. caddy.sh writes one file, sites/stickers.caddy, and nothing else. If its content is unchanged it does nothing — no validate, no reload. If caddy validate fails after a change, it restores its own previous files and does not reload. A main Caddyfile it does not recognise is left alone and reported as an error.

Landing page deploy

Section titled “Landing page deploy”
sequenceDiagram
    autonumber
    participant Dev as Developer
    participant Git as Git hosting
    participant Vc as Vercel
    participant Web as pack.eventtalks.uz
    Dev->>Dev: tsc, lint, test, build locally
    Dev->>Git: git push
    Git->>Vc: repository event
    Vc->>Vc: npm run build with project environment variables
    Vc->>Web: new deployment becomes current
  • The repository holds no deploy script and no platform configuration file. Its README.md describes importing the repository into Vercel with the Next.js preset and Node.js 22. Building and publishing on push is standard platform behaviour, not something the repository implements.
  • The landing’s CLAUDE.md lists the checks to run before a commit: npx tsc --noEmit, npm run lint, npm test, npm run build.
  • Environment variables (LEADS_ENDPOINT, LEADS_SECRET, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID, NEXT_PUBLIC_SITE_URL) are set in the platform, not in the repository. The domain pack.eventtalks.uz exists only there. A changed variable takes effect with the next deployment (standard platform behaviour; the README calls for a redeploy after changing NEXT_PUBLIC_SITE_URL).
  • Order when both sides change: deploy the server first, so POST /leads exists before the landing calls it.

Rollback options

Section titled “Rollback options”

Ordered from cheapest to heaviest. Pick the first one that fixes the problem.

# Option How Undoes Cost and limits
1 Environment flag Edit bot/.env on the server, run update.sh --yes A behaviour that has a switch One restart. Flags are read at process start only
2 Previous commit Locally git switch --detach <commit>, push.ps1 -Restart, git switch main A code change Files removed in newer commits stay on the server. The database is not rolled back
3 Database backup Stop both services, unpack a file from /var/backups/sticker-pack over bot/data/app.db, fix the owner, start the services Data damage Loses everything written after the backup (daily, 03:30 UTC). Brings back rows erased since then
4 Server snapshot Restore a snapshot in the hosting provider’s panel Anything on the server Replaces the whole server, including the co-tenant project and all data. Standard provider behaviour
5 Landing: previous deployment Promote or roll back to the previous deployment in Vercel, or revert the commit and push A landing change Standard platform behaviour
6 Landing: bypass the leads server Remove LEADS_SECRET in Vercel and redeploy Any problem in the leads path Requests go straight to Telegram, are not stored and get no status buttons

Useful flags for option 1 (all read from bot/.env):

Flag Effect
MAINTENANCE_MODE=1 Stops new guests and new jobs while you investigate
LEADS_ENABLED=0 /leads answers 404; the landing switches to its fallback; the bot stops publishing cards
CONSISTENCY_MODE=0 Figure pipeline without anchor and pack QA
LIKENESS_MODE=classic or empty Switches the face source (see Generation)
STICKER_RENDER_MODE=composite AI draws heads only; bodies come from templates
PRIMARY_AI_PROVIDER / FALLBACK_AI_PROVIDER Swap the provider order
EDGE_FLAT_CUT_CHECK=0, EDGE_ALLOW_TORSO_CROP=0, SKIN_NORM_MAX_DE=0 Turn individual quality rules off or back to their older behaviour

Notes on option 2:

  • Schema migrations only add columns and tables (db.migrate_schema, CREATE TABLE IF NOT EXISTS). Older code keeps working against a newer database file.
  • A migration that adds columns writes a copy app.db.bak-<timestamp> next to the database before it changes anything.
  • Unit files are rolled back too, because update.sh reinstalls whatever the deployed commit contains.

Notes on option 3: the exact commands are on Backups and data lifecycle.

Failure branches during a deploy

Section titled “Failure branches during a deploy”
Symptom Cause State afterwards
push.ps1 stops at “forbidden files” A secret, data file or backup is tracked by Git Nothing uploaded
push.ps1 stops at BOM/CR A file under deploy/ was saved with Windows line endings Nothing uploaded
Remote script prints pip: FAILED Dependency install failed New code is on disk, services still run the old code in memory
update.sh prints отменено, сервисы не трогал (“cancelled”) You declined the restart New code on disk, old code running
API exits right after restart config.validate found an error, for example an empty ADMIN_PANEL_PASSWORD in production systemd restarts it every 3 s; read journalctl -u sticker-api
Bot exits with STARTUP отменён (“start cancelled”) Another main.py holds the instance lock Stop the other process
https: НЕ ОТВЕЧАЕТ (“not answering”) while local health is fine Caddy or DNS problem API is up locally; see the kiosk 502 runbook

Where to go next

Section titled “Where to go next”

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.