Backups and data lifecycle

Who is this for

The engineer who must restore data or prove that photos were deleted, and the owner who answers a person’s request to delete their data.

TL;DR

Only the SQLite database is backed up: nightly at 03:30 UTC, compressed, integrity-checked, last 14 kept in /var/backups/sticker-pack/. Photos are never backed up.
The backups are on the same disk as the database. They protect against mistakes, not against losing the server. Use a DigitalOcean snapshot for that.
bot/.env is in no backup. Keep the secrets in a password manager.
Photos and derived files are deleted by the bot’s hourly cleanup no later than PHOTO_TTL_HOURS (72) after they were written. Cleanup runs only while sticker-bot runs.
Deletion on request: Сбросить гостя in the admin panel for guests, /stickers_forget <N> in the leads group for leads. Copies remain in backups for up to 14 days and in automatic app.db.bak-* files until you remove them.
After an event: keep the bot running for 72 hours, verify that no photo files remain, then remove old database copies.

What data exists

Data	Where	Personal data	Lifetime
Selfie	`/opt/ai-sticker-pack/bot/data/sessions/<session_token>/selfie_*.jpg`	Face	Deleted right after the pack is created; otherwise by TTL
Generated sheets, cut stickers	`bot/data/sessions/<session_token>/`	Stylised face	TTL
Debug sheets, anchor portrait, consistency reports	`bot/data/debug_sheets/job<N>_*`	Stylised face	TTL
Appearance description (text)	`jobs.appearance_json` in the database	Description of the face	Cleared when the job finishes; a safety pass runs hourly
Users, sessions, jobs	`bot/data/app.db`	Telegram id, first and last name, username, pack link	Until deleted on request or by you
Leads	`bot/data/app.db`	Name, Telegram username, phone, free-text comment, salted hash of the visitor address	Anonymised automatically: 30 days (spam) or 12 months (others) after the last status change
AI usage records, audit log, settings	`bot/data/app.db`	Job ids; the audit log holds Telegram ids and pack links for admin actions	Not deleted automatically
Sticker packs	Telegram	Stylised face	Never deleted by the system
Lead cards	Telegram leads group	As in the lead	Rewritten on anonymisation; never deleted by the system
Nightly backups	`/var/backups/sticker-pack/app-<time>.db.gz`	Whole database	Last 14
Automatic database copies	`bot/data/app.db.bak-<time>[-reset]`	Whole database	Never removed automatically
Application logs	`/opt/ai-sticker-pack/logs/`, systemd journal	Telegram ids, session tokens	14 days (files); system default (journal)
Caddy access log	`/var/log/caddy/stickers.log`	Client addresses	20 MiB × 14 files

Database backups

Source: deploy/backup.sh, sticker-backup.timer.

Property	Value
Schedule	Daily at 03:30 UTC, random delay up to 5 minutes, `Persistent=true` (a run missed while the server was off happens after boot)
Method	`sqlite3 .backup` — a consistent copy of the live database, safe while both services run
Check	`PRAGMA integrity_check` on the copy must return `ok`
Format and place	`/var/backups/sticker-pack/app-<YYYYmmdd-HHMMSS>.db.gz`, mode `600`, directory mode `700`
Retention	The 14 newest files
Log	`/var/log/sticker-backup.log` and the journal of `sticker-backup.service`
Runs as	root, low CPU and I/O priority
Not included	Photos, session folders, debug sheets, `bot/.env`, logs, Caddy configuration

Log lines:

Line	Meaning
`<time> UTC OK app-<time>.db.gz <size> integrity=ok jobs=N users=M удалено старых: K`	Success; “old removed: K”
`<time> UTC SKIP нет БД …`	No database yet (services never started)
`<time> UTC FAIL sqlite3 .backup вернул ошибку`	The copy failed
`<time> UTC FAIL integrity_check: … (копия оставлена для разбора: …)`	The copy is damaged; it is kept uncompressed for inspection

Commands:

tail -3 /var/log/sticker-backup.log
systemctl list-timers sticker-backup.timer
ls -lt /var/backups/sticker-pack/ | head
bash /opt/ai-sticker-pack/deploy/backup.sh          # extra backup now

A manual run counts toward the 14: each one pushes the oldest file out. A failed backup sends no alert; check the log after each event and when you change anything about the database.

Off-server copy (recommendation — not automated in the repository). The backups share a disk with the database. Before and after an event copy the latest one away and store it like a secret, because it contains personal data:

scp root@<server>:/var/backups/sticker-pack/app-<time>.db.gz C:\secure\

Automatic copies next to the database

Two operations copy the whole database first, into the same folder (bot/db/db.py, tools/reset_users.py):

Trigger	File	Notes
A schema migration at service start (new columns after a code update)	`bot/data/app.db.bak-<time>`	Logged as `Миграция БД …: добавлены колонки …, бэкап до миграции: …`. The migration is first rehearsed on a copy and verified (row counts, integrity check).
`Сбросить гостя` in the admin panel	`bot/data/app.db.bak-<time>-reset`	Made just before the deletion, so it still contains the deleted guest
`tools/reset_users.py … --yes`	`bot/data/app.db.bak-<time>`	Same

Nothing rotates these files. List them with ls -lh /opt/ai-sticker-pack/bot/data/app.db.bak-* and remove the ones you no longer need: they take disk space and hold personal data.

Restore

From a nightly backup

ls -lt /var/backups/sticker-pack/ | head
systemctl stop sticker-bot sticker-api
cd /opt/ai-sticker-pack/bot/data
cp -p app.db /root/app.db.before-restore-$(date +%Y%m%d-%H%M%S)
rm -f app.db-wal app.db-shm
gunzip -c /var/backups/sticker-pack/app-<time>.db.gz > app.db
chown sticker:sticker app.db
sqlite3 app.db 'PRAGMA integrity_check;'
sqlite3 app.db "SELECT 'jobs=' || (SELECT COUNT(*) FROM jobs) || ' users=' || (SELECT COUNT(*) FROM users);"
systemctl start sticker-api sticker-bot
curl -s localhost:8000/health

Expected: ok, counts that match the OK line of that backup in the log, and "ok":true,"db":"ok".

Removing app.db-wal and app.db-shm is a precaution: they belong to the database file you are replacing.
Everything written after the backup is gone: guests, jobs, pack links, leads, statuses, the balance override.
Jobs that were queued, processing or delivering at backup time come back in that state. Their 15-minute deadline has long passed, so the first sweep after the start fails them, messages those guests and sends you one alert per job. Maintenance mode does not prevent this; the deadlines run regardless. To avoid the messages, mark those jobs as failed before starting the services:
Terminal window
```
sqlite3 app.db "SELECT job_id, status, created_at FROM jobs WHERE status IN ('queued','processing','delivering');"
sqlite3 app.db "UPDATE jobs SET status='failed', error='restored from backup' WHERE status IN ('queued','processing','delivering');"
```
This is a manual database edit, not a tool from the repository. Use it only in this situation and only while both services are stopped.
Delete /root/app.db.before-restore-* when you are sure you do not need it.

From an automatic copy

Same procedure, with a plain copy instead of gunzip:

cp -p app.db.bak-<time> app.db

Database damaged, no usable backup

sqlite3 /opt/ai-sticker-pack/bot/data/app.db 'PRAGMA integrity_check;'

If this does not return ok and every backup is damaged as well, restore the server from a snapshot.

Droplet snapshots

A snapshot is a copy of the whole server disk at the hosting provider: code, bot/.env, database, photos on disk at that moment, and the neighbouring project on the same droplet.

Take: DigitalOcean → Droplets → the droplet → Snapshots → Take snapshot. For a consistent database stop the services while the snapshot starts (systemctl stop sticker-bot sticker-api), then start them.
When: after the final configuration before an event, and before any large update.
Restore: Droplets → the droplet → Backups/Snapshots → Restore. The public address stays; DNS does not change. Then systemctl status sticker-api sticker-bot caddy --no-pager and curl -s https://stickers.eventtalks.uz/health.
Privacy: a snapshot taken while guest photos are on disk keeps them beyond the TTL. Take event snapshots before guests arrive, and delete any snapshot taken during or shortly after an event once it is not needed (recommendation).

If the provider’s own scheduled backups are enabled for the droplet, the same privacy note applies to them. Whether they are enabled is a hosting-account setting, not something the repository controls.

Photo TTL

Source: bot/services/storage.py, periodic_cleanup in bot/main.py.

How it works:

PHOTO_TTL_HOURS must be 1 to 72; the service does not start otherwise.
The selfie is deleted immediately after the pack has been created. The TTL is the safety net.
A selfie of a failed job stays until the TTL so the job can be retried from the admin panel.
A photo rejected at upload (double tap, pack limit) is deleted at once.
The cleanup pass runs inside the bot process: once at start, then every hour. It deletes every file under bot/data/sessions/ and bot/data/debug_sheets/ whose modification time is older than PHOTO_TTL_HOURS minus one hour, then removes empty folders. The one-hour margin makes sure a file is gone no later than the TTL.
Age is measured per file, not per folder.
It runs only while sticker-bot is running. If the bot is stopped for days, nothing is deleted until it starts again.

Verify:

systemctl is-active sticker-bot
grep -h "storage cleanup" /opt/ai-sticker-pack/logs/app.log* | tail -3
# files older than 72 hours — must be 0
find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f -mmin +4320 | wc -l
# all remaining files and the oldest one
find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f | wc -l
find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort | head -3

Expected log line when something was removed: storage cleanup: удалено N файлов старше 71.0ч (“removed N files older than 71.0 h”). No line means nothing was old enough. Ошибка TTL-очистки (“TTL cleanup error”) means a pass failed; it retries in an hour, and the reason is in the traceback after it.

To delete everything now, for example before decommissioning a server (make sure no job is active):

curl -s localhost:8000/health        # "active_jobs":0
find /opt/ai-sticker-pack/bot/data/sessions /opt/ai-sticker-pack/bot/data/debug_sheets -type f -delete

Deleting a guest’s data on request

You need the guest’s Telegram id or username. Find the row in the admin panel search.

1. Collect what to remove from disk (before the reset; afterwards the rows are gone):

DB=/opt/ai-sticker-pack/bot/data/app.db
sqlite3 -readonly $DB "SELECT job_id, session_token, pack_name FROM jobs WHERE telegram_user_id=<telegram_user_id>;"
sqlite3 -readonly $DB "SELECT session_token FROM sessions WHERE telegram_user_id=<telegram_user_id>;"

2. Delete the database rows. Either in the admin panel: the guest’s row → Сбросить гостя (“reset guest”) → confirm. Or on the server:

cd /opt/ai-sticker-pack
sudo -u sticker venv/bin/python tools/reset_users.py <telegram_user_id>          # dry run: shows what would go
sudo -u sticker venv/bin/python tools/reset_users.py <telegram_user_id> --yes    # copy the database, then delete

Both remove the guest’s rows from users, sessions and jobs in one transaction. Expected output of the tool: удалено для <id>: users=1, sessions=N, jobs=M and integrity_check: ok.

3. Delete the files (or wait: the TTL removes them within 72 hours):

rm -rf /opt/ai-sticker-pack/bot/data/sessions/<session_token>
rm -f  /opt/ai-sticker-pack/bot/data/debug_sheets/job<N>_*

4. Delete the automatic copy that step 2 created, once you have checked the result:

ls -lt /opt/ai-sticker-pack/bot/data/app.db.bak-* | head -3
rm /opt/ai-sticker-pack/bot/data/app.db.bak-<time>-reset

What remains, and for how long

Remains	Why	Until
Nightly backups with the guest’s rows	Backups are not rewritten	At most 14 days (14 nightly runs)
Older `app.db.bak-*` copies	Never rotated	You delete them
Audit log entry `reset_user` with the Telegram id and the old pack links	Record of the admin action (panel reset only)	Not deleted automatically
AI usage rows for the job	Cost accounting; job id and token counts only	Not deleted
Log lines with the numeric Telegram id	Operational logs	14 days in files; journal by system default
The sticker pack in Telegram	The system never deletes packs	The guest can stop using it; the system has no delete function
Snapshots taken earlier	Whole-disk copies	You delete them

Tell the person honestly: live data is deleted now, backup copies expire within 14 days.

A reset also lets the guest make a new pack, because the one-pack limit counts their jobs. The next pack gets a new name; the old one is not overwritten.

Deleting a lead’s data on request

In the leads group, from an account in LEADS_ADMIN_IDS:

/stickers_forget <N>

Expected answer: Данные заявки #N удалены (имя, Telegram, телефон, комментарий). Из бэкапов сервера уйдут за 14 дней. В статистике заявка остаётся без контактов. (“Data of lead #N deleted (name, Telegram, phone, comment). It leaves the server backups within 14 days. The lead stays in the statistics without contacts.”)

What happens	Detail
Erased in the database	Name, Telegram username, phone, comment and the address hash, for the lead and for its repeat requests
Kept	Lead number, status and status history, language, UTM, dates — so the statistics do not change
Card in the group	Rewritten to a “data deleted” text. If Telegram refuses the edit, the log shows `LEAD_ERASE #N: карточку не переписать` and the old card stays: delete that message by hand.
Other messages	Reminder pings, replies and any `⚠️ Запасной канал` copy of the same request are not touched: delete them by hand
Without a number	`/stickers_forget` shows the latest leads as buttons
Log	`LEAD_ERASED #N по запросу, user=<id>` — no personal data

What remains: nightly backups for at most 14 days, older app.db.bak-* copies until you remove them, and any CSV export downloaded earlier (delete those files yourself).

Test leads. A lead with ТЕСТ or test in its name or comment can be removed completely, including from the statistics: /stickers_forget <N> тест. For any other lead the bot refuses the full delete.

Scheduled anonymisation of leads

The bot does the same erasure automatically, once a minute while it runs (bot/services/leads_schedule.py):

Lead status	Erased after
Spam	30 days since the last status change (or creation)
Any other	365 days since the last status change (or creation)

Log line: LEAD_ERASED #N по сроку (статус …) (“by schedule”). LEADS_CARD_PII=0 is a separate switch: it makes new cards in the group show no name, username or phone at all, while the database still stores them.

Exports

Export	How	Contains	Goes to
Leads CSV	`/stickers_export` in the leads group	All leads with contacts	A file `stickers_leads_<date>.csv` in the private chat of the admin who asked, never the group. The admin must have pressed Start in the bot once; otherwise the bot says so.
AI cost report	Workstation: `.venv\Scripts\python.exe tools\cost_report.py --ssh root@<server> --days 2 --csv out\event_cost.csv` (the `out` folder must exist)	Spend per Pacific-time day and per stage, job ids, token counts. No personal data.	Console table and optional CSV. Read-only access to the database.
Cost report on the server	`cd /opt/ai-sticker-pack && sudo -u sticker venv/bin/python tools/cost_report.py --days 2`	Same	Console
Event numbers	`/stats`, `/admin_stats`, `/stickers_stats`	Counts only	Chat
Job counts by status	`sqlite3 -readonly /opt/ai-sticker-pack/bot/data/app.db "SELECT status, COUNT(*) FROM jobs GROUP BY status;"`	Counts only	Console

There is no export of the guest list. If a client needs numbers, give counts, not names. A leads CSV is personal data: store it like a secret and delete it when the work is done.

After an event

#	Step	Command or screen	Expected
1	Keep `sticker-bot` running for 72 hours after the last guest	`systemctl is-active sticker-bot`	`active` — the TTL cleanup needs it
2	Retry or close failed jobs within those 72 hours	Admin panel → `Упал`	No failed job you still intend to retry
3	Take a database backup	`bash /opt/ai-sticker-pack/deploy/backup.sh`	`OK … integrity=ok`
4	Export what you need	Cost report; leads CSV if needed	Files stored securely
5	At 72 hours: confirm no photo files remain	The `find … \| wc -l` commands under Photo TTL	`0`
6	Remove automatic database copies	`ls -lh /opt/ai-sticker-pack/bot/data/app.db.bak-*`, then `rm` the old ones	Only copies you still need
7	Delete snapshots that contain guest photos	Hosting panel	None left from the event window
8	Decide about guest rows	Keep them (the one-pack limit and pack links keep working for the next event with the same audience) or remove individual guests on request	A conscious decision by the owner
9	Hand statistics to the client as counts	`/stats`, `/admin_stats`	No personal data leaves the system

Removing all guests at once is not a built-in operation. For a clean start at a new event with a different audience, deploy with a fresh database file (stop the services, move app.db away, start: an empty database is created) after you have backed up and exported what you need. This also removes leads, usage history and the audit log, so on a server that also serves the landing prefer per-guest resets.

Where to go next

Privacy — the rules these procedures implement.
Data erasure flow — erasure as a diagram.
Deploy and rollback — when a restore is the right rollback.
Disk and memory — when copies and logs fill the disk.
Event playbook — the after-event checklist in context.

Last verified: 2026-11-13 against ai-sticker-pack@b715418 and ai-stickers-landing@e350ff5.