pravda-jo

Runbook

Operational procedures for PRAVDA’s two operators. Read CLAUDE.md first for the conventions that shape why these steps are safe; this document is the concrete how. Every command below targets the pravda-jo Firebase project unless a step says otherwise.

1. Rotate a secret

Three secrets exist, and they do not sign the same things. Know which one you are rotating before you rotate it.

Secret Signs Rotating it…
OPERATOR_KEY Bearer auth on any route still gated by it; the fallback signer for SESSION_SECRET and TALENT_SESSION_SECRET when either is unset …closes the console to every operator session immediately, and — if SESSION_SECRET/TALENT_SESSION_SECRET are unset — signs out every talent session too.
SESSION_SECRET The pravda_ops cookie (lib/ops/auth.ts) …signs out every operator session (Ali’s and Khaled’s). Talent sessions are untouched if TALENT_SESSION_SECRET is set.
TALENT_SESSION_SECRET Talent HMAC sessions (lib/talent/auth.ts) …signs out every talent on every device. The console warns “signs every device out” next to the reissue button for the same reason — do this outside a shoot day, or tell whoever is mid-flow first.

To rotate any of the three:

firebase apphosting:secrets:set OPERATOR_KEY
firebase apphosting:secrets:set SESSION_SECRET
firebase apphosting:secrets:set TALENT_SESSION_SECRET

Generate a new value with openssl rand -base64 32. Redeploy for the change to take effect (App Hosting resolves secrets at build/start, not live).

If SESSION_SECRET and TALENT_SESSION_SECRET are both unset, rotating OPERATOR_KEY alone is effectively rotating all three at once — set both the day you read this, so a console-key rotation is never also a talent outage.

2. Reissue the Meta access token

Two clocks run on the same token, and only one of them is loud:

Steps:

  1. Business Settings → System Users → the PRAVDA system user → Generate New Token, with instagram_basic, instagram_manage_insights, pages_read_engagement, and ads_read (the Page role comes through Business Manager — without ads_read every call 403s and the error never mentions ads).
  2. Exchange it for a long-lived token: GET /oauth/access_token?grant_type=fb_exchange_token&client_id=...&client_secret=...&fb_exchange_token=<short-lived>
  3. firebase apphosting:secrets:set META_ACCESS_TOKEN, paste the long-lived token, redeploy.
  4. Confirm before relying on it: curl $SITE/api/health — read meta.expiresInDays and meta.dataAccessExpiresInDays. Both should be fresh (~60 and ~90 respectively). A meta.warning field means one of them is under 7 days; do not consider the reissue done until it is gone.

3. Reissue a talent’s passcode

Already wired end to end — /ops/talent, the reissue button on a talent’s row (components/ops/TalentManager.tsx). Under the hood: POST /api/ops/talent { action: 'reissue', id }. It signs every device that talent is currently logged in on out (sessionEpoch bump), by design — a passcode does no good if the old one still works somewhere. Tell them the new code by hand; nothing texts it automatically.

4. Re-run a read

Two ways, both live in the console today:

If both are somehow unavailable, there is no other supported path — the legacy /api/teardown route this runbook once pointed at no longer exists; it was deleted with the rest of the long-form report pipeline.

5. Create the Cloud Scheduler job for the read sweeper

One-time, per environment. The Cloud Scheduler API is not on by default in a Firebase project, so enable it first or the create fails with SERVICE_DISABLED:

gcloud services enable cloudscheduler.googleapis.com --project pravda-jo

S=$(gcloud secrets versions access latest --secret=CRON_SECRET --project pravda-jo)
gcloud scheduler jobs create http pravda-read-queue \
  --project pravda-jo \
  --location europe-west4 \
  --schedule="* * * * *" \
  --time-zone="Asia/Amman" \
  --uri="$SITE/api/cron/read" \
  --http-method=GET \
  --headers="x-pravda-cron=$S" \
  --attempt-deadline=300s \
  --max-retry-attempts=0

Reading the secret into a shell variable keeps it out of the terminal history and off the screen; it still lands in the job’s own configuration, where any project admin can read it, which is the same trust boundary Secret Manager already draws.

--location europe-west4 puts the job in the region the backend runs in. --attempt-deadline=300s is well past the sub-second this endpoint takes when the queue is empty and still leaves room for the three reads it will do when it is not. Retries are off deliberately: the job runs again in sixty seconds anyway, the work is claim-leased so a repeat is harmless rather than useful, and a retry backlog on a per-minute schedule only stacks.

Without this job, /api/cron/read is never called and the sweeper is dead code — the read guarantee (D8) depends entirely on this job existing.

Production: created 9 September 2026, pravda-read-queue in europe-west4, firing every minute and returning 200.

6. Create a secret before you declare it

apphosting.yaml resolves every secret: reference before the build starts and fails the whole rollout within seconds if one does not exist in Secret Manager with at least one version. Secret Manager also refuses an empty payload, so there is no such thing as a placeholder secret — and a placeholder value would be worse, because a present key switches its feature on (a fake Turnstile key locks the lead form). App Hosting also rejects a plain value: "" entry as “not formatted properly”. The two failed rollouts of 6 September 2026 were exactly these: first the empty values, then a secret that existed but had never been given a version.

The rule, therefore: create the secret, then add its stanza to apphosting.yaml. A channel that is not configured is absent from the file; every consumer treats an unset variable as “not configured”, and lib/config/check.ts reports it in the console. The stanzas for every channel not yet wired are kept, commented out, at the bottom of apphosting.yaml with the command that creates each secret.

To create one — the CLI prompts for the value and grants the backend access:

firebase apphosting:secrets:set OPERATOR_PHONE

For a secret the system should generate itself (SESSION_SECRET, CRON_SECRET, TALENT_SESSION_SECRET), pass a random value from a file so it is never typed or echoed:

openssl rand -base64 32 | tr -d '\n' > /tmp/s && firebase apphosting:secrets:set CRON_SECRET --data-file /tmp/s && rm /tmp/s

If a secret was created through the console rather than the CLI, grant the backend access once:

firebase apphosting:secrets:grantaccess SESSION_SECRET --backend my-web-app --location europe-west4

If the Firebase CLI refuses, check which account it is on before reauthenticating. It reports an expired credential and tells you to run firebase login --reauth, which re-authenticates the account it already has. On 10 September 2026 that account was one with no access to pravda-jo at all, so the reauth would only have traded an authentication error for a permission error. firebase login:list says who it thinks you are; the project’s owner account is the one that matters.

gcloud is the way through when that happens, since it keeps its own separate sign-in. firebase apphosting:secrets:set is a secret plus three IAM bindings, and nothing more:

printf '%s' "$VALUE" | gcloud secrets create OPERATOR_PHONE --project pravda-jo \
  --replication-policy=automatic --labels=firebase-managed=apphosting --data-file=-

then gcloud secrets add-iam-policy-binding granting roles/secretmanager.secretAccessor and roles/secretmanager.viewer to firebase-app-hosting-compute@pravda-jo.iam.gserviceaccount.com, and roles/secretmanager.secretVersionManager to service-532534291763@gcp-sa-firebaseapphosting.iam.gserviceaccount.com. Compare against an existing secret with gcloud secrets get-iam-policy CRON_SECRET before trusting it — a secret the backend cannot read fails the rollout exactly as a missing one does.

(Non-secret: variables — TELEGRAM_CHAT_ID, WHATSAPP_TEMPLATE, NEXT_PUBLIC_*, META_API_VERSION — are plain value: entries and need no Secret Manager setup; edit the file directly.)

Run node scripts/check-env.mjs (or npm run check-env) against a shell with the intended production values sourced to sanity-check what is and is not set before you deploy, not after.

7. Firestore TTL policies

Three collections carry an expiresAt field that only means something once a TTL policy is enabled — Firestore does not delete anything on its own until you run this, once, per collection:

gcloud firestore fields ttls update expiresAt --collection-group=ratelimit --enable-ttl
gcloud firestore fields ttls update expiresAt --collection-group=clients --enable-ttl
gcloud firestore fields ttls update expiresAt --collection-group=sheets --enable-ttl

The field has to be a Timestamp. A TTL policy deletes a document only when its TTL field holds a timestamp; a field holding a string, a number or null is not an error and not a warning — the sweep passes the document over in silence and it lives forever. So a policy enabled over a string-valued expiresAt looks healthy in the console and deletes nothing, which is the worst of both: a retention promise on paper and an unbounded collection in fact. The store modules write the field through lib/store/ttl.ts and read it back as an ISO string, so nothing above this line has to think about it.

Documents written before that landed carry the old string shape, and a policy will ignore every one of them. Correct them once, before or after enabling the policies — the order does not matter, since a policy only ever acts on what it can see:

node scripts/backfill-ttl-timestamps.mjs           # dry run: counts, writes nothing
node scripts/backfill-ttl-timestamps.mjs --apply   # rewrites the strings

It honours FIRESTORE_COLLECTION_PREFIX and never touches a document whose expiresAt is absent — absence is how an approved sheet and a won client say the retention clock has stopped, and stamping one would put a deletion date on a customer’s record.

Firestore’s TTL sweep is best-effort and can take up to 24 hours after a document’s expiresAt passes — verify the policy is enabled via the Firestore console’s TTL policy status page rather than waiting on a real document to prove it works. A policy reports CREATING for a few minutes before it reaches ACTIVE.

Production: the three policies were enabled on 9 September 2026 and are ACTIVE.

8. Managed daily backups + a restore drill

Firestore → Backups (console) → enable managed daily backups on the default database. No Cloud Function, no Cloud Scheduler — a native feature with its own retention window. The CLI does it too, which is repeatable:

gcloud firestore backups schedules create \
  --database='(default)' --project pravda-jo \
  --recurrence=daily --retention=14d

Point-in-time recovery is a separate switch, and the two answer different questions. A daily backup answers “restore yesterday”; PITR answers “put it back to 14:05, just before that write went out”, to any microsecond inside its window. Without it the window is one hour, which is shorter than it takes to notice most mistakes.

gcloud firestore databases update --database='(default)' --project pravda-jo --enable-pitr

Production: daily backups created 9 September 2026, fourteen days’ retention. PITR enabled the same day — the retention period went from one hour to seven days. Note that the recoverable window starts filling from the moment it is switched on, so earliestVersionTime only reaches a full seven days a week later; gcloud firestore databases describe reports where it currently stands.

Delete protection is the one guard neither backups nor PITR provide — they both live inside the database being protected — and it is on as of 9 September 2026:

gcloud firestore databases update --database='(default)' --project pravda-jo --delete-protection

The consequence to know before you meet it: (default) can no longer be deleted, by anyone or any script, until someone passes --no-delete-protection first. That is the entire point, and it is also why a teardown you actually intend will look broken for a minute. It does not touch documents or collections — deleting a client record, dropping a collection or running the itest suite is unaffected.

Run a restore drill, into a scratch project or a new database instance — never into pravda-jo directly, and record the result below. A backup nobody has ever restored is a belief, not a capability.

2026-09-09  Claude/Ali  database restore-drill-20260910 (PITR clone)  ⌷ worked

How that drill was run, since the daily backup had not taken its first snapshot yet and databases restore needs one. Point-in-time recovery gives the same proof by a different door: databases clone materialises the source as it stood at a chosen minute, into a brand-new database.

gcloud firestore databases clone --project pravda-jo \
  --source-database='projects/pravda-jo/databases/(default)' \
  --snapshot-time=2026-09-09T23:22:00Z \
  --destination-database=restore-drill-20260910

All nineteen documents across clients, sheets, ratelimit and talent came back byte for byte: same ids, same fields, same values, verified by hashing each document’s payload in both databases and diffing. The scratch database was deleted afterwards.

Three things that will cost you time if you do not know them:

Still untested: the databases restore --source-backup path, which needs a managed backup to exist. The first one is due within a day of the schedule being created on 9 September 2026; run the same comparison against it and add a second line above.

9. Staging backend

FIRESTORE_COLLECTION_PREFIX=staging_ on a second App Hosting backend sharing the same pravda-jo project — no second Firebase project needed, since firestore.rules already denies all client-side access regardless of prefix, and every store module already reads the prefix independently.

firebase apphosting:backends:create   # name it, e.g. pravda-staging

Give it its own OPERATOR_KEY/SESSION_SECRET (must differ from prod — this is the console’s isolation boundary; META_ACCESS_TOKEN/META_IG_USER_ID may be shared, since staging only ever reads public Instagram data prod would also read). Set FIRESTORE_COLLECTION_PREFIX=staging_ on the staging backend only. Seed it once from a developer machine with real ADC:

FIRESTORE_COLLECTION_PREFIX=staging_ GOOGLE_CLOUD_PROJECT=pravda-jo \
  node scripts/seed-content.mjs

staging_ and _itest_ are reserved prefixes — never let a real business’s handle collide with one (practically impossible, since handles are Instagram usernames, but worth knowing before scripting anything that takes a prefix as an argument).

10. Export a client’s data (PDPL request)

node scripts/export-client.mjs <handle>
node scripts/export-client.mjs <handle> --out export.json   # write to a file instead of stdout

Reads clients/{handle}, every sheet in its sheetTokens, its deal (by dealId, and defensively by clientHandle too), and every booking against that deal. Needs real ADC against pravda-jo (gcloud auth application-default login) and refuses to run if FIRESTORE_COLLECTION_PREFIX is set — a real PDPL request must never be quietly answered from staging or test fixtures.

Tested against the emulator with FIRESTORE_COLLECTION_PREFIX=_itest_ --allow-prefix before ever being pointed at a real handle — --allow-prefix exists for exactly that, and only that.

11. Delete a client’s data on request (PDPL request)

node scripts/delete-client.mjs <handle> --confirm

Deletes the same set export-client.mjs reads — the client document, every sheet, deal and booking it names — as one atomic batch. Refuses without the literal --confirm flag, and refuses against a prefixed environment the same way export-client.mjs does. This is permanent. Run the export first if there is any chance the operator will want their own copy of what was there.

Print the script’s own JSON output (it names what it deleted) into the operator’s own record before replying to the requester — that output is the only receipt this action leaves.

Seed script flags

scripts/seed-content.mjs: