greeneclipse/descrybe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
c9554f2f7c3e27396bde74a6bd48e30b5685f44e
descrybe/docs/production-checklist.md
T
greeneclipse 8580c996c3 Initial commit of Descrybe v2 without local scratch artifacts. 
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
2026-08-09 22:47:43 +02:00

25 KiB
Raw Blame History

Production checklist (Descrybe v2)

Actionable go-live checklist for Descrybe v2 (Go API + SvelteKit + PostgreSQL). Check every box before cutting traffic.

Migrator note: Schema migrations use goose on apps/api/sql/schema/*.sql. Do not use Drizzle Kit here — that belongs to the legacy Next.js repo (descrybe).

Tool Role How
goose Apply Postgres schema make migrate / .\scripts\migrate.ps1 → goose -dir sql/schema postgres "$DATABASE_URL" up
sqlc Generate Go query code after schema changes run as part of migrate scripts
cmd/migrator One-time MySQL → Postgres data cutover separate from goose; see cutover.md
drizzle-kit N/A for v2 legacy repo only

Pinned invoke (scripts): github.com/pressly/goose/v3/cmd/goose@v3.24.3.

Related docs: production-readiness.md, ops-runtime.md, go-live-checklist.md, cutover.md.

Scorecard (2026-08-09): Product (code) 10/10 · Cutover (ops) NO-GO / 9/10 (hard live left: SMTP + Stripe; Clerk removed from product — not a cutover blocker) — see production-readiness.md. Product evidence includes mobile browser PASS and npm run check GREEN (7c018ec6). Unchecked OPS OPEN boxes do not reduce Product.

Checkbox legend

Mark Meaning
[x] CODE DONE Shipped in repo (tooling/API/UI/schema/examples). Not an ops clear.
[x] LOCAL DONE Proven on local stack / dry-run rehearsal only. Not a production-host clear.
[ ] OPS OPEN Operator must run/prove on the target/cutover env (or live SMTP/Stripe).

Still OPS OPEN (do not check off here): live SMTP (#4), live Stripe+webhooks (#5). Clerk is removed from the product — #1 @legacy.local hygiene is optional data cleanup, not a hard NO-GO.

Operator NO-GO gate (do not duplicate): sequence the 10 cutover blockers with dry-run commands in cutover.md — NO-GO blockers gate. Canonical blocker list: production-readiness.md. This checklist keeps deploy/orphan/metrics/edge detail (§§0–1 / 1b / 1d / 1f) plus shipped cutover tooling below (§§T / 1e / 4b / 6).

T. Shipped cutover tooling (code vs ops)

Continuation cutover-prep wave shipped operator tooling only. Presence of these boxes as [x] does not clear DNS/cutover.

T1. Forgot-password (P1-18)

Design: forgot-password.md.

  • CODE DONE: Goose 041_password_reset_tokens + POST /api/auth/forgot-password + POST /api/auth/reset-password + /forgot-password / /reset-password (i18n); reuses mail.Send + EMAIL_DRY_RUN.
  • CODE DONE: Admin set-password re-issue remains for migration cohort (must_set_password / migrator invites) — not replaced by forgot-password.
  • OPS OPEN: Goose through 042 applied on the production host (see §1 / §4).
  • OPS OPEN: Live SMTP + one forgot-password → reset → login smoke (dry-run / EMAIL_DRY_RUN first; cutover blocker #4).

T2. Migrator operator flags

Full flag table: cutover.md — Migrator flags. Sequenced dry-runs: NO-GO gate.

  • CODE DONE: @legacy.local hygiene — -list-legacy-emails / -export-legacy-emails / -patch-emails (-dry-run or -confirm; never overwrites real emails).
  • CODE DONE: Membership promote — -list-member-memberships / -promote-company-admins (-dry-run or -confirm; unscoped promote refused; never promotes A1 a1=true; -company-id alone scopes).
  • CODE DONE: Plan gaps — -list-companies-without-plans / -assign-missing-plans + -plan-name (-dry-run or -confirm); ETL -fallback-plan-name.
  • CODE DONE: Set-password / demo helpers — -issue-set-password-invites, -set-password, -ensure-demo; offline -fixture dry-run (not a substitute for MySQL dry-run).
  • LOCAL DONE: Promote + plan-gap list/dry-run rehearsal on local Postgres (A1 promote skipped; Demo already admin via seed-demo; assign -confirm not claimed on fixture-heavy local DBs). Script: node scripts/cutover-local-rehearsal.mjs / make cutover-rehearsal.
  • CODE DONE: Clerk removed from product auth — no live Clerk dependency. Optional @legacy.local hygiene tooling remains for residual rows (not a hard NO-GO).
  • OPS OPEN: Cutover-DB promote/assign -confirm after reviewed dry-run (do not invent DSNs / mutate A1). Live SMTP/Stripe stay #4–#5.

T3. Admin diagnostics (presence flags)

  • CODE DONE: GET /api/admin/diagnostics + /admin/diagnostics — Stripe/mail presence only (no secrets/DSNs/keys in payload).
  • CODE DONE: Useful config.* / check fields: smtp_enabled, email_dry_run, smtp_host_set, stripe_secret_set, stripe_webhook_secret_set, stripe_mock, stripe_mock_rejected_in_prod; check stripe.secret_key_set / webhook_secret_set / mock.
  • OPS OPEN: Platform admin opens /admin/diagnostics on target env; flags match intended prod config (live SMTP/Stripe proof still cutover #4–#5).

0. Web adapter / deploy shape (code-decided)

Decided: @sveltejs/adapter-node@5.5.7 (apps/web/svelte.config.js). Not adapter-static — SSR hooks and server routes are in use (hooks.server.ts, +page.server.ts, +server.ts). Details: production-readiness.md.

  • Adapter pinned in repo (adapter-node; adapter-auto removed).
  • CODE DONE: Production start/build scripts — apps/web build + start (node build); root npm run build (web+api) and npm run start:web.
  • LOCAL DONE: npm run cutover:deploy-check / node scripts/cutover-deploy-check.mjs with HEALTH_BASE_URL=http://127.0.0.1:28471 — goose 039–042, /readyz worker ok, adapter-node host gates (local stack only).
  • OPS OPEN: Production host runs the Node build output (npm run start:web / node build in apps/web) — not a static CDN-only site.
  • OPS OPEN: Reverse proxy serves dashboard WEB_ORIGIN to that Node process; API remains on its own origin/PUBLIC_API_URL.
  • OPS OPEN: Web process gets SIGTERM drain comparable to API (rolling deploys).

Build + start (host):

npm run build --workspace=web   # or: npm run build (web + go build ./...)
npm run start:web               # node build (adapter-node output); requires prior build

Operator check (read-only, no infra changes): after schema+worker are up on the target env:

npm run cutover:deploy-check
# equivalent: node scripts/cutover-deploy-check.mjs
# code-only (no DB/API): npm run cutover:deploy-check:code
# optional on node invoke: --skip-goose | --skip-readyz
# optional: HEALTH_BASE_URL=http://127.0.0.1:28471 (overrides PUBLIC_API_URL for /readyz)

This verifies goose 039–042 applied, /readyz checks.worker=ok, and adapter-node host gates (pin, svelte.config.js import, build/start scripts). Host process deploy remains ops.

1. Health / readiness probes

Probe Path Expect Auth
Liveness GET /healthz 200 { "status": "ok", … } none
Readiness GET /readyz 200 { "status": "ready", … } or 503 if DB down or worker heartbeat missing/stale (>60s) none

/readyz checks Postgres ping and worker_heartbeats for worker_id processing (jobs.DefaultHeartbeatStaleAfter = 60s). Payload includes checks.database, checks.worker, checks.queue, and worker_last_seen_age_s when a heartbeat row exists (on-call triage field — see §1d).

Local / API-only: running cmd/api without cmd/worker → /readyz returns 503 (worker heartbeat missing or stale). That is expected — not a false healthz failure (/healthz stays 200).

Deploy order (required):

  1. goose up through 042_user_session_version (includes 039_worker_heartbeats + 040_job_hotpath_indexes — heartbeat table must exist before worker touch / API probe; 041 for self-serve forgot-password; 042 for cookie session_version revoke-on-reset).
  2. Restart worker (cmd/worker) so it upserts heartbeats.
  3. Then expect /readyz 200 (API restart alone is not enough if the table is new or the worker was never started).
  • Load balancer / k8s liveness → /healthz (no DB / no worker).
  • Load balancer / k8s readiness → /readyz (DB + fresh worker heartbeat ≤60s; 503 if pool missing, ping fails, or worker missing/stale).
  • Probes stay reachable under MAINTENANCE_MODE (middleware short-circuits only non-probe traffic).
  • Smoke from outside the VPC (API and worker running):
curl -sS -o /dev/null -w "%{http_code}\n" https://YOUR_API_HOST/healthz
curl -sS https://YOUR_API_HOST/readyz
make health

1b. Orphan processed cleanup (one-shot ops)

Sibling 97a2be08 shipped admin report/cleanup. After ETL/cutover data is loaded, platform admins run once on the cutover DB:

  1. Open /admin/orphan-processed (Ops nav) — loads the report (no delete). Or GET /api/admin/jobs/orphan-processed.
  2. Review counts/samples. Optional: Dry-run cleanup (POST without confirm) to re-check without deleting.
  3. If orphans > 0: Confirm delete… in the UI (sends {"confirm":true}), or POST /api/admin/jobs/orphan-processed-cleanup with confirm=true. Without confirm, POST is report-only.
  • CODE DONE: Admin report/cleanup API + UI (/admin/orphan-processed; dry-run vs confirm=true).
  • LOCAL DONE: Report-only / dry-run path on local stack (cutover-local-rehearsal.mjs orphan step — never passes confirm=true by default).
  • OPS OPEN: Report run on cutover DB; samples reviewed.
  • OPS OPEN: Confirmed cleanup on cutover DB if needed; re-report shows zero (or accepted residual).
  • Understood: this does not by itself make production GO (and does not clear SMTP/Stripe).

1c. Durable feed sync (worker-backed)

Sibling 014e2507: sync is durable via feed_sync_jobs + cmd/worker claim (FOR UPDATE SKIP LOCKED). Dashboard: 202 on POST /api/feeds/{id}/sync, then poll GET /api/feeds/{id}/sync-jobs/{jobID} (apps/web pollFeedSyncJob). API does not run unbound sync goroutines.

  • Worker process running (same requirement as /readyz heartbeat).
  • Smoke: trigger feed sync → 202 → job reaches completed / terminal via poll (or Sync History).
  • LISTEN wake + stuck running reclaim + same-feed pending dedupe shipped (2f1019f2 / 014e2507) — not open residuals. Worker still required for completion; load soak / multi-replica edge caps remain Cutover/ops (not Product code gaps).

1d. Metrics scrape + alerts (cutover)

Product code ships gated Prometheus text at GET /metrics (metrics.Gate: loopback or METRICS_PUBLIC=1 in production). Example scrape + alert files are CODE DONE under deploy/prometheus/; wiring live targets / Alertmanager remains OPS OPEN. Details + starter PromQL: production-readiness.md.

Topology: scrape API on HTTP_ADDR for HTTP RED; scrape worker on METRICS_ADDR (e.g. 127.0.0.1:9091) for sync series. Deny /metrics on the public VIP. Keep METRICS_PUBLIC unset unless scraping a private VIP.

Job Target Series
descrybe-api API private/mesh :HTTP_ADDR /metrics http_requests_total, http_request_duration_seconds
descrybe-worker Worker METRICS_ADDR /metrics sync_duration_seconds, sync_failures_total
descrybe-readyz blackbox → private GET /readyz probe_success (worker missing/stale → HTTP 503; age not a Prom series)

On-call worker freshness (not a Prom series): /metrics does not expose heartbeat age. Triage via GET /readyz: checks.worker (ok / missing / stale) and worker_last_seen_age_s when a heartbeat row exists (seconds since last processing touch; stale after 60s = jobs.DefaultHeartbeatStaleAfter). Same age field is printed by node scripts/cutover-deploy-check.mjs. Probe details: §1 above. Example alert: DescrybeWorkerHeartbeatStale in alerts.example.yml.

  • CODE DONE: Gated GET /metrics (API + optional worker METRICS_ADDR) + Gate unit coverage.
  • CODE DONE: Example scrape + alert rules (no secrets): deploy/prometheus/scrape.example.yml, deploy/prometheus/alerts.example.yml, deploy/prometheus/README.md.
  • OPS OPEN: Prometheus (or equivalent) scrapes API /metrics from loopback/sidecar/mesh — not the public internet (replace placeholder targets).
  • OPS OPEN: Worker has METRICS_ADDR bound privately; job descrybe-worker scrapes it (sync kinds: feed, woocommerce*, shopify*).
  • OPS OPEN: Public edge denies /metrics (defense-in-depth even if Gate holds).
  • OPS OPEN: Alerts loaded + routed (tune thresholds): API 5xx rate, API p99 latency, sync_failures_total increase, sync p95 duration, scrape up==0, plus blackbox/HTTP probe on /readyz (worker missing/stale >60s).
  • OPS OPEN: On-call can open /metrics via private path only; no scrape secrets committed.
  • OPS OPEN: On-call runbook uses /readyz worker_last_seen_age_s + checks.worker for worker-down triage (not /metrics).

1e. Admin diagnostics smoke (ops)

CODE DONE surface is §T3. This section is OPS OPEN only.

  • Platform session: open /admin/diagnostics (or GET /api/admin/diagnostics) — overall status + checks load.
  • Confirm mail presence: config.smtp_enabled / email_dry_run / smtp_host_set match host intent (no secret values visible).
  • Confirm Stripe presence: config.stripe_secret_set / stripe_webhook_secret_set / stripe_mock (prod expects mock off + secrets set when billing on).
  • Understood: green presence flags ≠ live Checkout/webhook or live mail delivery proof.

1f. Multi-replica edge rate limits (cutover #8)

CODE DONE: in-process HTTP limiters + RATE_LIMIT_REPLICAS ceil-division (apps/api/internal/httpapi/ratelimit.go). No Redis shared store in-repo (RATE_LIMIT_BACKEND forced to memory). Snippets: deploy/examples/edge-rate-limit.md. Runbook: cutover.md §8.

Env contract (when API replicas N>1):

Var Value Notes
RATE_LIMIT_REPLICAS N Divides HTTP middleware only; not lockout / StartLimiter / AI / email
RATE_LIMIT_MULTI_REPLICA true Ops ack; boots with warn
TRUSTED_PROXIES hop-1 edge CIDRs/IPs Required so IP RPM keys on client, not proxy
RATE_LIMIT_BACKEND memory (effective) redis/postgres requested → warn + memory
  • CODE DONE: rateLimitEffectiveCap + boot warn (ShouldWarnRateLimits / RateLimitWarningMessage).
  • CODE DONE: Edge nginx/Caddy example with shared zones (edge-rate-limit.md).
  • OPS OPEN: If N=1, leave defaults — edge optional.
  • OPS OPEN: If N>1, set env contract on every API replica + apply one shared edge config in front of all replicas.
  • OPS OPEN: Confirm boot warn appears; spot-burst auth/public and expect edge and/or app 429 (not silent N× budget).
  • Understood: RATE_LIMIT_REPLICAS alone is not a substitute for edge hard global RPM.

2. Graceful shutdown / process config

  • API (cmd/api) receives SIGTERM / SIGINT and drains with Shutdown (~10s timeout).
  • Worker (cmd/worker) uses signal.NotifyContext and stops claiming new jobs on cancel.
  • Orchestrator drain window ≥ shutdown timeout.
  • UPLOAD_DIR / feed artifacts on persistent disk (or object storage), not ephemeral container FS only.
  • Behind a LB: idle timeout lower than server keep-alive (or raise keep-alive) to avoid 502s on rolling deploys.

3. Logging without PII

  • API uses log/slog JSON via internal/logredact.NewJSONHandler (emails, Bearer tokens, api_key= / secrets, Stripe sk_* / whsec_, DB URLs with credentials → [REDACTED]).
  • Worker / stdlib log output is wrapped with logredact.Writer.
  • Never log raw Stripe webhook bodies, Woo/Shopify tokens, full Authorization headers, or marketing recipient addresses.
  • Production log level stays info (or quieter); avoid debug dumps of request bodies.
  • Spot-check after deploy: no emails/API keys in fresh log lines.
cd apps/api && go test ./internal/logredact/ ./internal/httpapi/ -count=1

4. Database schema migrations (goose)

Apply schema with goose only:

# Git Bash / WSL / macOS / Linux
export DATABASE_URL='postgres://…'
make migrate

# Windows PowerShell
$env:DATABASE_URL = 'postgres://…'
.\scripts\migrate.ps1

# Equivalent direct invoke (apps/api)
go run github.com/pressly/goose/v3/cmd/goose@v3.24.3 \
  -dir sql/schema postgres "$DATABASE_URL" up

SQL files under apps/api/sql/schema/ (goose +goose Up / +goose Down):

File Purpose
001_platform.sql Companies, users, sessions, memberships, plans
002_catalog.sql Categories, attributes, products
003_feeds.sql Feeds / export feeds
004_processing.sql Processing jobs / costs
005_woocommerce.sql Woo configs
006_feed_sync.sql Feed sync columns
007_standard_fields.sql / 008_… Standard fields
009_processing_step_progress.sql Step progress
010_woo_orders_reviews.sql Orders / reviews
011_email_campaigns.sql … 014_… Email / unsub
015_ai_providers.sql AI BYOK providers
016_stripe_billing.sql Stripe billing
017_shopify.sql Shopify connector
018_… … 038_… Indexes, support, sales, i18n, etc.
039_worker_heartbeats.sql Worker heartbeat rows for /readyz (sibling 63ee62c5)
040_job_hotpath_indexes.sql Claim/list composites for feed_sync_jobs / processing_jobs / processing_job_products
041_password_reset_tokens.sql Self-serve forgot-password reset tokens (password_reset_tokens)
042_user_session_version.sql users.session_version for revoke-on-password-reset cookie sessions
  • CODE DONE: Goose migrations through 042_user_session_version.sql exist in repo (incl. 039 / 040 / 041); one-command apply npm run migrate.
  • CODE DONE: Local/dev starts API+worker together (npm run dev / npm run dev:backend) so /readyz can green.
  • LOCAL DONE: Local goose through 042 + worker heartbeat + cutover-deploy-check PASS (see §0).
  • OPS OPEN: Backup DB before migrate.
  • OPS OPEN: Run goose against the production DATABASE_URL in a maintenance window (or verify already applied).
  • OPS OPEN: Confirm goose version table includes versions through 042.
  • OPS OPEN: After 039/040/041/042: restart worker, then confirm /readyz → 200 (API-only without worker stays 503).
  • After schema changes, regenerate sqlc (make migrate / migrate scripts do this).
  • Do not run drizzle-kit against this database.
  • MySQL → PG data load is cmd/migrator (separate checklist in cutover.md) — not a substitute for goose.

4b. Migrator data cutover (ops; tooling CODE/LOCAL)

Operator flags are CODE DONE (§T2); local list/dry-run rehearsal is LOCAL DONE. Cutover-DB execution + SMTP stay OPS OPEN — full commands in cutover.md.

  • CODE DONE / LOCAL DONE: Offline fixture dry-run + local list/dry-run inventory (promote/plans; no blind -confirm) — §T2.
  • OPS OPEN: MySQL dry-run with real MIGRATE_MYSQL_DSN (report + id-map; no migrated row writes).
  • OPS OPEN: Live load after freeze + reviewed dry-run.
  • OPS OPEN: Post-load: optional @legacy.local hygiene (not hard NO-GO) → #2 promote admins → #3 assign missing plans (dry-run then -confirm).
  • OPS OPEN: Set-password invites re-issued after email repair (-issue-set-password-invites / mailhooks) — live SMTP is #4.

5. Environment (.env.example)

  • Copy root .env.example → host secret store / local untracked root .env only. Do not maintain a second apps/api/.env.
  • Bootstrap (required for a working process): DATABASE_URL, APP_ENV, HTTP_ADDR, WEB_ORIGIN, PUBLIC_API_URL, SESSION_SECURE (production), APP_ENCRYPTION_KEY (or CREDENTIALS_ENCRYPTION_KEY), TOKEN_SIGNING_SECRET. Local defaults: README.md.
  • Stripe / EPREL / feed private-URL allowlist: configure in /admin/settings (GET/PUT /api/admin/settings → values.* such as stripe.secret_key, eprel.*, feeds.private_url_allowlist). Process env is an optional fallback — production does not require Stripe secrets at boot.
  • CODE DONE: Mail send path for platform invites + forgot-password (internal/mail, EMAIL_DRY_RUN / ApplyDryRun).
  • OPS OPEN: Platform invite / forgot-password SMTP — SMTP_* or platform admin mail settings for live sends; prove delivery (cutover #4) — see ops-runtime.md.
  • Do not put day-to-day product secrets in .env. Configure in the dashboard after login:
    • Platform Stripe / EPREL / feed allowlist → /admin/settings
    • AI → /integrations/ai
    • Marketing email → /integrations/email
    • Stores → /stores (and Woo/Shopify sibling routes)
  • Secrets from vault/CI only — never commit real values. No tmp placeholder env files.

6. Auth & public surface

  • CODE DONE: Self-serve forgot-password / reset (goose 041 + API/UI) — §T1.
  • OPS OPEN: Dashboard CSRF + session cookies secure on HTTPS.
  • OPS OPEN: Forgot-password live mail + reset login smoke (after SMTP proven).
  • /api/v1/* uses API keys; Stripe webhooks verify signatures when secret set.
  • Export-feed public URLs intentional; rotate leaked feed IDs if needed.
  • SSRF guards enabled for feed / Woo / Shopify HTTP clients.

7. Post-deploy smoke

  • GET /healthz → 200
  • GET /readyz → 200 with status=ready and checks.worker=ok (worker process running; not API-only)
  • Login works against prod session config
  • OPS OPEN: Forgot-password smoke (if SMTP live): request reset → email → set password → login
  • OPS OPEN: Admin diagnostics presence flags look right (§1e) — no secrets in payload
  • One product process job + one export feed generate
  • One feed sync: 202 + poll to terminal status (worker claiming feed_sync_jobs)
  • Worker claiming jobs (processing / feed sync / Woo / billing tick)
  • Orphan-processed report (+ cleanup if needed) — section 1b
  • Stripe webhook reachable (if billing on)
  • No emails/API keys in fresh log lines
cd apps/api && go build ./... && go test ./...
curl -sf https://YOUR_API_HOST/healthz && curl -sf https://YOUR_API_HOST/readyz

8. Rollback

  • Previous API/web image tags known and redeployable.
  • DB backup from pre-migrate retained.
  • Feature / env kills: stripe.mock (admin settings) / STRIPE_MOCK, EMAIL_DRY_RUN, MAINTENANCE_MODE, READ_ONLY_MODE understood by on-call.

Quick owner commands

# Combined cutover deploy gates (goose 039–042 + /readyz worker + adapter-node host gates)
npm run cutover:deploy-check
# equivalent: node scripts/cutover-deploy-check.mjs
# code-only adapter/scripts gate: npm run cutover:deploy-check:code

# Local dry-run rehearsal (no SMTP / Stripe / -confirm writes)
node scripts/cutover-local-rehearsal.mjs
# or: make cutover-rehearsal

# Schema migrator status only (goose — not Drizzle)
cd apps/api
go run github.com/pressly/goose/v3/cmd/goose@v3.24.3 -dir sql/schema postgres "$DATABASE_URL" status

# Probes (local; npm run health also works)
curl -sS http://127.0.0.1:28471/healthz
curl -sS http://127.0.0.1:28471/readyz

# Admin diagnostics presence flags (platform session cookie / API key as used in env)
# curl -sS -H "Cookie: …" http://127.0.0.1:28471/api/admin/diagnostics | head

# Migrator read-only inventory examples (OPS OPEN; from apps/api)
# go run ./cmd/migrator -list-legacy-emails -postgres "$DATABASE_URL"
# go run ./cmd/migrator -list-companies-without-plans -postgres "$DATABASE_URL"
# go run ./cmd/migrator -list-member-memberships -postgres "$DATABASE_URL"

# Redaction + health unit tests
go test ./internal/logredact/ ./internal/httpapi/ -count=1
Reference in New Issue View Git Blame Copy Permalink