descrybe/docs/ops-runtime.md

Runtime ops notes (WS7)

Local/dev and production operator notes for platform, mail, billing, and WooCommerce schedules. Do not commit secrets.

Env file policy: one root .env (or secrets mapped into process env). Do not duplicate into apps/api/.env or tmp placeholders. Platform Stripe / EPREL / feed allowlist → /admin/settings; tenant OpenAI / marketing email / stores → /integrations/ai, /integrations/email, /stores — see README.md.

Health

• GET /healthz — liveness (no DB)
• GET /readyz — readiness: Postgres ping and a fresh worker heartbeat (worker_id=processing, stale after 60s) plus queue probe; returns maintenance / read_only / hypercare flags and checks.{database,worker,queue}
• GET /metrics — Prometheus text (HTTP RED on API; worker sync series when METRICS_ADDR is set). Production Gate: loopback or METRICS_PUBLIC=1. Example scrape + alerts: deploy/prometheus/. Worker age for on-call is /readyz worker_last_seen_age_s (not a Prom series).

When the worker is down or stale, /readyz is 503 with short error plus operator reason (no secrets). Example (API-only / stale heartbeat):

{
  "status": "not_ready",
  "service": "api",
  "checks": { "database": "ok", "worker": "stale", "queue": "ok" },
  "error": "worker heartbeat stale",
  "reason": "Processing worker heartbeat older than 1m0s. API-only readiness 503 is expected — start or restart the worker (npm run dev includes it, or npm run dev:worker).",
  "queue_pending": 5,
  "worker_last_seen_age_s": 120
}

Prefer one stack: npm run dev runs api + web + worker (ports 28471 / 28472). Compose starts Postgres only — the worker is a host process. Browsers need the web process — API-only leaves :28472 CONNECTION_REFUSED even when /readyz is green.

If the API is already up without a worker (npm run dev:api), start one worker only (or use npm run dev:backend for api+worker):

npm run dev:worker
# equivalent: cd apps/api && go run ./cmd/worker
# api+worker (no web): npm run dev:backend

Do not start a second worker when checks.worker=ok / worker_last_seen_age_s is fresh — duplicate claim loops contend on A1 (or any tenant) job rows. Before restarting the full stack: node scripts/free-dev-ports.mjs (or rely on predev) then npm run dev, and stop any leftover standalone cmd/worker. Readiness itself is read-only (no A1 catalog writes). API-only → /readyz 503 is expected; see README troubleshooting.

Cutover rehearsal: probes must stay green while MAINTENANCE_MODE / READ_ONLY_MODE may block app traffic.

Postgres pgx pool (API + worker)

Shared by cmd/api and cmd/worker via internal/db.NewPool. Size pools for your Postgres max_connections and replica count — do not copy high defaults blindly.

Variable	Default	Purpose
DB_MAX_CONNS	20	Hard ceiling per process (MaxConns)
DB_MIN_CONNS	2	Warm floor (MinConns, ~10–30% of max)
DB_MAX_CONN_LIFETIME	1h	Recycle connections before server-side idle kills / DNS drift
DB_MAX_CONN_LIFETIME_JITTER	6m	Random extra lifetime (~10% of lifetime) — avoids thundering-herd reconnects; 0 disables
DB_MAX_CONN_IDLE_TIME	5m	Close idle conns during health checks
DB_HEALTH_CHECK_PERIOD	1m	Background idle health check interval
DB_STATEMENT_TIMEOUT	30s	Postgres statement_timeout GUC per connection; 0 disables

Formula sketch: MaxConns ≈ (max_connections − reserved) / instance_count. Typical per-process range is 20–50. Always keep jitter set in multi-instance deploys.

SMTP (invites + set-password)

Platform invite / set-password mail uses process env (internal/mail + cmd/mailhooks). This is separate from tenant marketing email (/integrations/email).

Env (no defaults that embed secrets):

Variable	Purpose
SMTP_ENABLED	true to send; otherwise no-op log (subject only, no PII)
SMTP_HOST / SMTP_PORT	SMTP server
SMTP_USER / SMTP_PASSWORD	Auth (optional for open relays)
SMTP_FROM	From address
WEB_ORIGIN	Base URL for accept-invite links
TOKEN_SIGNING_SECRET	HMAC for admin-issued set-password tokens (not migrator hooks)
EMAIL_DRY_RUN	Default true when unset (safe). Live mailhooks send fails closed until you pass -dry-run or set EMAIL_DRY_RUN=false (and/or disable dry-run in admin platform mail settings).

Migrator hooks → mail

After a live migrator run (or go run ./cmd/migrator -issue-set-password-invites), maps-dir contains password_invites.json and set-password-hooks.json (invite tokens + URLs for must_set_password users). See migration-readiness.md.

Operator smoke (no live SMTP)

Rehearse under dry-run before flipping SMTP. Prefer process env EMAIL_DRY_RUN=true (or leave unset — default is dry-run):

cd apps/api
# Expect exit 1: fail-closed when dry-run is on and -dry-run is omitted
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json

# Expect exit 0: counts subjects only; no SMTP dial
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json -dry-run
# log: mailhooks: dry-run subject="Set your Descrybe password"
# stdout: mailhooks: sent=N skipped=… failed=0 smtp_enabled=… total=…

Pass criteria: second command exits 0, failed=0, and subjects are logged without contacting an SMTP host. Unit gate: go test ./cmd/mailhooks/.

Live send (after SMTP proven)

cd apps/api
# Requires EMAIL_DRY_RUN=false (or admin platform mail dry-run off) + SMTP_ENABLED=true + host/from
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json

Rate limit via -delay-ms (default 100). Team invites use the same SMTP path from POST /api/team/invites.

Session idle policy

• Absolute session lifetime: 7 days
• Idle timeout: SESSION_IDLE_HOURS (default 24)
• Set SESSION_SECURE=true behind HTTPS in production

Credentials encryption (WooCommerce)

Variable	Purpose
CREDENTIALS_ENCRYPTION_KEY	Required in production for AES-GCM at-rest encryption of Woo consumer secrets

If unset, the API/worker derive a key from TOKEN_SIGNING_SECRET + DATABASE_URL. That derived key is local/dev only — rotate by setting an explicit CREDENTIALS_ENCRYPTION_KEY before storing production Woo credentials. Changing the key without re-saving configs makes existing ciphertext unreadable.

AI (OpenAI-compatible / Green Chat)

Preferred: company admin configures the provider in /integrations/ai (popular BYOK or custom OpenAI-compatible base URL + key). Encrypted at rest with APP_ENCRYPTION_KEY.

Optional process-env platform fallback (when the company leaves mode internal and a platform key is present in config):

Variable	Purpose
OPENAI_API_KEY	Bearer token; non-empty enables the platform Completer fallback
OPENAI_BASE_URL	Default https://api.openai.com/v1; LAN Green Chat e.g. http://HOST:PORT/v1
OPENAI_MODEL	Chat model id (GET /v1/models)

If set, put these in the root .env only — not apps/api/.env. Shared by worker (processing enhance) and API (campaign + SEO AI). Brand kit injects into those prompts. See green-chat-llm.md.

[UNCERTAIN] Other agents may remove env-based platform OpenAI in favor of dashboard-only configuration; prefer /integrations/ai for new setups.

Billing / processing costs

• Debit per processed product: processing_costs.product_processing + ceil(tokens/1000) * openai_token_k
• Defaults seeded by worker/EnsureDefaultCosts
• Plans admin / assign / add-credits: /api/admin/* (platform admin only)
• Billing cycles: worker every 15m via RunDueBillingCycles

EPREL energy labels

Optional enrichment during product processing. See eprel.md.

Preferred: platform admin → /admin/settings (values.eprel.enabled, eprel.base_url, eprel.timeout, eprel.fiche_language, eprel.api_key). Process EPREL_* env is an optional fallback.

Settings key / env	Purpose
eprel.enabled / EPREL_ENABLED	true to fetch label/fiche/class after AI steps
eprel.timeout / EPREL_TIMEOUT	HTTP timeout (default 10s)
eprel.fiche_language / EPREL_FICHE_LANGUAGE	Fiche PDF language (default EN)
eprel.api_key / EPREL_API_KEY	Optional; never log

Feed private-URL allowlist

SSRF allowlist for private/LAN feed URLs: /admin/settings → feeds.private_url_allowlist (CSV). Optional env FEED_URL_PRIVATE_ALLOWLIST remains a fallback (settings merge/override — see feeds.ApplyPrivateAllowlistCSV).

WooCommerce schedule

Worker already:

1. Claims pending_sync every 2s (ClaimNextPending → SyncCompany)
2. Every 15m enqueues due enabled configs (EnqueueDueScheduled, default 6h or sync_options.schedule_interval_hours)

Manual enqueue: POST /api/woocommerce/sync (dashboard) or operator cron hitting that endpoint / re-running enqueue SQL is unnecessary if the worker is up.

Schema migrations

# Git Bash / WSL / macOS / Linux
make migrate
# PowerShell
.\scripts\migrate.ps1

Current head: 042_user_session_version.sql (goose, not Drizzle; includes 039_worker_heartbeats for /readyz, 040_job_hotpath_indexes for claim/list, 041 for forgot-password tokens, 042 for users.session_version). Confirm goose status includes versions through 042 before relying on worker readiness / job claim indexes / self-serve reset / session revoke. MySQL→PG data cutover uses cmd/migrator separately — see production-checklist.md and cutover.md.

Read-only gate: npm run cutover:deploy-check / node scripts/cutover-deploy-check.mjs (goose 039–042 + /readyz worker + adapter-node host gates).

Cutover blockers (honest)

• No production cutover executed from this repo automation.
• Live migrator dry-run needs operator-supplied MIGRATE_MYSQL_DSN — do not invent credentials. Until a real DSN is available, cutover stays blocked.
• SMTP must be verified against set-password-hooks.json before DNS switch.