descrybe/docs/ops-runtime.md

# Runtime ops notes (WS7)

Local/dev and production operator notes for platform, mail, billing, and WooCommerce schedules. **Do not commit secrets.**

**Env file policy:** one **root** `.env` (or secrets mapped into process env). Do not duplicate into `apps/api/.env` or tmp placeholders. Platform Stripe / EPREL / feed allowlist → `/admin/settings`; tenant OpenAI / marketing email / stores → `/integrations/ai`, `/integrations/email`, `/stores` — see [README.md](../README.md#environment-one-file).

## Health

- `GET /healthz` — liveness (no DB)
- `GET /readyz` — readiness: Postgres ping **and** a fresh worker heartbeat (`worker_id=processing`, stale after **60s**) plus queue probe; returns `maintenance` / `read_only` / `hypercare` flags and `checks.{database,worker,queue}`
- `GET /metrics` — Prometheus text (HTTP RED on API; worker sync series when `METRICS_ADDR` is set). Production Gate: loopback or `METRICS_PUBLIC=1`. Example scrape + alerts: [`deploy/prometheus/`](../deploy/prometheus/). Worker age for on-call is `/readyz` `worker_last_seen_age_s` (not a Prom series).

When the worker is down or stale, `/readyz` is **503** with short `error` plus operator `reason` (no secrets). Example (API-only / stale heartbeat):

```json
{
  "status": "not_ready",
  "service": "api",
  "checks": { "database": "ok", "worker": "stale", "queue": "ok" },
  "error": "worker heartbeat stale",
  "reason": "Processing worker heartbeat older than 1m0s. API-only readiness 503 is expected — start or restart the worker (npm run dev includes it, or npm run dev:worker).",
  "queue_pending": 5,
  "worker_last_seen_age_s": 120
}
```

**Prefer one stack:** `npm run dev` runs **api + web + worker** (ports **28471** / **28472**). Compose starts **Postgres only** — the worker is a host process. Browsers need the web process — API-only leaves `:28472` **CONNECTION_REFUSED** even when `/readyz` is green.

If the API is already up without a worker (`npm run dev:api`), start **one** worker only (or use `npm run dev:backend` for api+worker):

```bash
npm run dev:worker
# equivalent: cd apps/api && go run ./cmd/worker
# api+worker (no web): npm run dev:backend
```

Do **not** start a second worker when `checks.worker=ok` / `worker_last_seen_age_s` is fresh — duplicate claim loops contend on A1 (or any tenant) job rows. Before restarting the full stack: `node scripts/free-dev-ports.mjs` (or rely on `predev`) then `npm run dev`, and stop any leftover standalone `cmd/worker`. Readiness itself is read-only (no A1 catalog writes). API-only → `/readyz` **503** is expected; see [README troubleshooting](../README.md#troubleshooting-readyz-returns-503).

Cutover rehearsal: probes must stay green while `MAINTENANCE_MODE` / `READ_ONLY_MODE` may block app traffic.

## Postgres pgx pool (API + worker)

Shared by `cmd/api` and `cmd/worker` via `internal/db.NewPool`. Size pools for **your** Postgres `max_connections` and replica count — do not copy high defaults blindly.

| Variable | Default | Purpose |
|---|---|---|
| `DB_MAX_CONNS` | `20` | Hard ceiling per process (`MaxConns`) |
| `DB_MIN_CONNS` | `2` | Warm floor (`MinConns`, ~10–30% of max) |
| `DB_MAX_CONN_LIFETIME` | `1h` | Recycle connections before server-side idle kills / DNS drift |
| `DB_MAX_CONN_LIFETIME_JITTER` | `6m` | Random extra lifetime (~10% of lifetime) — avoids thundering-herd reconnects; `0` disables |
| `DB_MAX_CONN_IDLE_TIME` | `5m` | Close idle conns during health checks |
| `DB_HEALTH_CHECK_PERIOD` | `1m` | Background idle health check interval |
| `DB_STATEMENT_TIMEOUT` | `30s` | Postgres `statement_timeout` GUC per connection; `0` disables |

Formula sketch: `MaxConns ≈ (max_connections − reserved) / instance_count`. Typical per-process range is 20–50. Always keep jitter set in multi-instance deploys.

## SMTP (invites + set-password)

Platform invite / set-password mail uses process env (`internal/mail` + `cmd/mailhooks`). This is separate from **tenant** marketing email (`/integrations/email`).

Env (no defaults that embed secrets):

| Variable | Purpose |
|---|---|
| `SMTP_ENABLED` | `true` to send; otherwise no-op log (subject only, no PII) |
| `SMTP_HOST` / `SMTP_PORT` | SMTP server |
| `SMTP_USER` / `SMTP_PASSWORD` | Auth (optional for open relays) |
| `SMTP_FROM` | From address |
| `WEB_ORIGIN` | Base URL for accept-invite links |
| `TOKEN_SIGNING_SECRET` | HMAC for admin-issued set-password tokens (not migrator hooks) |
| `EMAIL_DRY_RUN` | Default **true** when unset (safe). Live `mailhooks` send fails closed until you pass `-dry-run` or set `EMAIL_DRY_RUN=false` (and/or disable dry-run in admin platform mail settings). |

### Migrator hooks → mail

After a live migrator run (or `go run ./cmd/migrator -issue-set-password-invites`), maps-dir contains `password_invites.json` and `set-password-hooks.json` (invite tokens + URLs for `must_set_password` users). See [migration-readiness.md](migration-readiness.md).

#### Operator smoke (no live SMTP)

Rehearse under dry-run **before** flipping SMTP. Prefer process env `EMAIL_DRY_RUN=true` (or leave unset — default is dry-run):

```bash
cd apps/api
# Expect exit 1: fail-closed when dry-run is on and -dry-run is omitted
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json

# Expect exit 0: counts subjects only; no SMTP dial
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json -dry-run
# log: mailhooks: dry-run subject="Set your Descrybe password"
# stdout: mailhooks: sent=N skipped=… failed=0 smtp_enabled=… total=…
```

Pass criteria: second command exits `0`, `failed=0`, and subjects are logged without contacting an SMTP host. Unit gate: `go test ./cmd/mailhooks/`.

#### Live send (after SMTP proven)

```bash
cd apps/api
# Requires EMAIL_DRY_RUN=false (or admin platform mail dry-run off) + SMTP_ENABLED=true + host/from
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json
```

Rate limit via `-delay-ms` (default 100). Team invites use the same SMTP path from `POST /api/team/invites`.

## Session idle policy

- Absolute session lifetime: 7 days
- Idle timeout: `SESSION_IDLE_HOURS` (default 24)
- Set `SESSION_SECURE=true` behind HTTPS in production

## Credentials encryption (WooCommerce)

| Variable | Purpose |
|---|---|
| `CREDENTIALS_ENCRYPTION_KEY` | **Required in production** for AES-GCM at-rest encryption of Woo consumer secrets |

If unset, the API/worker derive a key from `TOKEN_SIGNING_SECRET` + `DATABASE_URL`. That derived key is **local/dev only** — rotate by setting an explicit `CREDENTIALS_ENCRYPTION_KEY` before storing production Woo credentials. Changing the key without re-saving configs makes existing ciphertext unreadable.

## AI (OpenAI-compatible / Green Chat)

**Preferred:** company admin configures the provider in **`/integrations/ai`** (popular BYOK or custom OpenAI-compatible base URL + key). Encrypted at rest with `APP_ENCRYPTION_KEY`.

Optional process-env platform fallback (when the company leaves mode **internal** and a platform key is present in config):

| Variable | Purpose |
|---|---|
| `OPENAI_API_KEY` | Bearer token; non-empty enables the platform Completer fallback |
| `OPENAI_BASE_URL` | Default `https://api.openai.com/v1`; LAN Green Chat e.g. `http://HOST:PORT/v1` |
| `OPENAI_MODEL` | Chat model id (`GET /v1/models`) |

If set, put these in the **root** `.env` only — not `apps/api/.env`. Shared by worker (processing enhance) and API (campaign + SEO AI). Brand kit injects into those prompts. See [green-chat-llm.md](green-chat-llm.md).

[UNCERTAIN] Other agents may remove env-based platform OpenAI in favor of dashboard-only configuration; prefer `/integrations/ai` for new setups.

## Billing / processing costs

- Debit per processed product: `processing_costs.product_processing` + `ceil(tokens/1000) * openai_token_k`
- Defaults seeded by worker/`EnsureDefaultCosts`
- Plans admin / assign / add-credits: `/api/admin/*` (platform admin only)
- Billing cycles: worker every 15m via `RunDueBillingCycles`

## EPREL energy labels

Optional enrichment during product processing. See [eprel.md](eprel.md).

**Preferred:** platform admin → **`/admin/settings`** (`values.eprel.enabled`, `eprel.base_url`, `eprel.timeout`, `eprel.fiche_language`, `eprel.api_key`). Process `EPREL_*` env is an optional fallback.

| Settings key / env | Purpose |
|---|---|
| `eprel.enabled` / `EPREL_ENABLED` | `true` to fetch label/fiche/class after AI steps |
| `eprel.timeout` / `EPREL_TIMEOUT` | HTTP timeout (default `10s`) |
| `eprel.fiche_language` / `EPREL_FICHE_LANGUAGE` | Fiche PDF language (default `EN`) |
| `eprel.api_key` / `EPREL_API_KEY` | Optional; never log |

## Feed private-URL allowlist

SSRF allowlist for private/LAN feed URLs: **`/admin/settings`** → `feeds.private_url_allowlist` (CSV). Optional env `FEED_URL_PRIVATE_ALLOWLIST` remains a fallback (settings merge/override — see `feeds.ApplyPrivateAllowlistCSV`).

## WooCommerce schedule

Worker already:

1. Claims `pending_sync` every 2s (`ClaimNextPending` → `SyncCompany`)
2. Every 15m enqueues due enabled configs (`EnqueueDueScheduled`, default 6h or `sync_options.schedule_interval_hours`)

Manual enqueue: `POST /api/woocommerce/sync` (dashboard) or operator cron hitting that endpoint / re-running enqueue SQL is unnecessary if the worker is up.

## Schema migrations

```bash
# Git Bash / WSL / macOS / Linux
make migrate
# PowerShell
.\scripts\migrate.ps1
```

Current head: **042_user_session_version.sql** (goose, not Drizzle; includes **039_worker_heartbeats** for `/readyz`, **040_job_hotpath_indexes** for claim/list, **041** for forgot-password tokens, **042** for `users.session_version`). Confirm goose status includes versions through **042** before relying on worker readiness / job claim indexes / self-serve reset / session revoke. MySQL→PG **data** cutover uses cmd/migrator separately — see [production-checklist.md](production-checklist.md) and [cutover.md](cutover.md).

Read-only gate: `npm run cutover:deploy-check` / `node scripts/cutover-deploy-check.mjs` (goose 039–042 + `/readyz` worker + adapter-node host gates).

## Cutover blockers (honest)

- **No production cutover executed** from this repo automation.
- Live migrator dry-run needs operator-supplied `MIGRATE_MYSQL_DSN` — do not invent credentials. Until a real DSN is available, cutover stays blocked.
- SMTP must be verified against `set-password-hooks.json` before DNS switch.