greeneclipse
8580c996c3
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
185 lines
10 KiB
Markdown
185 lines
10 KiB
Markdown
# Runtime ops notes (WS7)
|
||
|
||
Local/dev and production operator notes for platform, mail, billing, and WooCommerce schedules. **Do not commit secrets.**
|
||
|
||
**Env file policy:** one **root** `.env` (or secrets mapped into process env). Do not duplicate into `apps/api/.env` or tmp placeholders. Platform Stripe / EPREL / feed allowlist → `/admin/settings`; tenant OpenAI / marketing email / stores → `/integrations/ai`, `/integrations/email`, `/stores` — see [README.md](../README.md#environment-one-file).
|
||
|
||
## Health
|
||
|
||
- `GET /healthz` — liveness (no DB)
|
||
- `GET /readyz` — readiness: Postgres ping **and** a fresh worker heartbeat (`worker_id=processing`, stale after **60s**) plus queue probe; returns `maintenance` / `read_only` / `hypercare` flags and `checks.{database,worker,queue}`
|
||
- `GET /metrics` — Prometheus text (HTTP RED on API; worker sync series when `METRICS_ADDR` is set). Production Gate: loopback or `METRICS_PUBLIC=1`. Example scrape + alerts: [`deploy/prometheus/`](../deploy/prometheus/). Worker age for on-call is `/readyz` `worker_last_seen_age_s` (not a Prom series).
|
||
|
||
When the worker is down or stale, `/readyz` is **503** with short `error` plus operator `reason` (no secrets). Example (API-only / stale heartbeat):
|
||
|
||
```json
|
||
{
|
||
"status": "not_ready",
|
||
"service": "api",
|
||
"checks": { "database": "ok", "worker": "stale", "queue": "ok" },
|
||
"error": "worker heartbeat stale",
|
||
"reason": "Processing worker heartbeat older than 1m0s. API-only readiness 503 is expected — start or restart the worker (npm run dev includes it, or npm run dev:worker).",
|
||
"queue_pending": 5,
|
||
"worker_last_seen_age_s": 120
|
||
}
|
||
```
|
||
|
||
**Prefer one stack:** `npm run dev` runs **api + web + worker** (ports **28471** / **28472**). Compose starts **Postgres only** — the worker is a host process. Browsers need the web process — API-only leaves `:28472` **CONNECTION_REFUSED** even when `/readyz` is green.
|
||
|
||
If the API is already up without a worker (`npm run dev:api`), start **one** worker only (or use `npm run dev:backend` for api+worker):
|
||
|
||
```bash
|
||
npm run dev:worker
|
||
# equivalent: cd apps/api && go run ./cmd/worker
|
||
# api+worker (no web): npm run dev:backend
|
||
```
|
||
|
||
Do **not** start a second worker when `checks.worker=ok` / `worker_last_seen_age_s` is fresh — duplicate claim loops contend on A1 (or any tenant) job rows. Before restarting the full stack: `node scripts/free-dev-ports.mjs` (or rely on `predev`) then `npm run dev`, and stop any leftover standalone `cmd/worker`. Readiness itself is read-only (no A1 catalog writes). API-only → `/readyz` **503** is expected; see [README troubleshooting](../README.md#troubleshooting-readyz-returns-503).
|
||
|
||
Cutover rehearsal: probes must stay green while `MAINTENANCE_MODE` / `READ_ONLY_MODE` may block app traffic.
|
||
|
||
## Postgres pgx pool (API + worker)
|
||
|
||
Shared by `cmd/api` and `cmd/worker` via `internal/db.NewPool`. Size pools for **your** Postgres `max_connections` and replica count — do not copy high defaults blindly.
|
||
|
||
| Variable | Default | Purpose |
|
||
|---|---|---|
|
||
| `DB_MAX_CONNS` | `20` | Hard ceiling per process (`MaxConns`) |
|
||
| `DB_MIN_CONNS` | `2` | Warm floor (`MinConns`, ~10–30% of max) |
|
||
| `DB_MAX_CONN_LIFETIME` | `1h` | Recycle connections before server-side idle kills / DNS drift |
|
||
| `DB_MAX_CONN_LIFETIME_JITTER` | `6m` | Random extra lifetime (~10% of lifetime) — avoids thundering-herd reconnects; `0` disables |
|
||
| `DB_MAX_CONN_IDLE_TIME` | `5m` | Close idle conns during health checks |
|
||
| `DB_HEALTH_CHECK_PERIOD` | `1m` | Background idle health check interval |
|
||
| `DB_STATEMENT_TIMEOUT` | `30s` | Postgres `statement_timeout` GUC per connection; `0` disables |
|
||
|
||
Formula sketch: `MaxConns ≈ (max_connections − reserved) / instance_count`. Typical per-process range is 20–50. Always keep jitter set in multi-instance deploys.
|
||
|
||
## SMTP (invites + set-password)
|
||
|
||
Platform invite / set-password mail uses process env (`internal/mail` + `cmd/mailhooks`). This is separate from **tenant** marketing email (`/integrations/email`).
|
||
|
||
Env (no defaults that embed secrets):
|
||
|
||
| Variable | Purpose |
|
||
|---|---|
|
||
| `SMTP_ENABLED` | `true` to send; otherwise no-op log (subject only, no PII) |
|
||
| `SMTP_HOST` / `SMTP_PORT` | SMTP server |
|
||
| `SMTP_USER` / `SMTP_PASSWORD` | Auth (optional for open relays) |
|
||
| `SMTP_FROM` | From address |
|
||
| `WEB_ORIGIN` | Base URL for accept-invite links |
|
||
| `TOKEN_SIGNING_SECRET` | HMAC for admin-issued set-password tokens (not migrator hooks) |
|
||
| `EMAIL_DRY_RUN` | Default **true** when unset (safe). Live `mailhooks` send fails closed until you pass `-dry-run` or set `EMAIL_DRY_RUN=false` (and/or disable dry-run in admin platform mail settings). |
|
||
|
||
### Migrator hooks → mail
|
||
|
||
After a live migrator run (or `go run ./cmd/migrator -issue-set-password-invites`), maps-dir contains `password_invites.json` and `set-password-hooks.json` (invite tokens + URLs for `must_set_password` users). See [migration-readiness.md](migration-readiness.md).
|
||
|
||
#### Operator smoke (no live SMTP)
|
||
|
||
Rehearse under dry-run **before** flipping SMTP. Prefer process env `EMAIL_DRY_RUN=true` (or leave unset — default is dry-run):
|
||
|
||
```bash
|
||
cd apps/api
|
||
# Expect exit 1: fail-closed when dry-run is on and -dry-run is omitted
|
||
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json
|
||
|
||
# Expect exit 0: counts subjects only; no SMTP dial
|
||
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json -dry-run
|
||
# log: mailhooks: dry-run subject="Set your Descrybe password"
|
||
# stdout: mailhooks: sent=N skipped=… failed=0 smtp_enabled=… total=…
|
||
```
|
||
|
||
Pass criteria: second command exits `0`, `failed=0`, and subjects are logged without contacting an SMTP host. Unit gate: `go test ./cmd/mailhooks/`.
|
||
|
||
#### Live send (after SMTP proven)
|
||
|
||
```bash
|
||
cd apps/api
|
||
# Requires EMAIL_DRY_RUN=false (or admin platform mail dry-run off) + SMTP_ENABLED=true + host/from
|
||
go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json
|
||
```
|
||
|
||
Rate limit via `-delay-ms` (default 100). Team invites use the same SMTP path from `POST /api/team/invites`.
|
||
|
||
## Session idle policy
|
||
|
||
- Absolute session lifetime: 7 days
|
||
- Idle timeout: `SESSION_IDLE_HOURS` (default 24)
|
||
- Set `SESSION_SECURE=true` behind HTTPS in production
|
||
|
||
## Credentials encryption (WooCommerce)
|
||
|
||
| Variable | Purpose |
|
||
|---|---|
|
||
| `CREDENTIALS_ENCRYPTION_KEY` | **Required in production** for AES-GCM at-rest encryption of Woo consumer secrets |
|
||
|
||
If unset, the API/worker derive a key from `TOKEN_SIGNING_SECRET` + `DATABASE_URL`. That derived key is **local/dev only** — rotate by setting an explicit `CREDENTIALS_ENCRYPTION_KEY` before storing production Woo credentials. Changing the key without re-saving configs makes existing ciphertext unreadable.
|
||
|
||
## AI (OpenAI-compatible / Green Chat)
|
||
|
||
**Preferred:** company admin configures the provider in **`/integrations/ai`** (popular BYOK or custom OpenAI-compatible base URL + key). Encrypted at rest with `APP_ENCRYPTION_KEY`.
|
||
|
||
Optional process-env platform fallback (when the company leaves mode **internal** and a platform key is present in config):
|
||
|
||
| Variable | Purpose |
|
||
|---|---|
|
||
| `OPENAI_API_KEY` | Bearer token; non-empty enables the platform Completer fallback |
|
||
| `OPENAI_BASE_URL` | Default `https://api.openai.com/v1`; LAN Green Chat e.g. `http://HOST:PORT/v1` |
|
||
| `OPENAI_MODEL` | Chat model id (`GET /v1/models`) |
|
||
|
||
If set, put these in the **root** `.env` only — not `apps/api/.env`. Shared by worker (processing enhance) and API (campaign + SEO AI). Brand kit injects into those prompts. See [green-chat-llm.md](green-chat-llm.md).
|
||
|
||
[UNCERTAIN] Other agents may remove env-based platform OpenAI in favor of dashboard-only configuration; prefer `/integrations/ai` for new setups.
|
||
|
||
## Billing / processing costs
|
||
|
||
- Debit per processed product: `processing_costs.product_processing` + `ceil(tokens/1000) * openai_token_k`
|
||
- Defaults seeded by worker/`EnsureDefaultCosts`
|
||
- Plans admin / assign / add-credits: `/api/admin/*` (platform admin only)
|
||
- Billing cycles: worker every 15m via `RunDueBillingCycles`
|
||
|
||
## EPREL energy labels
|
||
|
||
Optional enrichment during product processing. See [eprel.md](eprel.md).
|
||
|
||
**Preferred:** platform admin → **`/admin/settings`** (`values.eprel.enabled`, `eprel.base_url`, `eprel.timeout`, `eprel.fiche_language`, `eprel.api_key`). Process `EPREL_*` env is an optional fallback.
|
||
|
||
| Settings key / env | Purpose |
|
||
|---|---|
|
||
| `eprel.enabled` / `EPREL_ENABLED` | `true` to fetch label/fiche/class after AI steps |
|
||
| `eprel.timeout` / `EPREL_TIMEOUT` | HTTP timeout (default `10s`) |
|
||
| `eprel.fiche_language` / `EPREL_FICHE_LANGUAGE` | Fiche PDF language (default `EN`) |
|
||
| `eprel.api_key` / `EPREL_API_KEY` | Optional; never log |
|
||
|
||
## Feed private-URL allowlist
|
||
|
||
SSRF allowlist for private/LAN feed URLs: **`/admin/settings`** → `feeds.private_url_allowlist` (CSV). Optional env `FEED_URL_PRIVATE_ALLOWLIST` remains a fallback (settings merge/override — see `feeds.ApplyPrivateAllowlistCSV`).
|
||
|
||
## WooCommerce schedule
|
||
|
||
Worker already:
|
||
|
||
1. Claims `pending_sync` every 2s (`ClaimNextPending` → `SyncCompany`)
|
||
2. Every 15m enqueues due enabled configs (`EnqueueDueScheduled`, default 6h or `sync_options.schedule_interval_hours`)
|
||
|
||
Manual enqueue: `POST /api/woocommerce/sync` (dashboard) or operator cron hitting that endpoint / re-running enqueue SQL is unnecessary if the worker is up.
|
||
|
||
## Schema migrations
|
||
|
||
```bash
|
||
# Git Bash / WSL / macOS / Linux
|
||
make migrate
|
||
# PowerShell
|
||
.\scripts\migrate.ps1
|
||
```
|
||
|
||
Current head: **042_user_session_version.sql** (goose, not Drizzle; includes **039_worker_heartbeats** for `/readyz`, **040_job_hotpath_indexes** for claim/list, **041** for forgot-password tokens, **042** for `users.session_version`). Confirm goose status includes versions through **042** before relying on worker readiness / job claim indexes / self-serve reset / session revoke. MySQL→PG **data** cutover uses cmd/migrator separately — see [production-checklist.md](production-checklist.md) and [cutover.md](cutover.md).
|
||
|
||
Read-only gate: `npm run cutover:deploy-check` / `node scripts/cutover-deploy-check.mjs` (goose 039–042 + `/readyz` worker + adapter-node host gates).
|
||
|
||
## Cutover blockers (honest)
|
||
|
||
- **No production cutover executed** from this repo automation.
|
||
- Live migrator dry-run needs operator-supplied `MIGRATE_MYSQL_DSN` — do not invent credentials. Until a real DSN is available, cutover stays blocked.
|
||
- SMTP must be verified against `set-password-hooks.json` before DNS switch.
|