Initial commit of Descrybe v2 without local scratch artifacts.
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# Edge rate limits (multi-replica)
|
||||
|
||||
App HTTP limiters in `apps/api/internal/httpapi/ratelimit.go` are **per process**. Put **one** nginx or Caddy in front of all API replicas so RPM is cluster-wide.
|
||||
|
||||
## Env contract
|
||||
|
||||
```text
|
||||
RATE_LIMIT_REPLICAS=<N> # N = API OS processes behind this edge (default 1)
|
||||
RATE_LIMIT_MULTI_REPLICA=true # acknowledge multi-replica without a shared store (boot warns)
|
||||
TRUSTED_PROXIES=<edge CIDRs/IPs> # hop-1 peers only so RemoteAddr is client IP
|
||||
# RATE_LIMIT_BACKEND=memory # redis|postgres not implemented — forced to memory
|
||||
```
|
||||
|
||||
- `RATE_LIMIT_REPLICAS` divides **HTTP middleware only** via `rateLimitEffectiveCap` (ceil) so even load ≈ documented RPM.
|
||||
- **Not** a shared counter. Does **not** cover login email lockout, `StartLimiter`, `AIRateLimiter`, or email send limiters.
|
||||
- **Not** a substitute for the edge zones below. Edge = hard global RPM; `RATE_LIMIT_REPLICAS` = optional soft split of in-app HTTP caps.
|
||||
|
||||
Company-keyed app budgets (heavy sync/process/export **30**/min, marketing, `/api/v1` company **120**/min) stay in-app — edge below is **per client IP**.
|
||||
|
||||
Documented IP-ish targets (align zones to these):
|
||||
|
||||
| Surface | RPM | Key |
|
||||
|---------|-----|-----|
|
||||
| Auth login / invite / set-password / sales-contact | 10 | IP |
|
||||
| Auth register | 5 | IP |
|
||||
| `/api/public/*` | 30 | IP |
|
||||
| Public export probe (invalid token shape) | 15 | IP |
|
||||
| `/api/v1` with API key present | 60 | IP |
|
||||
|
||||
Checklist / cutover: [docs/production-checklist.md](../../docs/production-checklist.md) §1f · [docs/cutover.md](../../docs/cutover.md#8-multi-replica-edge-rate-limits).
|
||||
|
||||
---
|
||||
|
||||
## nginx (shared `limit_req_zone` across upstreams)
|
||||
|
||||
```nginx
|
||||
# Shared memory → one budget for all replicas behind this proxy.
|
||||
limit_req_zone $binary_remote_addr zone=auth_login:10m rate=10r/m;
|
||||
limit_req_zone $binary_remote_addr zone=auth_register:10m rate=5r/m;
|
||||
limit_req_zone $binary_remote_addr zone=public_api:10m rate=30r/m;
|
||||
limit_req_zone $binary_remote_addr zone=public_export:10m rate=30r/m;
|
||||
limit_req_zone $binary_remote_addr zone=apikey_ip:10m rate=60r/m;
|
||||
|
||||
upstream descrybe_api {
|
||||
least_conn;
|
||||
server 10.0.1.10:28471;
|
||||
server 10.0.1.11:28471;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name api.example.com;
|
||||
|
||||
location = /api/auth/register {
|
||||
limit_req zone=auth_register burst=2 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location ~ ^/api/auth/(login|forgot-password|reset-password|invite-preview|accept-invite|complete-set-password)$ {
|
||||
limit_req zone=auth_login burst=3 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location = /api/sales/contact {
|
||||
limit_req zone=auth_login burst=3 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location ~ ^/api/public/export-feeds/ {
|
||||
limit_req zone=public_export burst=5 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location /api/public/ {
|
||||
limit_req zone=public_api burst=5 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location /api/v1/ {
|
||||
limit_req zone=apikey_ip burst=10 nodelay;
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://descrybe_api;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`429` from nginx is expected under burst; app still returns its own `429` + `Retry-After` / `RateLimit*` when in-process caps trip.
|
||||
|
||||
---
|
||||
|
||||
## Caddy (needs `rate_limit` handler — xcaddy / module)
|
||||
|
||||
```caddy
|
||||
# xcaddy build --with github.com/mholt/caddy-ratelimit
|
||||
api.example.com {
|
||||
rate_limit {
|
||||
zone auth_login {
|
||||
key {remote_host}
|
||||
events 10
|
||||
window 1m
|
||||
}
|
||||
zone auth_register {
|
||||
key {remote_host}
|
||||
events 5
|
||||
window 1m
|
||||
}
|
||||
zone public_api {
|
||||
key {remote_host}
|
||||
events 30
|
||||
window 1m
|
||||
}
|
||||
zone apikey_ip {
|
||||
key {remote_host}
|
||||
events 60
|
||||
window 1m
|
||||
}
|
||||
}
|
||||
|
||||
@register path /api/auth/register
|
||||
rate_limit @register zone auth_register
|
||||
|
||||
@auth path /api/auth/login /api/auth/forgot-password /api/auth/reset-password \
|
||||
/api/auth/invite-preview /api/auth/accept-invite /api/auth/complete-set-password \
|
||||
/api/sales/contact
|
||||
rate_limit @auth zone auth_login
|
||||
|
||||
@public path /api/public/*
|
||||
rate_limit @public zone public_api
|
||||
|
||||
@v1 path /api/v1/*
|
||||
rate_limit @v1 zone apikey_ip
|
||||
|
||||
reverse_proxy 10.0.1.10:28471 10.0.1.11:28471 {
|
||||
header_up X-Forwarded-For {http.request.header.X-Forwarded-For}
|
||||
header_up X-Real-IP {remote_host}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Match zone event counts to the table above. Prefer nginx if you want stock OpenResty/nginx without a custom Caddy build.
|
||||
|
||||
See also: [docs/security-notes.md](../../docs/security-notes.md) (ops knobs), [docs/production-readiness.md](../../docs/production-readiness.md) § edge rate limits.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Prometheus examples (Descrybe v2)
|
||||
|
||||
Starter scrape + alert configs for gated `GET /metrics` and `/readyz` worker freshness.
|
||||
**Examples only** — not a live Prometheus install. No secrets; replace placeholder hosts with private/mesh targets.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| [`scrape.example.yml`](scrape.example.yml) | Job snippets: `descrybe-api`, `descrybe-worker`, blackbox `/readyz` |
|
||||
| [`alerts.example.yml`](alerts.example.yml) | RED + sync + scrape-down + worker readiness rules |
|
||||
|
||||
Canonical guidance: [docs/production-readiness.md — Ops: Prometheus scrape](../../docs/production-readiness.md#ops-prometheus-scrape).
|
||||
Cutover checklist: [docs/production-checklist.md §1d](../../docs/production-checklist.md#1d-metrics-scrape--alerts-cutover).
|
||||
|
||||
## Topology (summary)
|
||||
|
||||
```
|
||||
Prometheus ──scrape──► API HTTP_ADDR GET /metrics → HTTP RED series
|
||||
──scrape──► Worker METRICS_ADDR GET /metrics → sync_* series
|
||||
──probe───► API HTTP_ADDR GET /readyz → worker freshness (HTTP 200)
|
||||
Public VIP ──deny────► /metrics
|
||||
```
|
||||
|
||||
- Production Gate: loopback (or `METRICS_PUBLIC=1` on a **private** VIP only).
|
||||
- Worker sync series require `METRICS_ADDR` (e.g. `127.0.0.1:9091`).
|
||||
- Heartbeat age is **not** a Prom series. `/readyz` returns **503** when `checks.worker` is `missing`/`stale` (stale after **60s**). On-call triage: JSON `worker_last_seen_age_s` + `checks.worker`.
|
||||
|
||||
## Status (cutover blocker #9)
|
||||
|
||||
| Layer | Status |
|
||||
|-------|--------|
|
||||
| Code + example scrape/alert files | **CODE DONE** |
|
||||
| Live scrape targets + alertmanager routing on the host | **OPS OPEN** |
|
||||
@@ -0,0 +1,94 @@
|
||||
# Example Prometheus alerting rules for Descrybe v2.
|
||||
# Load via rule_files / Prometheus Operator PrometheusRule.
|
||||
# Tune thresholds to your baseline before paging. No secrets.
|
||||
#
|
||||
# Docs: docs/production-readiness.md#ops-prometheus-scrape
|
||||
# Worker triage (not PromQL): GET /readyz → checks.worker + worker_last_seen_age_s
|
||||
# (stale after 60s = jobs.DefaultHeartbeatStaleAfter).
|
||||
|
||||
groups:
|
||||
- name: descrybe-red
|
||||
rules:
|
||||
- alert: DescrybeAPIHigh5xxRate
|
||||
expr: |
|
||||
(
|
||||
sum(rate(http_requests_total{job="descrybe-api",code=~"5.."}[5m]))
|
||||
/
|
||||
clamp_min(sum(rate(http_requests_total{job="descrybe-api"}[5m])), 1e-9)
|
||||
) > 0.05
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
service: descrybe-api
|
||||
annotations:
|
||||
summary: "Descrybe API 5xx rate >5% (10m)"
|
||||
description: "RED Errors — investigate recent deploys, DB, and upstreams."
|
||||
|
||||
- alert: DescrybeAPIHighP99Latency
|
||||
expr: |
|
||||
histogram_quantile(
|
||||
0.99,
|
||||
sum by (le) (rate(http_request_duration_seconds_bucket{job="descrybe-api"}[5m]))
|
||||
) > 2
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
service: descrybe-api
|
||||
annotations:
|
||||
summary: "Descrybe API p99 latency >2s (15m)"
|
||||
description: "RED Duration — check slow routes and DB pool saturation."
|
||||
|
||||
- name: descrybe-sync
|
||||
rules:
|
||||
- alert: DescrybeSyncFailures
|
||||
expr: |
|
||||
sum by (kind) (increase(sync_failures_total{job="descrybe-worker"}[15m])) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
service: descrybe-worker
|
||||
annotations:
|
||||
summary: "Descrybe sync failures for kind {{ $labels.kind }}"
|
||||
description: "Sustained sync_failures_total increase — check feed/Woo/Shopify connectors and worker logs."
|
||||
|
||||
- alert: DescrybeSyncSlowP95
|
||||
expr: |
|
||||
histogram_quantile(
|
||||
0.95,
|
||||
sum by (le, kind) (rate(sync_duration_seconds_bucket{job="descrybe-worker"}[15m]))
|
||||
) > 120
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
service: descrybe-worker
|
||||
annotations:
|
||||
summary: "Descrybe sync p95 >120s for kind {{ $labels.kind }}"
|
||||
description: "Sync stall / slow — check source feed size and worker claim loops."
|
||||
|
||||
- name: descrybe-availability
|
||||
rules:
|
||||
- alert: DescrybeMetricsScrapeDown
|
||||
expr: up{job=~"descrybe-api|descrybe-worker"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: descrybe
|
||||
annotations:
|
||||
summary: "Prometheus cannot scrape {{ $labels.job }}"
|
||||
description: "Check private scrape path, Gate (loopback / METRICS_PUBLIC), and process health. Do not open /metrics on the public VIP."
|
||||
|
||||
# /readyz returns 503 when worker heartbeat is missing or stale (>60s).
|
||||
# Requires job descrybe-readyz from scrape.example.yml (blackbox http_2xx).
|
||||
- alert: DescrybeWorkerHeartbeatStale
|
||||
expr: probe_success{job="descrybe-readyz",probe="readyz"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
service: descrybe-worker
|
||||
annotations:
|
||||
summary: "Descrybe /readyz probe failing (worker missing or stale >60s)"
|
||||
description: >-
|
||||
On-call triage (not a Prom series): curl private GET /readyz and read
|
||||
checks.worker (ok|missing|stale) and worker_last_seen_age_s when present.
|
||||
Stale after 60s. Do not start a second worker while age is still fresh.
|
||||
See docs/ops-runtime.md and scripts/cutover-deploy-check.mjs.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Example Prometheus scrape snippets for Descrybe v2.
|
||||
# Merge under scrape_configs (or use as a separate file included by your Prometheus).
|
||||
# Replace placeholder hosts with private / loopback / mesh DNS only.
|
||||
# Do NOT scrape the public VIP. No scrape secrets belong in this file.
|
||||
#
|
||||
# Docs: docs/production-readiness.md#ops-prometheus-scrape
|
||||
# Checklist: docs/production-checklist.md#1d-metrics-scrape--alerts-cutover
|
||||
|
||||
scrape_configs:
|
||||
# API HTTP RED — same process as HTTP_ADDR (local default :28471).
|
||||
- job_name: descrybe-api
|
||||
metrics_path: /metrics
|
||||
scrape_interval: 15s
|
||||
static_configs:
|
||||
- targets:
|
||||
- "127.0.0.1:28471" # TODO(ops): private mesh DNS, e.g. api.internal:8080
|
||||
labels:
|
||||
service: descrybe-api
|
||||
|
||||
# Worker sync series — only when METRICS_ADDR is set on cmd/worker.
|
||||
- job_name: descrybe-worker
|
||||
metrics_path: /metrics
|
||||
scrape_interval: 15s
|
||||
static_configs:
|
||||
- targets:
|
||||
- "127.0.0.1:9091" # TODO(ops): matches METRICS_ADDR (loopback / private only)
|
||||
labels:
|
||||
service: descrybe-worker
|
||||
|
||||
# Worker freshness via /readyz (not a /metrics series).
|
||||
# /readyz returns 503 when worker heartbeat is missing or stale (>60s).
|
||||
# Requires blackbox_exporter; adjust blackbox address + module to your install.
|
||||
- job_name: descrybe-readyz
|
||||
metrics_path: /probe
|
||||
params:
|
||||
module: [http_2xx]
|
||||
scrape_interval: 15s
|
||||
static_configs:
|
||||
- targets:
|
||||
- "http://127.0.0.1:28471/readyz" # TODO(ops): private API base + /readyz
|
||||
labels:
|
||||
service: descrybe-api
|
||||
probe: readyz
|
||||
relabel_configs:
|
||||
- source_labels: [__address__]
|
||||
target_label: __param_target
|
||||
- source_labels: [__param_target]
|
||||
target_label: instance
|
||||
- target_label: __address__
|
||||
replacement: "127.0.0.1:9115" # TODO(ops): blackbox_exporter listen address
|
||||
Reference in New Issue
Block a user