Initial commit of Descrybe v2 without local scratch artifacts.
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
# Admin capabilities and diagnostics
|
||||
|
||||
Platform admins use /admin/* (session + RequirePlatformAdmin). Support desk staff get ticket routes only (RequireSupportDesk).
|
||||
|
||||
## Useful admin APIs
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| GET /api/admin/diagnostics | Health: DB, queue, cache, storage, mail; config sanity (booleans only); recent job/AI failures |
|
||||
| GET /api/admin/analytics | Operational metrics dashboard data |
|
||||
| GET /api/admin/readiness | Cutover hypercare: must_set_password, companies without admin/plan |
|
||||
| GET /api/admin/jobs | Recent processing_jobs |
|
||||
| POST /api/admin/jobs/stuck-cleanup | Stuck running jobs cleanup |
|
||||
| GET/PUT /api/admin/settings | Platform settings (secrets masked on GET) |
|
||||
| GET/POST /api/admin/support/kb/articles | Support Knowledge CRUD |
|
||||
| GET/PUT /api/admin/support/auto-config | FAQ + AI auto-reply switchboard |
|
||||
| Users / companies / plans / credits | Org + billing admin |
|
||||
|
||||
UI: /admin/support/knowledge; diagnostics and analytics under the admin shell.
|
||||
|
||||
## Diagnose a stuck process
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Job stuck or failed] --> B{Worker running?}
|
||||
B -->|No| C["Start: go run ./cmd/worker"]
|
||||
B -->|Yes| D["GET /api/admin/diagnostics"]
|
||||
D --> E{queue.failed or stuck_running?}
|
||||
E -->|Yes| F["GET /api/admin/jobs + stuck-cleanup"]
|
||||
E -->|No| G["Check /readyz + company credits"]
|
||||
F --> H["Retry job or re-POST /products/process"]
|
||||
```
|
||||
|
||||
### Checklist
|
||||
|
||||
1. GET /healthz (liveness) and GET /readyz (Postgres + maintenance/read_only flags).
|
||||
2. GET /api/admin/diagnostics — overall ok|degraded|fail; never expect secrets in the payload.
|
||||
3. Confirm **worker** process is up (API alone does not drain processing).
|
||||
4. Filter recent failures: ?status=failed&failures_limit=25.
|
||||
5. Support auto AI: ticket auto_reply_status, /admin/support inbox flag=needs_human, AI role support under /admin/settings.
|
||||
6. Cutover: GET /api/admin/readiness before DNS switch.
|
||||
|
||||
Diagnostics intentionally excludes marketing charts — use analytics for trends, diagnostics for troubleshooting.
|
||||
|
||||
Sources: apps/api/internal/httpapi/admin_diagnostics_handlers.go, server.go admin routes.
|
||||
@@ -0,0 +1,58 @@
|
||||
# Public API and Postman A1
|
||||
|
||||
Base URL (local): http://127.0.0.1:28471
|
||||
Auth: Authorization Bearer api_key or X-API-Key
|
||||
OpenAPI: GET /api/v1/openapi.yaml (no key). Health: GET /api/v1/health or GET /healthz.
|
||||
|
||||
## Core /api/v1 groups (API key)
|
||||
|
||||
| Group | Examples |
|
||||
|-------|----------|
|
||||
| Products | GET /products, GET /products/{id}, PATCH /products/{id}, POST /products/process, GET /products/process/{id} |
|
||||
| Feeds | GET/POST /feeds, POST /feeds/{id}/sync, mappings, extract-schema |
|
||||
| Categories / attributes | CRUD under /categories, /attributes |
|
||||
| Export | /export-feeds plus generate / export-products |
|
||||
| Process jobs | POST /process, list/get/cancel/retry |
|
||||
| Marketing calendar | /marketing/calendar (legacy /campaigns aliases) |
|
||||
|
||||
Mounted in apps/api/internal/httpapi/v1.go via mountV1.
|
||||
|
||||
## A1 two-EAN Postman flow
|
||||
|
||||
Collection: docs/postman/Descrybe-v2-A1-two-EANs.postman_collection.json
|
||||
Prerequisite: npm run seed:a1 (restores processing jobs so poll works).
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant P as Postman
|
||||
participant API as /api/v1
|
||||
participant W as worker
|
||||
P->>API: GET /health
|
||||
P->>API: GET /feeds
|
||||
P->>API: GET /products?feed_id=
|
||||
P->>API: POST /products/process (2 EANs)
|
||||
API-->>P: data.process_id
|
||||
loop until done
|
||||
P->>API: GET /products/process/{processId}
|
||||
end
|
||||
Note over W: Worker claims processing_jobs
|
||||
P->>API: GET /products?search=EAN
|
||||
P->>API: GET /export-feeds
|
||||
P->>API: GET /api/public/export-feeds/{token}.csv
|
||||
```
|
||||
|
||||
### Steps (collection order)
|
||||
|
||||
0. Health (no auth)
|
||||
1. List feeds (find Elkotex)
|
||||
2. List products on feed
|
||||
3. **Process two EANs** — processing_type full — copy data.process_id into processId
|
||||
4. Poll process status
|
||||
5–6. Search results by EAN
|
||||
7. Get product by UUID
|
||||
8. List export feeds
|
||||
9. Public CSV (no API key)
|
||||
|
||||
Demo API key and feed/EAN vars live in the Postman collection (local demo only). Full surface: docs/postman/Descrybe-v2-Demo-A1-all-v1.postman_collection.json.
|
||||
|
||||
Rate limits: process/sync/export POSTs are capped per company (RateLimitV1Process). Prefer enqueue + worker under load.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Architecture overview
|
||||
|
||||
Descrybe v2 is a **Go chi API + SvelteKit web + PostgreSQL** rewrite (no Clerk).
|
||||
|
||||
## Runtime processes
|
||||
|
||||
| Process | Role |
|
||||
|---------|------|
|
||||
| apps/api/cmd/api | HTTP API (sessions, CSRF, public /api/v1, admin) |
|
||||
| apps/api/cmd/worker | Processing jobs, Woo/Shopify claim, support AI auto jobs, billing cycles |
|
||||
| apps/web | SvelteKit 2 / Svelte 5 UI (Vite proxies /api to API) |
|
||||
| PostgreSQL 16 | System of record (goose migrations under apps/api/sql/schema) |
|
||||
|
||||
`npm run dev` starts API + web. **Worker is separate** — without it, process jobs and many background syncs stall.
|
||||
|
||||
## Component diagram
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Browser["Browser :28472"] --> Web["SvelteKit apps/web"]
|
||||
Web -->|"/api proxy"| API["Go chi API :28471"]
|
||||
API --> PG[(PostgreSQL)]
|
||||
Worker["cmd/worker"] --> PG
|
||||
API -->|"NOTIFY processing_jobs"| Worker
|
||||
Worker -->|"ClaimNext SKIP LOCKED"| PG
|
||||
Ext["OpenAI / Woo / Shopify / Stripe / SMTP"] -.-> API
|
||||
Ext -.-> Worker
|
||||
```
|
||||
|
||||
## Auth surfaces
|
||||
|
||||
- **Dashboard session:** cookie + CSRF (X-CSRF-Token) under /api/*
|
||||
- **Public API key:** Bearer or X-API-Key under /api/v1 (no CSRF)
|
||||
- **Public tokens:** /api/public/* (export feeds CSV/XML, unsubscribe, brand logos)
|
||||
- **Platform admin:** /api/admin/* after RequirePlatformAdmin (support desk subset for support_staff)
|
||||
|
||||
## Key packages
|
||||
|
||||
- internal/httpapi — routes + middleware
|
||||
- internal/processing — product description jobs
|
||||
- internal/feeds / woocommerce / shopify — ingest and connectors
|
||||
- internal/support — tickets, KB, FAQ/AI auto-reply
|
||||
- internal/billing — plans, credits, Stripe
|
||||
- internal/jobs — enqueue processing (Postgres pending + NOTIFY; River client deferred)
|
||||
|
||||
Sources: README.md, apps/api/internal/httpapi/server.go, apps/api/cmd/worker/main.go.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Configuration and bootstrap environment
|
||||
|
||||
**One root .env** — copy from .env.example. Do **not** create apps/api/.env. Product secrets belong in the dashboard after login.
|
||||
|
||||
## Required bootstrap (names only — never paste real secrets)
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| DATABASE_URL | Postgres (local compose often host port 5433) |
|
||||
| APP_ENV | development / staging / production |
|
||||
| HTTP_ADDR | API listen (dev commonly :28471) |
|
||||
| WEB_ORIGIN | Browser origin for CORS/cookies (:28472 local) |
|
||||
| PUBLIC_API_URL | Public API origin for the web app |
|
||||
| SESSION_SECURE | Cookie Secure; must be true in production |
|
||||
| TOKEN_SIGNING_SECRET | Session/invite HMAC (openssl rand -hex 32) |
|
||||
| APP_ENCRYPTION_KEY | At-rest encryption for BYOK/store secrets (preferred) |
|
||||
|
||||
## Optional bootstrap (safe to override)
|
||||
|
||||
TRUSTED_PROXIES (comma CIDRs/IPs of hop-1 reverse proxies only — enables TrustedRealIP rewrite of RemoteAddr for rate limits; empty = ignore X-Forwarded-For), RATE_LIMIT_REPLICAS (optional; divides HTTP middleware RPM caps when N>1 — still per-process; edge still required for hard global RPM; does not affect lockout/StartLimiter/AI/email), SESSION_COOKIE_NAME, CSRF_COOKIE_NAME, PUBLIC_CSRF_COOKIE_NAME, SESSION_IDLE_HOURS, UPLOAD_DIR, MAINTENANCE_MODE, READ_ONLY_MODE, CREDENTIALS_ENCRYPTION_KEY (legacy alias for APP_ENCRYPTION_KEY), DOTENV_PATH.
|
||||
|
||||
## Prefer dashboard (not root .env)
|
||||
|
||||
| Area | UI |
|
||||
|------|-----|
|
||||
| Stripe, EPREL kill-switch, feed private-URL allowlist | /admin/settings |
|
||||
| Tenant AI | /integrations/ai |
|
||||
| Marketing email | /integrations/email |
|
||||
| Stores | /stores |
|
||||
|
||||
Optional process-env fallbacks still accepted by some resolvers (OPENAI_*, SMTP_*, EPREL_*, FEED_URL_PRIVATE_ALLOWLIST) — prefer UI for day-to-day.
|
||||
|
||||
Production fail-closed (APP_ENV=production): SESSION_SECURE=true, https WEB_ORIGIN, APP_ENCRYPTION_KEY, TOKEN_SIGNING_SECRET, STRIPE_MOCK=false.
|
||||
|
||||
Never commit real secrets. Diagnostics exposes **presence flags** only (*_set), never values.
|
||||
|
||||
Source: root .env.example, README.md Environment section, docs/ops-runtime.md.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Jobs, queues, and integrations
|
||||
|
||||
## Processing queue
|
||||
|
||||
ASSUMPTION in code: full River client is deferred. Production MVP uses **Postgres processing_jobs** with FOR UPDATE SKIP LOCKED + pg_notify('processing_jobs').
|
||||
|
||||
- Enqueue: internal/jobs.Queue.EnqueueProcessingJob
|
||||
- Workers: internal/processing.JobSlots.Fill → ClaimNext (count from config / ClampProcessingWorkers)
|
||||
- Process starts also hit HTTP rate limits (RPM per company)
|
||||
|
||||
## Worker loop (what runs)
|
||||
|
||||
From apps/api/cmd/worker:
|
||||
|
||||
1. Fill processing job slots
|
||||
2. ProcessPendingAutoJobs (support AI fallback)
|
||||
3. WooCommerce / Shopify ClaimNextPendingJob + sync
|
||||
4. Periodic: EnqueueDueScheduled (stores), RunDueBillingCycles
|
||||
|
||||
API also runs a light RunAutoJobsLoop for support AI — keep **worker** in production.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph ingest [Ingest]
|
||||
FeedURL[Feed URL / CSV]
|
||||
Woo[WooCommerce]
|
||||
Shop[Shopify]
|
||||
end
|
||||
subgraph core [Core]
|
||||
Jobs[(processing_jobs)]
|
||||
Worker[cmd/worker]
|
||||
Catalog[(products)]
|
||||
end
|
||||
subgraph out [Outbound]
|
||||
Export[export feeds CSV/XML]
|
||||
StorePush[Woo/Shopify push]
|
||||
end
|
||||
FeedURL --> Catalog
|
||||
Woo --> Catalog
|
||||
Shop --> Catalog
|
||||
Catalog --> Jobs
|
||||
Jobs --> Worker
|
||||
Worker --> Catalog
|
||||
Catalog --> Export
|
||||
Worker --> StorePush
|
||||
```
|
||||
|
||||
## Integrations (where configured)
|
||||
|
||||
| Integration | Preferred config | Notes |
|
||||
|-------------|------------------|-------|
|
||||
| AI (BYOK / OpenAI-compatible) | Tenant /integrations/ai | Encrypted with APP_ENCRYPTION_KEY; optional process OPENAI_* fallback |
|
||||
| Marketing email | /integrations/email | Separate from platform invite SMTP |
|
||||
| Woo / Shopify / feeds | /stores | Woo secrets at rest; Shopify Admin domain SSRF-hardened |
|
||||
| Stripe / EPREL / feed private-URL allowlist | /admin/settings | Env fallbacks exist; prefer UI |
|
||||
| Platform invite SMTP | Process SMTP_* | See docs/ops-runtime.md |
|
||||
|
||||
Support AI auto-reply jobs: table support_auto_jobs → TryAutoReplyLLM after FAQ miss.
|
||||
|
||||
Sources: apps/api/internal/jobs/river.go, apps/api/cmd/worker/main.go, docs/ops-runtime.md.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Security and operational runbook
|
||||
|
||||
Grounded in docs/security-notes.md and docs/ops-runtime.md.
|
||||
|
||||
## Controls in place
|
||||
|
||||
| Area | Control |
|
||||
|------|---------|
|
||||
| CSRF | Double-submit cookie + X-CSRF-Token on dashboard /api/* (skipped for /api/v1, /api/public/*, webhooks) |
|
||||
| Sessions | scs + Postgres store; HttpOnly; idle SESSION_IDLE_HOURS (default 24); absolute 7d |
|
||||
| CORS | Allowlist = WEB_ORIGIN only; credentials allowed |
|
||||
| SSRF | Feed + Woo URL checks; Shopify *.myshopify.com; optional FEED_URL_PRIVATE_ALLOWLIST / settings allowlist |
|
||||
| Uploads | CSV/logo size + type caps under UPLOAD_DIR/{company_id}/ |
|
||||
| AuthZ | Session company context; API key company binding; admin vs support_staff |
|
||||
| Rate limits | Auth POSTs / IP; process/sync/export / company (in-process — not cluster-global) |
|
||||
|
||||
## Ops runbook
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Deploy --> Migrate["scripts/migrate.ps1 / goose up"]
|
||||
Migrate --> API[cmd/api]
|
||||
Migrate --> Worker[cmd/worker]
|
||||
API --> Probes["/healthz /readyz"]
|
||||
Worker --> Probes
|
||||
Probes --> Hypercare["/api/admin/readiness + diagnostics"]
|
||||
```
|
||||
|
||||
1. **Bring up:** Docker Postgres → migrate → API + **worker** → web.
|
||||
2. **Probes:** /healthz no DB; /readyz pings Postgres and reports maintenance/read_only.
|
||||
3. **Maintenance:** MAINTENANCE_MODE / READ_ONLY_MODE — keep probes green during cutover rehearsal.
|
||||
4. **Mail:** Platform invites need SMTP_ENABLED + host/from; tenant marketing mail is separate.
|
||||
5. **Credentials:** Set APP_ENCRYPTION_KEY before storing production Woo/AI secrets; rotating without re-save breaks ciphertext.
|
||||
6. **Stuck jobs:** diagnostics → stuck-cleanup → retry; ensure worker is running.
|
||||
7. **Support auto-reply:** default off (enabled=false); publish KB + raise threshold before enabling FAQ; AI needs support role configured.
|
||||
8. **Never log:** Stripe/OpenAI/SMTP/Woo/Shopify/EPREL secrets.
|
||||
|
||||
## Known residual risks (honest)
|
||||
|
||||
In-process rate limits do not cluster; broad private feed allowlists re-enable SSRF; public export tokens rely on entropy; demo API keys are local-only.
|
||||
|
||||
For cutover blockers and SMTP verification, see docs/ops-runtime.md and docs/production-checklist.md.
|
||||
Reference in New Issue
Block a user