171 lines
16 KiB
JSON
171 lines
16 KiB
JSON
|
{
|
|||
|
|
"version": 1,
|
|||
|
|
"articles": [
|
|||
|
|
{
|
|||
|
|
"slug": "tech-architecture-overview",
|
|||
|
|
"title": "Architecture overview (Go API + SvelteKit)",
|
|||
|
|
"body_md": "# Architecture overview\n\nDescrybe v2 is a **Go chi API + SvelteKit web + PostgreSQL** rewrite (no Clerk).\n\n## Runtime processes\n\n| Process | Role |\n|---------|------|\n| apps/api/cmd/api | HTTP API (sessions, CSRF, public /api/v1, admin) |\n| apps/api/cmd/worker | Processing jobs, Woo/Shopify claim, support AI auto jobs, billing cycles |\n| apps/web | SvelteKit 2 / Svelte 5 UI (Vite proxies /api to API) |\n| PostgreSQL 16 | System of record (goose migrations under apps/api/sql/schema) |\n\n`npm run dev` starts API + web. **Worker is separate** — without it, process jobs and many background syncs stall.\n\n## Component diagram\n\n`mermaid\nflowchart LR\n Browser[\"Browser :28472\"] --> Web[\"SvelteKit apps/web\"]\n Web -->|\"/api proxy\"| API[\"Go chi API :28471\"]\n API --> PG[(PostgreSQL)]\n Worker[\"cmd/worker\"] --> PG\n API -->|\"NOTIFY processing_jobs\"| Worker\n Worker -->|\"ClaimNext SKIP LOCKED\"| PG\n Ext[\"OpenAI / Woo / Shopify / Stripe / SMTP\"] -.-> API\n Ext -.-> Worker\n`\n\n## Auth surfaces\n\n- **Dashboard session:** cookie + CSRF (X-CSRF-Token) under /api/*\n- **Public API key:** Bearer or X-API-Key under /api/v1 (no CSRF)\n- **Public tokens:** /api/public/* (export feeds CSV/XML, unsubscribe, brand logos)\n- **Platform admin:** /api/admin/* after RequirePlatformAdmin (support desk subset for support_staff)\n\n## Key packages\n\n- internal/httpapi — routes + middleware\n- internal/processing — product description jobs\n- internal/feeds / woocommerce / shopify — ingest and connectors\n- internal/support — tickets, KB, FAQ/AI auto-reply\n- internal/billing — plans, credits, Stripe\n- internal/jobs — enqueue processing (Postgres pending + NOTIFY; River client deferred)\n\nSources: README.md, apps/api/internal/httpapi/server.go, apps/api/cmd/worker/main.go.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"architecture",
|
|||
|
|
"operations",
|
|||
|
|
"admin"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"architecture",
|
|||
|
|
"stack",
|
|||
|
|
"worker",
|
|||
|
|
"sveltekit",
|
|||
|
|
"chi",
|
|||
|
|
"postgres",
|
|||
|
|
"overview",
|
|||
|
|
"diagram"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"how_is_descrybe_built",
|
|||
|
|
"what_processes_run",
|
|||
|
|
"architecture_overview"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 100
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"slug": "tech-api-v1-postman-a1",
|
|||
|
|
"title": "Public API /api/v1 and Postman A1 flows",
|
|||
|
|
"body_md": "# Public API and Postman A1\n\nBase URL (local): http://127.0.0.1:28471 \nAuth: Authorization Bearer api_key or X-API-Key \nOpenAPI: GET /api/v1/openapi.yaml (no key). Health: GET /api/v1/health or GET /healthz.\n\n## Core /api/v1 groups (API key)\n\n| Group | Examples |\n|-------|----------|\n| Products | GET /products, GET /products/{id}, PATCH /products/{id}, POST /products/process, GET /products/process/{id} |\n| Feeds | GET/POST /feeds, POST /feeds/{id}/sync, mappings, extract-schema |\n| Categories / attributes | CRUD under /categories, /attributes |\n| Export | /export-feeds plus generate / export-products |\n| Process jobs | POST /process, list/get/cancel/retry |\n| Marketing calendar | /marketing/calendar (legacy /campaigns aliases) |\n\nMounted in apps/api/internal/httpapi/v1.go via mountV1.\n\n## A1 two-EAN Postman flow\n\nCollection: docs/postman/Descrybe-v2-A1-two-EANs.postman_collection.json \nPrerequisite: npm run seed:a1 (restores processing jobs so poll works).\n\n`mermaid\nsequenceDiagram\n participant P as Postman\n participant API as /api/v1\n participant W as worker\n P->>API: GET /health\n P->>API: GET /feeds\n P->>API: GET /products?feed_id=\n P->>API: POST /products/process (2 EANs)\n API-->>P: data.process_id\n loop until done\n P->>API: GET /products/process/{processId}\n end\n Note over W: Worker claims processing_jobs\n P->>API: GET /products?search=EAN\n P->>API: GET /export-feeds\n P->>API: GET /api/public/export-feeds/{token}.csv\n`\n\n### Steps (collection order)\n\n0. Health (no auth) \n1. List feeds (find Elkotex) \n2. List products on feed \n3. **Process two EANs** — processing_type full — copy data.process_id into processId \n4. Poll process status \n5–6. Search results by EAN \n7. Get product by UUID \n8. List export feeds \n9. Public CSV (no API key)\n\nDemo API key and feed/EAN vars live in the Postman collection (local demo only). Full surface: docs/postman/Descrybe-v2-Demo-A1-all-v1.postman_collection.json.\n\nRate limits: process/sync/export POSTs are capped per company (RateLimitV1Process). Prefer enqueue + worker under load.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"api",
|
|||
|
|
"processing",
|
|||
|
|
"integrations"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"api",
|
|||
|
|
"postman",
|
|||
|
|
"ean",
|
|||
|
|
"process",
|
|||
|
|
"openapi",
|
|||
|
|
"v1",
|
|||
|
|
"bearer",
|
|||
|
|
"export",
|
|||
|
|
"feed"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"how_to_use_api",
|
|||
|
|
"postman_a1",
|
|||
|
|
"process_eans_api"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 95
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"slug": "tech-admin-capabilities-diagnostics",
|
|||
|
|
"title": "Admin capabilities and diagnosing issues",
|
|||
|
|
"body_md": "# Admin capabilities and diagnostics\n\nPlatform admins use /admin/* (session + RequirePlatformAdmin). Support desk staff get ticket routes only (RequireSupportDesk).\n\n## Useful admin APIs\n\n| Path | Purpose |\n|------|---------|\n| GET /api/admin/diagnostics | Health: DB, queue, cache, storage, mail; config sanity (booleans only); recent job/AI failures |\n| GET /api/admin/analytics | Operational metrics dashboard data |\n| GET /api/admin/readiness | Cutover hypercare: must_set_password, companies without admin/plan |\n| GET /api/admin/jobs | Recent processing_jobs |\n| POST /api/admin/jobs/stuck-cleanup | Stuck running jobs cleanup |\n| GET/PUT /api/admin/settings | Platform settings (secrets masked on GET) |\n| GET/POST /api/admin/support/kb/articles | Support Knowledge CRUD |\n| GET/PUT /api/admin/support/auto-config | FAQ + AI auto-reply switchboard |\n| Users / companies / plans / credits | Org + billing admin |\n\nUI: /admin/support/knowledge; diagnostics and analytics under the admin shell.\n\n## Diagnose a stuck process\n\n`mermaid\nflowchart TD\n A[Job stuck or failed] --> B{Worker running?}\n B -->|No| C[\"Start: go run ./cmd/worker\"]\n B -->|Yes| D[\"GET /api/admin/diagnostics\"]\n D --> E{queue.failed or stuck_running?}\n E -->|Yes| F[\"GET /api/admin/jobs + stuck-cleanup\"]\n E -->|No| G[\"Check /readyz + company credits\"]\n F --> H[\"Retry job or re-POST /products/process\"]\n`\n\n### Checklist\n\n1. GET /healthz (liveness) and GET /readyz (Postgres + maintenance/read_only flags). \n2. GET /api/admin/diagnostics — overall ok|degraded|fail; never expect secrets in the payload. \n3. Confirm **worker** process is up (API alone does not drain processing). \n4. Filter recent failures: ?status=failed&failures_limit=25. \n5. Support auto AI: ticket auto_reply_status, /admin/support inbox flag=needs_human, AI role support under /admin/settings. \n6. Cutover: GET /api/admin/readiness before DNS switch.\n\nDiagnostics intentionally excludes marketing charts — use analytics for trends, diagnostics for troubleshooting.\n\nSources: apps/api/internal/httpapi/admin_diagnostics_handlers.go, server.go admin routes.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"admin",
|
|||
|
|
"operations",
|
|||
|
|
"bug"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"admin",
|
|||
|
|
"diagnostics",
|
|||
|
|
"readiness",
|
|||
|
|
"stuck",
|
|||
|
|
"jobs",
|
|||
|
|
"healthz",
|
|||
|
|
"readyz",
|
|||
|
|
"troubleshoot"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"how_to_diagnose",
|
|||
|
|
"admin_diagnostics",
|
|||
|
|
"stuck_job"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 90
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"slug": "tech-jobs-queues-integrations",
|
|||
|
|
"title": "Jobs, queues, and integrations",
|
|||
|
|
"body_md": "# Jobs, queues, and integrations\n\n## Processing queue\n\nASSUMPTION in code: full River client is deferred. Production MVP uses **Postgres processing_jobs** with FOR UPDATE SKIP LOCKED + pg_notify('processing_jobs').\n\n- Enqueue: internal/jobs.Queue.EnqueueProcessingJob\n- Workers: internal/processing.JobSlots.Fill → ClaimNext (count from config / ClampProcessingWorkers)\n- Process starts also hit HTTP rate limits (RPM per company)\n\n## Worker loop (what runs)\n\nFrom apps/api/cmd/worker:\n\n1. Fill processing job slots \n2. ProcessPendingAutoJobs (support AI fallback) \n3. WooCommerce / Shopify ClaimNextPendingJob + sync \n4. Periodic: EnqueueDueScheduled (stores), RunDueBillingCycles\n\nAPI also runs a light RunAutoJobsLoop for support AI — keep **worker** in production.\n\n`mermaid\nflowchart TB\n subgraph ingest [Ingest]\n FeedURL[Feed URL / CSV]\n Woo[WooCommerce]\n Shop[Shopify]\n end\n subgraph core [Core]\n Jobs[(processing_jobs)]\n Worker[cmd/worker]\n Catalog[(products)]\n end\n subgraph out [Outbound]\n Export[export feeds CSV/XML]\n StorePush[Woo/Shopify push]\n end\n FeedURL --> Catalog\n Woo --> Catalog\n Shop --> Catalog\n Catalog --> Jobs\n Jobs --> Worker\n Worker --> Catalog\n Catalog --> Export\n Worker --> StorePush\n`\n\n## Integrations (where configured)\n\n| Integration | Preferred config | Notes |\n|-------------|------------------|-------|\n| AI (BYOK / OpenAI-compatible) | Tenant /integrations/ai | Encrypted with APP_ENCRYPTION_KEY; optional process OPENAI_* fallback |\n| Marketing email | /integrations/email | Separate from platform invite SMTP |\n| Woo / Shopify / feeds | /stores | Woo secrets at rest; Shopify Admin domain SSRF-hardened |\n| Stripe / EPREL / feed private-URL allowlist | /admin/settings | Env fallbacks exist; prefer UI |\n| Platform invite SMTP | Process SMTP_* | See docs/ops-runtime.md |\n\nSupport AI auto-reply jobs: table support_auto_jobs → TryAutoReplyLLM after FAQ miss.\n\nSources: apps/api/internal/jobs/river.go, apps/api/cmd/worker/main.go, docs/ops-runtime.md.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"operations",
|
|||
|
|
"integrations",
|
|||
|
|
"processing"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"queue",
|
|||
|
|
"worker",
|
|||
|
|
"jobs",
|
|||
|
|
"woocommerce",
|
|||
|
|
"shopify",
|
|||
|
|
"billing",
|
|||
|
|
"river",
|
|||
|
|
"sync",
|
|||
|
|
"integration"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"how_queues_work",
|
|||
|
|
"worker_required",
|
|||
|
|
"store_sync"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 88
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"slug": "tech-configuration-env",
|
|||
|
|
"title": "Configuration and bootstrap environment",
|
|||
|
|
"body_md": "# Configuration and bootstrap environment\n\n**One root .env** — copy from .env.example. Do **not** create apps/api/.env. Product secrets belong in the dashboard after login.\n\n## Required bootstrap (names only — never paste real secrets)\n\n| Variable | Purpose |\n|----------|---------|\n| DATABASE_URL | Postgres (local compose often host port 5433) |\n| APP_ENV | development / staging / production |\n| HTTP_ADDR | API listen (dev commonly :28471) |\n| WEB_ORIGIN | Browser origin for CORS/cookies (:28472 local) |\n| PUBLIC_API_URL | Public API origin for the web app |\n| SESSION_SECURE | Cookie Secure; must be true in production |\n| TOKEN_SIGNING_SECRET | Session/invite HMAC (openssl rand -hex 32) |\n| APP_ENCRYPTION_KEY | At-rest encryption for BYOK/store secrets (preferred) |\n\n## Optional bootstrap (safe to override)\n\nTRUSTED_PROXIES, SESSION_COOKIE_NAME, CSRF_COOKIE_NAME, PUBLIC_CSRF_COOKIE_NAME, SESSION_IDLE_HOURS, UPLOAD_DIR, MAINTENANCE_MODE, READ_ONLY_MODE, CREDENTIALS_ENCRYPTION_KEY (legacy alias for APP_ENCRYPTION_KEY), DOTENV_PATH.\n\n## Prefer dashboard (not root .env)\n\n| Area | UI |\n|------|-----|\n| Stripe, EPREL kill-switch, feed private-URL allowlist | /admin/settings |\n| Tenant AI | /integrations/ai |\n| Marketing email | /integrations/email |\n| Stores | /stores |\n\nOptional process-env fallbacks still accepted by some resolvers (OPENAI_*, SMTP_*, EPREL_*, FEED_URL_PRIVATE_ALLOWLIST) — prefer UI for day-to-day.\n\nProduction fail-closed (APP_ENV=production): SESSION_SECURE=true, https WEB_ORIGIN, APP_ENCRYPTION_KEY, TOKEN_SIGNING_SECRET, STRIPE_MOCK=false.\n\nNever commit real secrets. Diagnostics exposes **presence flags** only (*_set), never values.\n\nSource: root .env.example, README.md Environment section, docs/ops-runtime.md.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"configuration",
|
|||
|
|
"operations",
|
|||
|
|
"security"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"env",
|
|||
|
|
"configuration",
|
|||
|
|
"DATABASE_URL",
|
|||
|
|
"TOKEN_SIGNING_SECRET",
|
|||
|
|
"APP_ENCRYPTION_KEY",
|
|||
|
|
"WEB_ORIGIN",
|
|||
|
|
"bootstrap"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"how_to_configure",
|
|||
|
|
"env_vars",
|
|||
|
|
"production_settings"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 85
|
|||
|
|
},
|
|||
|
|
{
|
|||
|
|
"slug": "tech-security-ops-runbook",
|
|||
|
|
"title": "Security and operational runbook",
|
|||
|
|
"body_md": "# Security and operational runbook\n\nGrounded in docs/security-notes.md and docs/ops-runtime.md.\n\n## Controls in place\n\n| Area | Control |\n|------|---------|\n| CSRF | Double-submit cookie + X-CSRF-Token on dashboard /api/* (skipped for /api/v1, /api/public/*, webhooks) |\n| Sessions | scs + Postgres store; HttpOnly; idle SESSION_IDLE_HOURS (default 24); absolute 7d |\n| CORS | Allowlist = WEB_ORIGIN only; credentials allowed |\n| SSRF | Feed + Woo URL checks; Shopify *.myshopify.com; optional FEED_URL_PRIVATE_ALLOWLIST / settings allowlist |\n| Uploads | CSV/logo size + type caps under UPLOAD_DIR/{company_id}/ |\n| AuthZ | Session company context; API key company binding; admin vs support_staff |\n| Rate limits | Auth POSTs / IP; process/sync/export / company (in-process — not cluster-global) |\n\n## Ops runbook\n\n`mermaid\nflowchart LR\n Deploy --> Migrate[\"scripts/migrate.ps1 / goose up\"]\n Migrate --> API[cmd/api]\n Migrate --> Worker[cmd/worker]\n API --> Probes[\"/healthz /readyz\"]\n Worker --> Probes\n Probes --> Hypercare[\"/api/admin/readiness + diagnostics\"]\n`\n\n1. **Bring up:** Docker Postgres → migrate → API + **worker** → web. \n2. **Probes:** /healthz no DB; /readyz pings Postgres and reports maintenance/read_only. \n3. **Maintenance:** MAINTENANCE_MODE / READ_ONLY_MODE — keep probes green during cutover rehearsal. \n4. **Mail:** Platform invites need SMTP_ENABLED + host/from; tenant marketing mail is separate. \n5. **Credentials:** Set APP_ENCRYPTION_KEY before storing production Woo/AI secrets; rotating without re-save breaks ciphertext. \n6. **Stuck jobs:** diagnostics → stuck-cleanup → retry; ensure worker is running. \n7. **Support auto-reply:** default off (enabled=false); publish KB + raise threshold before enabling FAQ; AI needs support role configured. \n8. **Never log:** Stripe/OpenAI/SMTP/Woo/Shopify/EPREL secrets.\n\n## Known residual risks (honest)\n\nIn-process rate limits do not cluster; broad private feed allowlists re-enable SSRF; public export tokens rely on entropy; demo API keys are local-only.\n\nFor cutover blockers and SMTP verification, see docs/ops-runtime.md and docs/production-checklist.md.",
|
|||
|
|
"category_slugs": [
|
|||
|
|
"security",
|
|||
|
|
"operations",
|
|||
|
|
"admin"
|
|||
|
|
],
|
|||
|
|
"keywords": [
|
|||
|
|
"security",
|
|||
|
|
"csrf",
|
|||
|
|
"ssrf",
|
|||
|
|
"runbook",
|
|||
|
|
"maintenance",
|
|||
|
|
"encryption",
|
|||
|
|
"rate limit",
|
|||
|
|
"cutover"
|
|||
|
|
],
|
|||
|
|
"intent_keys": [
|
|||
|
|
"security_checklist",
|
|||
|
|
"ops_runbook",
|
|||
|
|
"production_hardening"
|
|||
|
|
],
|
|||
|
|
"is_published": true,
|
|||
|
|
"priority_weight": 92
|
|||
|
|
}
|
|||
|
|
],
|
|||
|
|
"scope": "tech-ops-admin",
|
|||
|
|
"agent": 7
|
|||
|
|
}
|