Initial commit of Descrybe v2 without local scratch artifacts.

Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
2026-08-09 22:47:43 +02:00
commit 8580c996c3
1285 changed files with 325780 additions and 0 deletions
@@ -0,0 +1,45 @@
+# Admin capabilities and diagnostics
+
+Platform admins use /admin/* (session + RequirePlatformAdmin). Support desk staff get ticket routes only (RequireSupportDesk).
+
+## Useful admin APIs
+
+| Path | Purpose |
+|------|---------|
+| GET /api/admin/diagnostics | Health: DB, queue, cache, storage, mail; config sanity (booleans only); recent job/AI failures |
+| GET /api/admin/analytics | Operational metrics dashboard data |
+| GET /api/admin/readiness | Cutover hypercare: must_set_password, companies without admin/plan |
+| GET /api/admin/jobs | Recent processing_jobs |
+| POST /api/admin/jobs/stuck-cleanup | Stuck running jobs cleanup |
+| GET/PUT /api/admin/settings | Platform settings (secrets masked on GET) |
+| GET/POST /api/admin/support/kb/articles | Support Knowledge CRUD |
+| GET/PUT /api/admin/support/auto-config | FAQ + AI auto-reply switchboard |
+| Users / companies / plans / credits | Org + billing admin |
+
+UI: /admin/support/knowledge; diagnostics and analytics under the admin shell.
+
+## Diagnose a stuck process
+
+```mermaid
+flowchart TD
+  A[Job stuck or failed] --> B{Worker running?}
+  B -->|No| C["Start: go run ./cmd/worker"]
+  B -->|Yes| D["GET /api/admin/diagnostics"]
+  D --> E{queue.failed or stuck_running?}
+  E -->|Yes| F["GET /api/admin/jobs + stuck-cleanup"]
+  E -->|No| G["Check /readyz + company credits"]
+  F --> H["Retry job or re-POST /products/process"]
+```
+
+### Checklist
+
+1. GET /healthz (liveness) and GET /readyz (Postgres + maintenance/read_only flags).  
+2. GET /api/admin/diagnostics — overall ok|degraded|fail; never expect secrets in the payload.  
+3. Confirm **worker** process is up (API alone does not drain processing).  
+4. Filter recent failures: ?status=failed&failures_limit=25.  
+5. Support auto AI: ticket auto_reply_status, /admin/support inbox flag=needs_human, AI role support under /admin/settings.  
+6. Cutover: GET /api/admin/readiness before DNS switch.
+
+Diagnostics intentionally excludes marketing charts — use analytics for trends, diagnostics for troubleshooting.
+
+Sources: apps/api/internal/httpapi/admin_diagnostics_handlers.go, server.go admin routes.
@@ -0,0 +1,58 @@
+# Public API and Postman A1
+
+Base URL (local): http://127.0.0.1:28471  
+Auth: Authorization Bearer api_key or X-API-Key  
+OpenAPI: GET /api/v1/openapi.yaml (no key). Health: GET /api/v1/health or GET /healthz.
+
+## Core /api/v1 groups (API key)
+
+| Group | Examples |
+|-------|----------|
+| Products | GET /products, GET /products/{id}, PATCH /products/{id}, POST /products/process, GET /products/process/{id} |
+| Feeds | GET/POST /feeds, POST /feeds/{id}/sync, mappings, extract-schema |
+| Categories / attributes | CRUD under /categories, /attributes |
+| Export | /export-feeds plus generate / export-products |
+| Process jobs | POST /process, list/get/cancel/retry |
+| Marketing calendar | /marketing/calendar (legacy /campaigns aliases) |
+
+Mounted in apps/api/internal/httpapi/v1.go via mountV1.
+
+## A1 two-EAN Postman flow
+
+Collection: docs/postman/Descrybe-v2-A1-two-EANs.postman_collection.json  
+Prerequisite: npm run seed:a1 (restores processing jobs so poll works).
+
+```mermaid
+sequenceDiagram
+  participant P as Postman
+  participant API as /api/v1
+  participant W as worker
+  P->>API: GET /health
+  P->>API: GET /feeds
+  P->>API: GET /products?feed_id=
+  P->>API: POST /products/process (2 EANs)
+  API-->>P: data.process_id
+  loop until done
+    P->>API: GET /products/process/{processId}
+  end
+  Note over W: Worker claims processing_jobs
+  P->>API: GET /products?search=EAN
+  P->>API: GET /export-feeds
+  P->>API: GET /api/public/export-feeds/{token}.csv
+```
+
+### Steps (collection order)
+
+0. Health (no auth)  
+1. List feeds (find Elkotex)  
+2. List products on feed  
+3. **Process two EANs** — processing_type full — copy data.process_id into processId  
+4. Poll process status  
+5–6. Search results by EAN  
+7. Get product by UUID  
+8. List export feeds  
+9. Public CSV (no API key)
+
+Demo API key and feed/EAN vars live in the Postman collection (local demo only). Full surface: docs/postman/Descrybe-v2-Demo-A1-all-v1.postman_collection.json.
+
+Rate limits: process/sync/export POSTs are capped per company (RateLimitV1Process). Prefer enqueue + worker under load.
@@ -0,0 +1,46 @@
+# Architecture overview
+
+Descrybe v2 is a **Go chi API + SvelteKit web + PostgreSQL** rewrite (no Clerk).
+
+## Runtime processes
+
+| Process | Role |
+|---------|------|
+| apps/api/cmd/api | HTTP API (sessions, CSRF, public /api/v1, admin) |
+| apps/api/cmd/worker | Processing jobs, Woo/Shopify claim, support AI auto jobs, billing cycles |
+| apps/web | SvelteKit 2 / Svelte 5 UI (Vite proxies /api to API) |
+| PostgreSQL 16 | System of record (goose migrations under apps/api/sql/schema) |
+
+`npm run dev` starts API + web. **Worker is separate** — without it, process jobs and many background syncs stall.
+
+## Component diagram
+
+```mermaid
+flowchart LR
+  Browser["Browser :28472"] --> Web["SvelteKit apps/web"]
+  Web -->|"/api proxy"| API["Go chi API :28471"]
+  API --> PG[(PostgreSQL)]
+  Worker["cmd/worker"] --> PG
+  API -->|"NOTIFY processing_jobs"| Worker
+  Worker -->|"ClaimNext SKIP LOCKED"| PG
+  Ext["OpenAI / Woo / Shopify / Stripe / SMTP"] -.-> API
+  Ext -.-> Worker
+```
+
+## Auth surfaces
+
+- **Dashboard session:** cookie + CSRF (X-CSRF-Token) under /api/*
+- **Public API key:** Bearer or X-API-Key under /api/v1 (no CSRF)
+- **Public tokens:** /api/public/* (export feeds CSV/XML, unsubscribe, brand logos)
+- **Platform admin:** /api/admin/* after RequirePlatformAdmin (support desk subset for support_staff)
+
+## Key packages
+
+- internal/httpapi — routes + middleware
+- internal/processing — product description jobs
+- internal/feeds / woocommerce / shopify — ingest and connectors
+- internal/support — tickets, KB, FAQ/AI auto-reply
+- internal/billing — plans, credits, Stripe
+- internal/jobs — enqueue processing (Postgres pending + NOTIFY; River client deferred)
+
+Sources: README.md, apps/api/internal/httpapi/server.go, apps/api/cmd/worker/main.go.
@@ -0,0 +1,37 @@
+# Configuration and bootstrap environment
+
+**One root .env** — copy from .env.example. Do **not** create apps/api/.env. Product secrets belong in the dashboard after login.
+
+## Required bootstrap (names only — never paste real secrets)
+
+| Variable | Purpose |
+|----------|---------|
+| DATABASE_URL | Postgres (local compose often host port 5433) |
+| APP_ENV | development / staging / production |
+| HTTP_ADDR | API listen (dev commonly :28471) |
+| WEB_ORIGIN | Browser origin for CORS/cookies (:28472 local) |
+| PUBLIC_API_URL | Public API origin for the web app |
+| SESSION_SECURE | Cookie Secure; must be true in production |
+| TOKEN_SIGNING_SECRET | Session/invite HMAC (openssl rand -hex 32) |
+| APP_ENCRYPTION_KEY | At-rest encryption for BYOK/store secrets (preferred) |
+
+## Optional bootstrap (safe to override)
+
+TRUSTED_PROXIES (comma CIDRs/IPs of hop-1 reverse proxies only — enables TrustedRealIP rewrite of RemoteAddr for rate limits; empty = ignore X-Forwarded-For), RATE_LIMIT_REPLICAS (optional; divides HTTP middleware RPM caps when N>1 — still per-process; edge still required for hard global RPM; does not affect lockout/StartLimiter/AI/email), SESSION_COOKIE_NAME, CSRF_COOKIE_NAME, PUBLIC_CSRF_COOKIE_NAME, SESSION_IDLE_HOURS, UPLOAD_DIR, MAINTENANCE_MODE, READ_ONLY_MODE, CREDENTIALS_ENCRYPTION_KEY (legacy alias for APP_ENCRYPTION_KEY), DOTENV_PATH.
+
+## Prefer dashboard (not root .env)
+
+| Area | UI |
+|------|-----|
+| Stripe, EPREL kill-switch, feed private-URL allowlist | /admin/settings |
+| Tenant AI | /integrations/ai |
+| Marketing email | /integrations/email |
+| Stores | /stores |
+
+Optional process-env fallbacks still accepted by some resolvers (OPENAI_*, SMTP_*, EPREL_*, FEED_URL_PRIVATE_ALLOWLIST) — prefer UI for day-to-day.
+
+Production fail-closed (APP_ENV=production): SESSION_SECURE=true, https WEB_ORIGIN, APP_ENCRYPTION_KEY, TOKEN_SIGNING_SECRET, STRIPE_MOCK=false.
+
+Never commit real secrets. Diagnostics exposes **presence flags** only (*_set), never values.
+
+Source: root .env.example, README.md Environment section, docs/ops-runtime.md.
@@ -0,0 +1,60 @@
+# Jobs, queues, and integrations
+
+## Processing queue
+
+ASSUMPTION in code: full River client is deferred. Production MVP uses **Postgres processing_jobs** with FOR UPDATE SKIP LOCKED + pg_notify('processing_jobs').
+
+- Enqueue: internal/jobs.Queue.EnqueueProcessingJob
+- Workers: internal/processing.JobSlots.Fill → ClaimNext (count from config / ClampProcessingWorkers)
+- Process starts also hit HTTP rate limits (RPM per company)
+
+## Worker loop (what runs)
+
+From apps/api/cmd/worker:
+
+1. Fill processing job slots  
+2. ProcessPendingAutoJobs (support AI fallback)  
+3. WooCommerce / Shopify ClaimNextPendingJob + sync  
+4. Periodic: EnqueueDueScheduled (stores), RunDueBillingCycles
+
+API also runs a light RunAutoJobsLoop for support AI — keep **worker** in production.
+
+```mermaid
+flowchart TB
+  subgraph ingest [Ingest]
+    FeedURL[Feed URL / CSV]
+    Woo[WooCommerce]
+    Shop[Shopify]
+  end
+  subgraph core [Core]
+    Jobs[(processing_jobs)]
+    Worker[cmd/worker]
+    Catalog[(products)]
+  end
+  subgraph out [Outbound]
+    Export[export feeds CSV/XML]
+    StorePush[Woo/Shopify push]
+  end
+  FeedURL --> Catalog
+  Woo --> Catalog
+  Shop --> Catalog
+  Catalog --> Jobs
+  Jobs --> Worker
+  Worker --> Catalog
+  Catalog --> Export
+  Worker --> StorePush
+```
+
+## Integrations (where configured)
+
+| Integration | Preferred config | Notes |
+|-------------|------------------|-------|
+| AI (BYOK / OpenAI-compatible) | Tenant /integrations/ai | Encrypted with APP_ENCRYPTION_KEY; optional process OPENAI_* fallback |
+| Marketing email | /integrations/email | Separate from platform invite SMTP |
+| Woo / Shopify / feeds | /stores | Woo secrets at rest; Shopify Admin domain SSRF-hardened |
+| Stripe / EPREL / feed private-URL allowlist | /admin/settings | Env fallbacks exist; prefer UI |
+| Platform invite SMTP | Process SMTP_* | See docs/ops-runtime.md |
+
+Support AI auto-reply jobs: table support_auto_jobs → TryAutoReplyLLM after FAQ miss.
+
+Sources: apps/api/internal/jobs/river.go, apps/api/cmd/worker/main.go, docs/ops-runtime.md.
@@ -0,0 +1,42 @@
+# Security and operational runbook
+
+Grounded in docs/security-notes.md and docs/ops-runtime.md.
+
+## Controls in place
+
+| Area | Control |
+|------|---------|
+| CSRF | Double-submit cookie + X-CSRF-Token on dashboard /api/* (skipped for /api/v1, /api/public/*, webhooks) |
+| Sessions | scs + Postgres store; HttpOnly; idle SESSION_IDLE_HOURS (default 24); absolute 7d |
+| CORS | Allowlist = WEB_ORIGIN only; credentials allowed |
+| SSRF | Feed + Woo URL checks; Shopify *.myshopify.com; optional FEED_URL_PRIVATE_ALLOWLIST / settings allowlist |
+| Uploads | CSV/logo size + type caps under UPLOAD_DIR/{company_id}/ |
+| AuthZ | Session company context; API key company binding; admin vs support_staff |
+| Rate limits | Auth POSTs / IP; process/sync/export / company (in-process — not cluster-global) |
+
+## Ops runbook
+
+```mermaid
+flowchart LR
+  Deploy --> Migrate["scripts/migrate.ps1 / goose up"]
+  Migrate --> API[cmd/api]
+  Migrate --> Worker[cmd/worker]
+  API --> Probes["/healthz /readyz"]
+  Worker --> Probes
+  Probes --> Hypercare["/api/admin/readiness + diagnostics"]
+```
+
+1. **Bring up:** Docker Postgres → migrate → API + **worker** → web.  
+2. **Probes:** /healthz no DB; /readyz pings Postgres and reports maintenance/read_only.  
+3. **Maintenance:** MAINTENANCE_MODE / READ_ONLY_MODE — keep probes green during cutover rehearsal.  
+4. **Mail:** Platform invites need SMTP_ENABLED + host/from; tenant marketing mail is separate.  
+5. **Credentials:** Set APP_ENCRYPTION_KEY before storing production Woo/AI secrets; rotating without re-save breaks ciphertext.  
+6. **Stuck jobs:** diagnostics → stuck-cleanup → retry; ensure worker is running.  
+7. **Support auto-reply:** default off (enabled=false); publish KB + raise threshold before enabling FAQ; AI needs support role configured.  
+8. **Never log:** Stripe/OpenAI/SMTP/Woo/Shopify/EPREL secrets.
+
+## Known residual risks (honest)
+
+In-process rate limits do not cluster; broad private feed allowlists re-enable SSRF; public export tokens rely on entropy; demo API keys are local-only.
+
+For cutover blockers and SMTP verification, see docs/ops-runtime.md and docs/production-checklist.md.
@@ -0,0 +1,213 @@
+// Command seed-support-kb upserts platform Support Knowledge articles from JSON.
+//
+// Targets table support_kb_articles (migration 032). Idempotent on slug.
+//
+// Usage (from apps/api, DATABASE_URL set or passed):
+//
+//	go run ./cmd/seed-support-kb -postgres "$DATABASE_URL"
+//	go run ./cmd/seed-support-kb -file ../../scripts/seed/support-kb-articles.json
+//	go run ./cmd/seed-support-kb -file ../../scripts/seed/support-kb-articles-tech.json
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"log"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+	"unicode"
+	"unicode/utf8"
+
+	"github.com/jackc/pgx/v5/pgxpool"
+)
+
+const (
+	maxSlugLen      = 120
+	maxTitleLen     = 200
+	maxBodyLen      = 20000
+	maxKeywordLen   = 64
+	maxKeywords     = 40
+	maxIntents      = 20
+	maxCatSlugs     = 20
+)
+
+type seedFile struct {
+	Version  int           `json:"version"`
+	Articles []seedArticle `json:"articles"`
+}
+
+type seedArticle struct {
+	Slug            string   `json:"slug"`
+	Title           string   `json:"title"`
+	BodyMD          string   `json:"body_md"`
+	CategorySlugs   []string `json:"category_slugs"`
+	Keywords        []string `json:"keywords"`
+	IntentKeys      []string `json:"intent_keys"`
+	IsPublished     bool     `json:"is_published"`
+	PriorityWeight  int      `json:"priority_weight"`
+}
+
+func main() {
+	postgresURL := flag.String("postgres", os.Getenv("DATABASE_URL"), "Postgres URL")
+	filePath := flag.String("file", "", "Path to support-kb-articles.json (default: repo scripts/seed/...)")
+	flag.Parse()
+
+	if strings.TrimSpace(*postgresURL) == "" {
+		log.Fatal("-postgres / DATABASE_URL is required")
+	}
+
+	path := strings.TrimSpace(*filePath)
+	if path == "" {
+		path = defaultArticlesPath()
+	}
+
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		log.Fatalf("read %s: %v", path, err)
+	}
+	var sf seedFile
+	if err := json.Unmarshal(raw, &sf); err != nil {
+		log.Fatalf("parse json: %v", err)
+	}
+	if len(sf.Articles) == 0 {
+		log.Fatal("no articles in seed file")
+	}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+	defer cancel()
+
+	pool, err := pgxpool.New(ctx, *postgresURL)
+	if err != nil {
+		log.Fatalf("postgres: %v", err)
+	}
+	defer pool.Close()
+
+	var tableOK bool
+	if err := pool.QueryRow(ctx, `
+		SELECT EXISTS (
+			SELECT 1 FROM information_schema.tables
+			WHERE table_schema = 'public' AND table_name = 'support_kb_articles'
+		)`).Scan(&tableOK); err != nil {
+		log.Fatalf("check table: %v", err)
+	}
+	if !tableOK {
+		log.Fatal("support_kb_articles missing — run goose migrations through 032_support_kb_auto_reply first")
+	}
+
+	upserted := 0
+	for i, a := range sf.Articles {
+		slug, err := normalizeSlug(a.Slug)
+		if err != nil {
+			log.Fatalf("article[%d] slug: %v", i, err)
+		}
+		title := clipRunes(strings.TrimSpace(a.Title), maxTitleLen)
+		body := clipRunes(strings.TrimSpace(a.BodyMD), maxBodyLen)
+		if title == "" || body == "" {
+			log.Fatalf("article[%d] (%s): title and body_md are required", i, slug)
+		}
+		cats := normalizeList(a.CategorySlugs, maxKeywordLen, maxCatSlugs)
+		keywords := normalizeList(a.Keywords, maxKeywordLen, maxKeywords)
+		intents := normalizeList(a.IntentKeys, maxKeywordLen, maxIntents)
+
+		tag, err := pool.Exec(ctx, `
+			INSERT INTO support_kb_articles (
+				slug, title, body_md, category_slugs, keywords, intent_keys,
+				is_published, priority_weight, updated_at
+			) VALUES ($1,$2,$3,$4,$5,$6,$7,$8, now())
+			ON CONFLICT (slug) DO UPDATE SET
+				title = EXCLUDED.title,
+				body_md = EXCLUDED.body_md,
+				category_slugs = EXCLUDED.category_slugs,
+				keywords = EXCLUDED.keywords,
+				intent_keys = EXCLUDED.intent_keys,
+				is_published = EXCLUDED.is_published,
+				priority_weight = EXCLUDED.priority_weight,
+				updated_at = now()`,
+			slug, title, body, cats, keywords, intents, a.IsPublished, a.PriorityWeight)
+		if err != nil {
+			log.Fatalf("upsert %s: %v", slug, err)
+		}
+		if tag.RowsAffected() > 0 {
+			upserted++
+			fmt.Printf("upserted %s (%s)\n", slug, title)
+		}
+	}
+
+	var published, total int64
+	_ = pool.QueryRow(ctx, `SELECT count(*) FROM support_kb_articles`).Scan(&total)
+	_ = pool.QueryRow(ctx, `SELECT count(*) FROM support_kb_articles WHERE is_published`).Scan(&published)
+	fmt.Printf("done: %d articles from file; table total=%d published=%d\n", upserted, total, published)
+}
+
+func defaultArticlesPath() string {
+	// Prefer repo-relative path when run from apps/api.
+	candidates := []string{
+		filepath.Join("..", "..", "scripts", "seed", "support-kb-articles.json"),
+		filepath.Join("scripts", "seed", "support-kb-articles.json"),
+	}
+	if wd, err := os.Getwd(); err == nil {
+		candidates = append(candidates,
+			filepath.Join(wd, "scripts", "seed", "support-kb-articles.json"),
+			filepath.Join(wd, "..", "..", "scripts", "seed", "support-kb-articles.json"),
+		)
+	}
+	for _, c := range candidates {
+		if st, err := os.Stat(c); err == nil && !st.IsDir() {
+			return c
+		}
+	}
+	return candidates[0]
+}
+
+func normalizeSlug(s string) (string, error) {
+	s = strings.ToLower(strings.TrimSpace(s))
+	s = strings.ReplaceAll(s, " ", "-")
+	if s == "" {
+		return "", fmt.Errorf("empty slug")
+	}
+	if utf8.RuneCountInString(s) > maxSlugLen {
+		return "", fmt.Errorf("slug too long")
+	}
+	for _, r := range s {
+		if unicode.IsLetter(r) || unicode.IsDigit(r) || r == '-' || r == '_' {
+			continue
+		}
+		return "", fmt.Errorf("invalid slug char %q", r)
+	}
+	return s, nil
+}
+
+func normalizeList(in []string, maxItem, maxCount int) []string {
+	seen := make(map[string]struct{}, len(in))
+	out := make([]string, 0, len(in))
+	for _, raw := range in {
+		s := strings.ToLower(strings.TrimSpace(raw))
+		if s == "" {
+			continue
+		}
+		s = clipRunes(s, maxItem)
+		if _, ok := seen[s]; ok {
+			continue
+		}
+		seen[s] = struct{}{}
+		out = append(out, s)
+		if len(out) >= maxCount {
+			break
+		}
+	}
+	if out == nil {
+		return []string{}
+	}
+	return out
+}
+
+func clipRunes(s string, max int) string {
+	if max <= 0 || utf8.RuneCountInString(s) <= max {
+		return s
+	}
+	return string([]rune(s)[:max])
+}