Initial commit of Descrybe v2 without local scratch artifacts.

Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
2026-08-09 22:47:43 +02:00
commit 8580c996c3
1285 changed files with 325780 additions and 0 deletions
@@ -0,0 +1,266 @@
+# Descrybe v2 — go-live checklist
+
+**Date:** 2026-08-04  
+**Stack:** Go API + SvelteKit + PostgreSQL (`f:/laragon/www/_MY/descrybe-v2`)  
+**Legacy (still live):** Next.js + Clerk + MySQL (`f:/laragon/www/_MY/descrybe`)
+
+**Verdict: NO-GO for production cutover.** Staging PG has live-migrated data; production DNS/switch stays blocked until emails, membership roles, and SMTP/set-password login are proven.
+
+**Staging rehearsal:** **GO for login testing** after Clerk email patch (or synthetic-email awareness) + set-password invites / `-set-password` bootstrap. See [migration-run-log.md](migration-run-log.md).
+
+**Demo account:** `demo@descrybe.test` / `DemoPass123!` — platform admin of **Platform Demo** only (not A1). A1 catalog stays on `a1-primary@descrybe.local`. Re-seed with `go run ./cmd/seed-demo`. Canonical: [demo-user.md](demo-user.md), [safe-test-fixtures.md](safe-test-fixtures.md).
+
+Synthesized from [status-and-gaps.md](status-and-gaps.md), [design-gaps.md](design-gaps.md), [migration-readiness.md](migration-readiness.md), [cutover.md](cutover.md), [ops-runtime.md](ops-runtime.md), [schema-map.md](schema-map.md), `apps/api/internal/httpapi/server.go`, and `apps/api/cmd/migrator`.
+
+---
+
+## 1. Design parity status
+
+### Cloned / aligned
+
+| Area | Status |
+|------|--------|
+| Design tokens (seasalt / russian-violet / majorelle-blue) | Matched in `layout.css` from legacy globals |
+| Shared UI kit (`$lib/components/ui`) + PageShell | Present |
+| Core nav destinations | Dashboard, Products, Feeds (+ mapping), Export Feeds, Categories (+ title/description formula), Attributes, Standard Fields, Background Tasks, Billing, Settings, WooCommerce, Admin (+ subpages) |
+| Auth shells | `/login`, `/register`, `/accept-invite` |
+| Extra shells | `/plans`, `/structured-descriptions`, `/vector-categories` (routes exist; not all in nav) |
+| Formula builders | Drag-reorder HTML5 DnD present; closer to legacy than status docs sometimes imply |
+
+Nav order (v2): Dashboard → Products → Feeds → Export Feeds → Categories → Attributes → Standard Fields → Background Tasks → Usage & Billing → Settings → **WooCommerce** → **Admin**.
+
+### Still imperfect (do not claim pixel / UX cutover)
+
+From [design-gaps.md](design-gaps.md) — **P0 UI blockers** for daily-ops parity:
+
+1. ~~Feed mapping — no live XML/CSV tree preview / schema extract~~ **DONE** (extract-schema + `FeedSourcePreview` / `MappingPreviewPanel`)
+2. Export create/edit — thin dialog, not full `ExportFeedBuilder` routes — **WAIVED** (`DESIGN-P0-EXPORT-BUILDER-2026-08-08`; dialog covers daily ops)
+3. Products “Export Selected” — client CSV blob, not export-feed picker
+4. “Reset to Unprocessed” — stub toast
+5. ~~Standard Fields — “API not available” banner~~ **DONE** (`/api/standard-fields` + field-groups CRUD mounted)
+6. ~~Product Attributes tab — read-only~~ **DONE (MVP)** (editable key/value → PATCH `attributes`; Add Category Attribute deferred P1)
+7. Attributes — no per-attribute Manage Categories tree dialog
+8. Admin Settings / Admin home — `saveDemo` (UI demo, not persisted)
+9. Admin Logs — no API
+10. Admin always visible in customer sidebar (legacy keeps Admin separate)
+11. Categories Bulk Formula Builder — stub button
+12. ~~No floating TaskStatusIndicator in layout~~ **DONE** (`TaskStatusIndicator` in `+layout.svelte`)
+
+**P1 shells with API banners:** plans, structured-descriptions, vector-categories (UI cloned; backends missing).
+
+**IA differences:** WooCommerce is top-level in v2 (legacy: export-feed dialogs); Admin in customer Nav; path renames (`/register`, `/accept-invite`, `/processing`).
+
+**Honest summary:** Visual shell and most route shells are cloned. Mapping preview, standard-fields API, product attribute edit (MVP), and floating task indicator are closed or waived. Remaining primary blockers for some tenants: export-selected→feed picker, reset-to-unprocessed, admin demos/logs, bulk formula, Manage Categories.
+
+---
+
+## 2. Backend / feature stubs blocking real use
+
+Prefer [status-and-gaps.md](status-and-gaps.md) over [features.md](features.md) — the phase checklist is **stale** in places (still lists feed sync / export / AI / Woo / admin as stubs while code has moved on).
+
+### Hard blockers for many production tenants
+
+| Gap | Detail |
+|-----|--------|
+| Real emails | Most users have synthetic `…@legacy.local` — Clerk emails not in MySQL; patch before invites |
+| Set-password delivery | Hooks tooling exists; SMTP + mailhooks **unproven** on staging |
+| Membership roles | All imported as `role=member` (`profiles.role` absent) — promote via migrator `-list-member-memberships` / `-promote-company-admins` (`-dry-run` then `-confirm`; see [cutover.md](cutover.md)) |
+| `company_plans` | 2 rows skipped (`plan_id=6` missing in `plans`) |
+| API keys migration | `api_keys` / `descrybe_api_key` **not** migrated — clients must mint new keys |
+| Company settings | Language / merge-by-GTIN **not** fully migrated (`company_settings`) |
+| File blobs | Metadata only; no blob copy; `raw_products.file_id` unset |
+
+### Product / API leftovers (in-use phases)
+
+| Area | Works today | Still blocking / incomplete |
+|------|-------------|-----------------------------|
+| Feed sync | URL + CSV/XML → `raw_products` | FTP/FTPS, Excel, uploaded-file paths rejected |
+| Export | Streaming generate + public XML/CSV | Schedule/cron; object-storage persist |
+| Processing | OpenAI + heuristic + optional Pinecone; DB claim queue | River optional; prompt/index tuning |
+| `/api/v1` | Mounted (Bearer / `X-API-Key`); OpenAPI YAML | Live API-key DB round-trip harness not run |
+| WooCommerce | REST test + queued sync + 15m enqueue | Rich maps UI; live store E2E; set `CREDENTIALS_ENCRYPTION_KEY` |
+| Catalog | CRUD, CSV, formulas, ListFilter | S3 (`UPLOAD_DIR` only); trees >2000; merge-by-GTIN beyond CSV |
+| Structured descriptions / vector categories | — | **No backend routes** (UI 404 banners) |
+| Standard fields | `/api/standard-fields`, `/api/field-groups` | **Mounted** (CRUD + bulk enable) |
+| Job queue | DB claim + NOTIFY | River client deferred (`jobs/river.go`) |
+| Org export/import, WebSockets | — | Missing vs legacy |
+
+### Routes mounted (session API — excerpt)
+
+From `server.go`: auth (register/login/logout/accept-invite/complete-set-password/me/set-password), admin (users/companies/jobs/plans/credits/emails), company/team/api-keys/billing, categories/attributes/variables/products, feeds (+ sync/mappings), export-feeds (+ generate), processing jobs, WooCommerce, public export XML/CSV, `/healthz` + `/readyz`, `/api/v1/*`.
+
+**Not mounted:** `/api/structured-descriptions`, `/api/vector-categories/*`.  
+**Mounted:** `/api/standard-fields`, `/api/field-groups` (CRUD + bulk enable).
+
+---
+
+## 3. DB migration status & how to run
+
+### Goose schema (Postgres)
+
+| Item | Status |
+|------|--------|
+| Migrations `001`–`007` | Applied on staging PG (see [migration-run-log.md](migration-run-log.md)) |
+| Apply tooling | `make migrate` / `.\scripts\migrate.ps1` / `scripts/migrate.sh` (pin goose if Go &lt; 1.25.7) |
+| Default local DB | Postgres 16 on host **5433** (`docker compose up -d`) |
+| Default DSN | `postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable` |
+
+```powershell
+# Windows — apply schema + sqlc
+$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
+.\scripts\migrate.ps1
+```
+
+```bash
+export DATABASE_URL="postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
+./scripts/migrate.sh
+# or: make migrate
+```
+
+### MySQL → Postgres ETL (`cmd/migrator`)
+
+| Item | Status |
+|------|--------|
+| Tooling / fixture dry-run | Ready |
+| Live MySQL dry-run | **Done** (2026-08-03) — exit 0 |
+| Live load + validation | **Done** on staging PG — orphans **16/16 pass**; see [migration-run-log.md](migration-run-log.md) |
+| Staging data (written) | 27 companies, 21 users, 15 memberships, 8278 categories, **57630** attributes (unique collapse), **118784** raw products (+7732 GTIN-deduped), 7219 processed, 31 feeds, 8 export feeds |
+| Coverage gaps | api_keys, full company_settings, billing_cycles, feed tags, Woo configs (unless domain `woo`), jobs/history (unless domain `jobs`; cutover default empty), file blob bytes |
+| Auth design | UUID + `legacy_*` + `must_set_password=true` + `admin_users` → `is_platform_admin`; **no password hashes imported** |
+| Post-import | This live run used `-skip-post-import` — re-issue invites when emails are real |
+
+```bash
+cd apps/api
+
+# Offline smoke ONLY (not cutover proof)
+go run ./cmd/migrator -dry-run -fixture ./cmd/migrator/testdata/fixture.json -maps-dir ../../artifacts
+
+# Live dry-run / load (already completed on staging — re-run only if reloading)
+go run ./cmd/migrator \
+  -mysql "$MIGRATE_MYSQL_DSN" \
+  -postgres "$DATABASE_URL" \
+  -dry-run \
+  -maps-dir ../../artifacts \
+  -id-map ../../artifacts/id-map.json
+
+go run ./cmd/migrator \
+  -mysql "$MIGRATE_MYSQL_DSN" \
+  -postgres "$DATABASE_URL" \
+  -maps-dir ../../artifacts \
+  -id-map ../../artifacts/id-map.json
+```
+
+Do **not** invent DSNs. Do **not** commit `artifacts/` (id-map, validation report, set-password tokens).
+
+Full matrix: [migration-readiness.md](migration-readiness.md). Runbook: [cutover.md](cutover.md). Evidence: [migration-run-log.md](migration-run-log.md).
+
+---
+
+## 4. Post-migrate auth (set password)
+
+Migrated users get `password_hash = NULL` and `must_set_password = true`. Clerk sessions do not carry over.
+
+### Operator sequence
+
+1. Prefer real emails first (Clerk export → patch `users.email`); then issue invites (live load used `-skip-post-import`).
+2. Confirm artifact: `<maps-dir>/set-password-hooks.json` (also `password_invites.json`).
+3. Configure SMTP (`SMTP_ENABLED=true`, host/port/user/password/from, `WEB_ORIGIN`). See [ops-runtime.md](ops-runtime.md).
+4. Send mail:
+
+```bash
+cd apps/api
+# Under EMAIL_DRY_RUN (default true): omit -dry-run → exit 1; with -dry-run → subject-only smoke (no SMTP)
+go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json -dry-run   # smoke
+# Real send only after EMAIL_DRY_RUN=false + SMTP_ENABLED=true — see ops-runtime.md
+go run ./cmd/mailhooks -hooks ../../artifacts/set-password-hooks.json            # real send
+```
+
+5. User opens `/accept-invite?token=…` → sets password → login → session cookie.
+6. Alternates: `POST /api/auth/complete-set-password`, admin `POST /api/admin/emails/set-password`, or migrator `-set-password` for local bootstrap.
+7. **Gate:** do not flip DNS until at least one platform admin and one normal user can log in on v2.
+8. Confirm `is_platform_admin` and `/admin` for migrated admins; promote tenant admins from `member` as needed (`-promote-company-admins -dry-run` then `-confirm`).
+
+### Staging login rehearsal (exact commands)
+
+Prefer the no-SMTP link-copy path first: [staging-auth-rehearsal.md](staging-auth-rehearsal.md) (`scripts/staging-auth-rehearsal.ps1` / `.sh`).
+
+```bash
+cd apps/api
+export DATABASE_URL="postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
+
+# Re-issue invites for must_set_password users (Postgres only)
+go run ./cmd/migrator -issue-set-password-invites \
+  -postgres "$DATABASE_URL" \
+  -maps-dir ../../artifacts
+
+# Dev-only: set one user's password directly (not for production cutover)
+go run ./cmd/migrator -set-password "email@example.com:YourPass123" \
+  -postgres "$DATABASE_URL"
+```
+
+### Risks
+
+- SMTP off → users locked out after cutover  
+- Synthetic `@legacy.local` emails until Clerk export  
+- Tenant admins may land as `member`  
+- API customers need **new** API keys after cutover  
+
+---
+
+## 5. Go / no-go for cutover TODAY
+
+### Checklist
+
+| Gate | Status |
+|------|--------|
+| Goose schema applyable locally | **Done** on staging (001–007) |
+| Staging stack runnable (API + worker + web) | Ready for **staging** use |
+| UI visual shell cloned | Mostly yes |
+| UI daily-ops parity (mapping, export builder, product attrs, …) | **Partial** — mapping/standard-fields/attr-edit/task indicator closed or waived; export-selected/reset/admin stubs remain |
+| Feature parity for in-use phases | Partial — FTP/Excel, S3, scheduled export, several APIs missing |
+| Live MySQL migrator dry-run | **Yes** (2026-08-03) |
+| Staging live load + zero orphan FKs | **Yes** — 16/16 orphan checks pass |
+| `migration-run-log.md` / artifacts present | **Yes** (log committed; artifacts gitignored) |
+| Real emails (Clerk) | **No** — synthetic `@legacy.local` |
+| Membership roles verified | **No** — all `member` |
+| SMTP + set-password smoke | **No** (tooling only; post-import skipped this run) |
+| `api_keys` / company_settings migration | **No** |
+| Production secrets (`CREDENTIALS_ENCRYPTION_KEY`, SMTP, `TOKEN_SIGNING_SECRET`) | Operator-owned; not verified here |
+| Cutover runbook executed | **No** |
+| CI / GitHub Actions | Partial unit tests only |
+
+### Recommendation
+
+**NO-GO — do not cut over production traffic.**
+
+Staging PG data load **succeeded**. Use staging for **login rehearsal** after set-password (and preferably after Clerk email patch). Production DNS switch stays blocked until at least:
+
+1. Clerk (or other) email export → patch synthetic addresses; issue set-password invites; SMTP smoke for admin + member.  
+2. Explicit acceptance or fix of migration gaps: roles (`member` → admin), `plan_id=6` company_plans, api_keys re-issue plan, company_settings, file strategy.  
+3. P0 daily-ops UI/API gaps closed **or** signed waiver for tenants that do not need those paths.  
+   - **Done (2026-08-08):** **P0-8** / **P0-9** remainder waived — `UX-P0-8-PROPOSAL-API-2026-08-08`, `UX-P0-9-SERVER-EMAIL-2026-08-08` in [ux-backlog.md](ux-backlog.md) (MVP Needs Review + local Alerts shipped).  
+4. Production secrets + `/readyz` deploy + freeze/rollback owners per [cutover.md](cutover.md).
+
+Until then, keep legacy Descrybe as the live system.
+
+### Suggested next actions (order)
+
+1. Clerk email export → patch `users.email` → `-issue-set-password-invites` → SMTP/mailhooks → login smoke  
+2. Promote tenant admins; resolve skipped `plan_id=6` company_plans  
+3. Close or waive design P0s that match real tenant workflows — **P0-8 / P0-9 waived** (see [ux-backlog.md](ux-backlog.md))  
+4. Re-issue API key runbook for `/api/v1` customers  
+5. Execute [cutover.md](cutover.md) only after gates above are green  
+
+---
+
+## Related docs
+
+- [demo-user.md](demo-user.md) — staging demo login + company data counts  
+- [migration-run-log.md](migration-run-log.md) — live dry-run + load evidence  
+- [status-and-gaps.md](status-and-gaps.md) — product/backend gap analysis  
+- [design-gaps.md](design-gaps.md) — UI/UX parity audit  
+- [migration-readiness.md](migration-readiness.md) — ETL coverage & blockers  
+- [cutover.md](cutover.md) — freeze → migrate → DNS → Clerk decommission  
+- [ops-runtime.md](ops-runtime.md) — SMTP, sessions, Woo encryption  
+- [schema-map.md](schema-map.md) — ID remapping  
+- [features.md](features.md) — phase checkboxes (**may lag** status-and-gaps)