greeneclipse
8580c996c3
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
16 KiB
16 KiB
Descrybe v2  status, gaps, and next work
Last updated: 2026-08-08
Location: f:/laragon/www/_MY/descrybe-v2
Legacy app (still live until cutover): f:/laragon/www/_MY/descrybe
This document is the honest gap analysis vs the rewrite plan and the legacy product. Use it to prioritize work. Related docs:
- features.md — phase checklist (kept in sync with this status)
- schema-map.md — MySQL → Postgres ID/field mapping
- migration-readiness.md — ETL status + post-import set-password
- migration-run-log.md — live MySQL→PG dry-run + load evidence (2026-08-03 SUCCESS)
- go-live-checklist.md — staging GO / production NO-GO
- cutover.md — production cutover runbook (not yet executed)
- forgot-password.md — self-serve reset gap + recommended design (deferred)
- ux-backlog.md — UX P0/P1; P0-8 / P0-9 signed cutover waivers (2026-08-08)
- production-readiness.md — Wave 8 verdict + remaining blockers
Summary
| Area | Status | Notes |
|---|---|---|
| Scaffold / monorepo | Done | Go API + SvelteKit + Docker Postgres |
| Auth (no Clerk) | Mostly done | Register/login/invite/session/CSRF; no self-serve forgot-password — see forgot-password.md |
| Platform (team, settings, keys, billing) | Mostly done | /readyz, profile PATCH, invites, SMTP + cmd/mailhooks, admin UI, plans/credits; see ops-runtime.md |
| Catalog API | Mostly done | ListFilter pagination, search/edit, CSV, tree/attr links/formulas; leftovers: drag-drop formula UI, S3 (UPLOAD_DIR only), merge-by-GTIN beyond CSV, trees >2000 |
| Feeds / export | Partial | URL CSV/XML sync + export streaming; FTP/Excel/schedule/disk leftovers |
| AI processing | Mostly done | OpenAI + optional Pinecone; heuristic fallback; DB claim queue; Needs Review MVP (P0-8 [w] — proposal API waived) |
| WooCommerce | Partial MVP | REST + worker claim + 15m EnqueueDueScheduled; set CREDENTIALS_ENCRYPTION_KEY in prod; leftovers: rich maps UI, live E2E |
| Migrator | Live load succeeded | Staging PG loaded 2026-08-03 (orphans 16/16); follow-ups: synthetic emails, all member roles, plan_id=6 skipped — see migration-run-log.md |
Public /api/v1 |
Mounted | Bearer/X-API-Key; CSRF skipped on /api/v1 only; OpenAPI YAML |
| Admin ops | Mostly done | Platform-admin APIs + /admin UI + stuck cleanup; polish leftovers |
| Cutover | Not started | Data on staging ≠ production ready; blocked on emails/roles/SMTP login proof + phase gates in cutover.md |
| CI / tests | Partial | go test unit suite + Makefile; no GitHub Actions |
Bottom line: Major workstreams (WS1–WS9) are Done or Partial. Live MySQL→Postgres migration succeeded on staging (27 companies, 21 users, 15 memberships, 8278 categories, 57630 attributes, 118784 raw products, 7219 processed, 31 feeds, 8 export feeds). Production cutover remains blocked until Clerk emails, membership roles, SMTP/set-password login, and cutover.md gates are green.
What is done
Infrastructure
- Sibling monorepo (
apps/api,apps/web,docs,scripts) docker-compose.yml Postgres 16 on host port5433- Goose migrations
001–007applied on staging PG (platform, catalog, feeds, processing, WooCommerce, feed_sync, …) - sqlc config + generated queries (handlers often still use raw pgx)
cmd/api,cmd/worker,cmd/migrator,cmd/mailhooksbuild and run- Root
Makefile,scripts/migrate.sh/migrate.ps1 - SvelteKit 2 + Svelte 5 + Tailwind shell with CSRF-aware
api()helper
UI / design
- Visual shell now aligned to legacy design tokens (seasalt / russian-violet / majorelle-blue) across Nav, auth, catalog, and settings; shared
$lib/components/uikit +layout.css. Not pixel-perfect feature parity with the legacy Next.js UI.
Pixel-clone pass (2026-08-03)
Screens cloned into SvelteKit (layout/Nav + page shells against live APIs):
- Dashboard (
/), Products, Feeds (+ mapping), Export Feeds - Categories (+ title/description formula builders)
- Attributes, Standard Fields
- Background Tasks (
/processing;/tasksredirects) - Billing, Settings, WooCommerce, Admin (+ subpages: users, billing/plans, analytics, logs, stuck-products, settings, bootstrap, migrate-orgs, tasks-cleanup)
- Auth: login, register, accept-invite
/plans(customer plans page — admin plans live under/admin/billing)/structured-descriptions(UI cloned; backend APIs still gap)/vector-categories(UI cloned; backend APIs still gap)
Nav order (legacy-aligned + v2 extras): Products → Feeds → Export Feeds → Categories → Attributes → Standard Fields → Background Tasks → Usage & Billing → Settings → WooCommerce → Admin.
Still missing vs legacy UI (no v2 routes yet): none of the leftover screens above — UI shells exist. Backend gaps remain for structured-descriptions and vector-categories.
Auth & platform (Phase A  largely working)
- Email/password with argon2id
- Session cookies via
alexedwards/scs+ Postgressessionstable - CSRF double-submit cookie (
descrybe_csrf+X-CSRF-Token) - Register (creates company + admin membership + credit_balances row)
- Login / logout /
/api/auth/me/ select company / set password - Self-serve forgot / reset password (accounts with password already set) — gap; design in forgot-password.md. Existing set-password tokens/mail are
must_set_password/ admin-only (P0-10), not reusable safely. - Accept invite flow (API + Svelte page)
- Team list, create invite, remove member
- Company get/patch + JSON settings
- API keys create/list/revoke (hashed)
- Billing credits overview +
ConsumeCreditsused by processing stub /healthz+/readyz(DB ping; exempt from maintenance gate)- Profile
PATCH /api/auth/me; invite list + revoke - Svelte: login, register, accept-invite, settings (profile/company/team/keys), billing
Catalog (Phase B — mostly done)
- Categories CRUD + title/description formula PATCH + tree/CSV + attr links
- Attributes CRUD +
ListFilterSQL pagination/search (shared with products;ProductFilteralias) - Custom variables CRUD
- Products list/search/edit + CSV import (local
UPLOAD_DIR; merge-by-GTIN in CSV path) - Migrator paths for categories, attributes, variables, products, feeds (best-effort)
Feeds / export (Phase C — partial)
- Input feed CRUD + mapping get/put
- Export feed create/list + generate + public XML/CSV streaming
- Real URL sync (CSV/XML) →
raw_products(FTP/Excel leftover;SyncStubaliasesSync)
Processing (Phase D — mostly done)
- Processing jobs create/list/get/cancel + start →
/api/processing/jobs - Worker claims pending jobs; OpenAI/heuristic pipeline + optional Pinecone
- Credit consumption hook on processed items
WooCommerce (Phase E — Partial MVP)
- Config get/put (URL validated; credentials AES-GCM encrypted at rest)
- Real REST connection test (Basic Auth over HTTPS)
- Queued product sync (batch create/update; SKU / product_ids map idempotency)
- Worker claim hook for pending Woo syncs
- Worker 15m cron: billing cycles, stuck jobs,
EnqueueDueScheduled - [~] Category/attribute map APIs exist; rich mapping UI / live E2E still open
- Prod: set
CREDENTIALS_ENCRYPTION_KEY(falls back to derived material — prefer explicit key) - See ops-runtime.md for SMTP, mailhooks, encryption, billing
Docs
- Schema map, cutover runbook, features checklist
What is missing or incomplete
Critical for any real use (P0)
-
Feed sync leftovers (URL+CSV/XML done; not full parity)
- Download URL, parse CSV/XML, mappings -> upsert
raw_products, chunked progress - FTP|FTPS / Excel / uploaded-file paths as in legacy
- Download URL, parse CSV/XML, mappings -> upsert
-
Export generation — mostly done (
internal/feeds/export.go)- Build XML/CSV from processed products + export template (streamed)
- On-demand generate; public XML/CSV stream real rows
- Schedule/cron generation; persist files to object storage
-
AI processing pipeline — mostly done
- Categorize -> attributes -> enhance; OpenAI + optional Pinecone; credits/retries
- [~] River client deferred (DB claim + NOTIFY); tune prompts / enable Pinecone index
-
Public
/api/v1surface — mounted- Bearer /
X-API-Key; products/categories/attributes/feeds/export-feeds/process - OpenAPI YAML at
/api/v1/openapi.yaml(Redoc/Scalar UI optional) - Live DB integration test for API-key round-trip (BLOCKER: needs
DATABASE_URLharness)
- Bearer /
-
Migrator production readiness — live dry-run + staging load done; cutover follow-ups open
- Fixture offline path (
-dry-run -fixture ./cmd/migrator/testdata/fixture.json); id-map + validation artifacts gitignored admin_users->is_platform_admin; feed mappings; count/orphan report; set-password hooks- Live MySQL dry-run + staging PG load (2026-08-03) — orphans 16/16 pass; evidence in migration-run-log.md
- BLOCKER (cutover): synthetic
@legacy.localemails — export Clerk emails and patch before invites - BLOCKER (cutover): all memberships imported as
role=member— promote company admins - BLOCKER (cutover): SMTP + set-password login smoke (this load used
-skip-post-import) - Data: 2
company_plansskipped (plan_id=6); api_keys / blobs / full company_settings still not migrated - Ops: local PG on host port 5433; see cutover.md / go-live-checklist.md
- Fixture offline path (
Important product gaps (P1)
-
Catalog leftovers (WS6 API largely done)
- Category tree API/CSV + attr links + formula JSON; product search/edit/CSV; ListFilter pagination
- Drag-drop formula builder UI (legacy editors)
- S3/object storage (local
UPLOAD_DIRonly today) - Merge-by-GTIN beyond CSV import path; category trees >2000 rows
-
Feeds UX / advanced
- Mapping UI (schema extraction, XPath, transforms)
- Preview APIs (input + export)
- River (or real queue) instead of poll stub in
jobs/river.go - Schema extraction tasks worker
- Feed tags, templates, priorities, vendor IDs as in legacy
-
Billing / usage completeness
- Plans / assign / add-credits admin APIs + token packs + billing cycles cron (WS7)
- Usage metrics / daily stats APIs polish
- Low-credits UX polish
- Stripe still out of scope unless requested
-
Platform polish
/readyz(DB ping)- Profile name update (
PATCH /api/auth/me) - Invite revoke + list pending invites
- SMTP +
cmd/mailhooks(operator:SMTP_ENABLED+ host; run mailhooks after migrator) - Session idle policy documented in ops-runtime.md
-
Admin
- Platform admin gate on routes (
is_platform_admin) - Admin APIs +
/adminUI (WS7); polish leftovers - Migrate
admin_usersin migrator
- Platform admin gate on routes (
-
WooCommerce
- Real REST client + connection test + batched product push (MVP)
- Worker sync claim hook + 15m
EnqueueDueScheduled - Rich category/attribute mapping UI / auto-map
- Live E2E against a real store
- Prod: set
CREDENTIALS_ENCRYPTION_KEY
Engineering / ops gaps (P2)
packages/migratorlayout  migrator lives inapps/api/cmd/migrator(fine); plan mentionedpackages/migrator(optional move)- Handlers still raw pgx  sqlc generated but not fully wired
- CI — [~] Makefile test/vet/check-web + go test ./...; no GitHub Actions yet
- Automated tests — [~] unit auth/CSRF/v1/tenant/migrator; live API-key DB round-trip blocked
- Object storage — uploads/exports files on disk/S3
- Observability — structured logs, request IDs (chi has RequestID), metrics
- Deploy — reverse proxy, TLS, env secrets, worker process supervision
- WebSocket / live feed updates — legacy had WS; not in v2
- Cron equivalents — worker 15m: billing cycles, stuck reset, Woo
EnqueueDueScheduled; feed sync schedule still open
Stubs to replace (code pointers)
| Stub | File | Replace with |
|---|---|---|
| Feed sync | internal/feeds Sync (was SyncStub) |
Done for URL+CSV/XML; FTP/Excel leftover |
| Export XML/CSV | internal/feeds/export.go public generators |
Done: template XML/CSV stream + generate hook; schedule/disk still open |
| Processing | internal/processing pipeline (OpenAI/heuristic + optional Pinecone) |
Tune prompts / enable Pinecone index; River client optional later |
| Job queue | internal/jobs/river.go |
Real River client or keep DB claim + document |
| Woo test/sync | internal/woocommerce |
Partial MVP done; rich maps UI / live E2E leftover |
Recommended work order
Do not start cutover until P0 items for the phases you actually use in production are green.
1. P0 Clerk email patch + set-password invites + SMTP login smoke (staging data already loaded)
2. P0 promote membership roles; fix skipped plan_id=6 company_plans
3. P0 feed sync leftovers (FTP/Excel) if required for cutover tenants
4. P0 export schedule/disk if required; AI prompt/index tuning
5. P0 /api/v1 API-key re-issue runbook for customers
6. P1 catalog formula UI + S3 leftovers
7. P1 billing / Woo polish if required for cutover
8. P2 CI/tests/deploy
9. Execute cutover.md only after gates green
Phase “done when†(practical gates)
- Phase A done for cutover: invites email,
/readyz, admin flag migrated, API keys work with/api/v1 - Phase B done: CSV import + product edit/search parity for daily ops
- Phase C done: sync + export non-stub for all active feed types
- Phase D done: processing quality acceptable vs legacy sample set; credits match expectations
- Phase E done: Woo sync if used; admin stuck-job tools; cutover checklist executed
Parity snapshot vs legacy Descrybe
| Legacy capability | v2 today |
|---|---|
| Clerk auth / orgs | Replaced by first-party auth |
| Dashboard products table + filters | List + ListFilter SQL pagination/search |
| Title/description formula builders | API formulas; no drag-drop UI |
| XML/CSV/Excel/FTP ingest | URL CSV/XML sync done; FTP/Excel open |
| Field mapping + schema extract | Mapping JSON store; no extractor |
| Export feeds builder | Streaming generate + public XML/CSV |
| AI pipeline + queue/cron | OpenAI/heuristic + DB claim worker |
| Pinecone auto-categorize | Optional; needs index config |
| WooCommerce sync | Partial MVP (REST + worker + schedule enqueue) |
| Billing plans/cycles/admin | Plans/credits/token packs + 15m cycles |
| Public v1 API + OpenAPI docs | Mounted + OpenAPI YAML |
| Admin analytics / stuck cleanup | APIs + /admin UI (polish open) |
| Org export/import | Missing |
| WebSocket live updates | Missing |
How to run what exists
cd f:/laragon/www/_MY/descrybe-v2
docker compose up -d
cd apps/api && go run ./cmd/api
# optional worker
go run ./cmd/worker
# web
cd ../web && npm install && npm run dev
Migrator: live staging load already done — see migration-run-log.md. For login rehearsal:
cd apps/api
export DATABASE_URL="postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
go run ./cmd/migrator -issue-set-password-invites -postgres "$DATABASE_URL" -maps-dir ../../artifacts
# Dev-only bootstrap one user:
go run ./cmd/migrator -set-password "email@example.com:YourPass123" -postgres "$DATABASE_URL"
Tracking
Update this file when a gap closes. Keep features.md checkboxes aligned. Do not mark cutover complete until emails/roles/SMTP login and P0 for in-use phases are verified on staging.