descrybe/docs/status-and-gaps.md

# Descrybe v2 — status, gaps, and next work

Last updated: 2026-08-08  
Location: `f:/laragon/www/_MY/descrybe-v2`  
Legacy app (still live until cutover): `f:/laragon/www/_MY/descrybe`

This document is the honest gap analysis vs the rewrite plan and the legacy product. Use it to prioritize work. Related docs:

- [features.md](features.md) — phase checklist (kept in sync with this status)
- [schema-map.md](schema-map.md) — MySQL → Postgres ID/field mapping
- [migration-readiness.md](migration-readiness.md) — ETL status + post-import set-password
- [migration-run-log.md](migration-run-log.md) — live MySQL→PG dry-run + load evidence (2026-08-03 **SUCCESS**)
- [go-live-checklist.md](go-live-checklist.md) — staging GO / production NO-GO
- [cutover.md](cutover.md) — production cutover runbook (not yet executed)
- [forgot-password.md](forgot-password.md) — self-serve reset gap + recommended design (deferred)
- [ux-backlog.md](ux-backlog.md) — UX P0/P1; **P0-8 / P0-9 signed cutover waivers** (2026-08-08)
- [production-readiness.md](production-readiness.md) — Wave 8 verdict + remaining blockers

---

## Summary

| Area | Status | Notes |
|------|--------|--------|
| Scaffold / monorepo | **Done** | Go API + SvelteKit + Docker Postgres |
| Auth (no Clerk) | **Mostly done** | Register/login/invite/session/CSRF; **no self-serve forgot-password** — see [forgot-password.md](forgot-password.md) |
| Platform (team, settings, keys, billing) | **Mostly done** | `/readyz`, profile PATCH, invites, SMTP + `cmd/mailhooks`, admin UI, plans/credits; see [ops-runtime.md](ops-runtime.md) |
| Catalog API | **Mostly done** | ListFilter pagination, search/edit, CSV, tree/attr links/formulas; leftovers: drag-drop formula UI, S3 (`UPLOAD_DIR` only), merge-by-GTIN beyond CSV, trees >2000 |
| Feeds / export | **Partial** | URL CSV/XML sync + export streaming; FTP/Excel/schedule/disk leftovers |
| AI processing | **Mostly done** | OpenAI + optional Pinecone; heuristic fallback; DB claim queue; Needs Review MVP (P0-8 [w] — proposal API waived) |
| WooCommerce | **Partial MVP** | REST + worker claim + 15m `EnqueueDueScheduled`; set `CREDENTIALS_ENCRYPTION_KEY` in prod; leftovers: rich maps UI, live E2E |
| Migrator | **Live load succeeded** | Staging PG loaded 2026-08-03 (orphans 16/16); follow-ups: synthetic emails, all `member` roles, `plan_id=6` skipped — see [migration-run-log.md](migration-run-log.md) |
| Public `/api/v1` | **Mounted** | Bearer/`X-API-Key`; CSRF skipped on `/api/v1` only; OpenAPI YAML |
| Admin ops | **Mostly done** | Platform-admin APIs + `/admin` UI + stuck cleanup; polish leftovers |
| Cutover | **Not started** | Data on staging ≠ production ready; **blocked** on emails/roles/SMTP login proof + phase gates in [cutover.md](cutover.md) |
| CI / tests | **Partial** | `go test` unit suite + Makefile; no GitHub Actions |

**Bottom line:** Major workstreams (WS1–WS9) are Done or Partial. **Live MySQL→Postgres migration succeeded on staging** (27 companies, 21 users, 15 memberships, 8278 categories, 57630 attributes, 118784 raw products, 7219 processed, 31 feeds, 8 export feeds). Production cutover remains **blocked** until Clerk emails, membership roles, SMTP/set-password login, and [cutover.md](cutover.md) gates are green.

---

## What is done

### Infrastructure
- [x] Sibling monorepo (`apps/api`, `apps/web`, `docs`, `scripts`)
- [x] `docker-compose.yml` — Postgres 16 on host port `5433`
- [x] Goose migrations `001`–`007` applied on staging PG (platform, catalog, feeds, processing, WooCommerce, feed_sync, …)
- [x] sqlc config + generated queries (handlers often still use raw pgx)
- [x] `cmd/api`, `cmd/worker`, `cmd/migrator`, `cmd/mailhooks` build and run
- [x] Root `Makefile`, `scripts/migrate.sh` / `migrate.ps1`
- [x] SvelteKit 2 + Svelte 5 + Tailwind shell with CSRF-aware `api()` helper

### UI / design
- Visual shell now aligned to legacy design tokens (seasalt / russian-violet / majorelle-blue) across Nav, auth, catalog, and settings; shared `$lib/components/ui` kit + `layout.css`. Not pixel-perfect feature parity with the legacy Next.js UI.

#### Pixel-clone pass (2026-08-03)
Screens cloned into SvelteKit (layout/Nav + page shells against live APIs):

- Dashboard (`/`), Products, Feeds (+ mapping), Export Feeds
- Categories (+ title/description formula builders)
- Attributes, Standard Fields
- Background Tasks (`/processing`; `/tasks` redirects)
- Billing, Settings, WooCommerce, Admin (+ subpages: users, billing/plans, analytics, logs, stuck-products, settings, bootstrap, migrate-orgs, tasks-cleanup)
- Auth: login, register, accept-invite
- `/plans` (customer plans page — admin plans live under `/admin/billing`)
- `/structured-descriptions` (UI cloned; backend APIs still gap)
- `/vector-categories` (UI cloned; backend APIs still gap)

Nav order (legacy-aligned + v2 extras): Products → Feeds → Export Feeds → Categories → Attributes → Standard Fields → Background Tasks → Usage & Billing → Settings → WooCommerce → Admin.

Still missing vs legacy UI (no v2 routes yet): none of the leftover screens above — UI shells exist. Backend gaps remain for structured-descriptions and vector-categories.

### Auth & platform (Phase A — largely working)
- [x] Email/password with argon2id
- [x] Session cookies via `alexedwards/scs` + Postgres `sessions` table
- [x] CSRF double-submit cookie (`descrybe_csrf` + `X-CSRF-Token`)
- [x] Register (creates company + admin membership + credit_balances row)
- [x] Login / logout / `/api/auth/me` / select company / set password
- [ ] Self-serve forgot / reset password (accounts with password already set) — **gap**; design in [forgot-password.md](forgot-password.md). Existing set-password tokens/mail are `must_set_password` / admin-only (P0-10), not reusable safely.
- [x] Accept invite flow (API + Svelte page)
- [x] Team list, create invite, remove member
- [x] Company get/patch + JSON settings
- [x] API keys create/list/revoke (hashed)
- [x] Billing credits overview + `ConsumeCredits` used by processing stub
- [x] `/healthz` + `/readyz` (DB ping; exempt from maintenance gate)
- [x] Profile `PATCH /api/auth/me`; invite list + revoke
- [x] Svelte: login, register, accept-invite, settings (profile/company/team/keys), billing

### Catalog (Phase B — mostly done)
- [x] Categories CRUD + title/description formula PATCH + tree/CSV + attr links
- [x] Attributes CRUD + `ListFilter` SQL pagination/search (shared with products; `ProductFilter` alias)
- [x] Custom variables CRUD
- [x] Products list/search/edit + CSV import (local `UPLOAD_DIR`; merge-by-GTIN in CSV path)
- [x] Migrator paths for categories, attributes, variables, products, feeds (best-effort)

### Feeds / export (Phase C — partial)
- [x] Input feed CRUD + mapping get/put
- [x] Export feed create/list + generate + public XML/CSV streaming
- [x] Real URL sync (CSV/XML) → `raw_products` (FTP/Excel leftover; `SyncStub` aliases `Sync`)

### Processing (Phase D — mostly done)
- [x] Processing jobs create/list/get/cancel + start → `/api/processing/jobs`
- [x] Worker claims pending jobs; OpenAI/heuristic pipeline + optional Pinecone
- [x] Credit consumption hook on processed items

### WooCommerce (Phase E — Partial MVP)
- [x] Config get/put (URL validated; credentials AES-GCM encrypted at rest)
- [x] Real REST connection test (Basic Auth over HTTPS)
- [x] Queued product sync (batch create/update; SKU / product_ids map idempotency)
- [x] Worker claim hook for pending Woo syncs
- [x] Worker 15m cron: billing cycles, stuck jobs, `EnqueueDueScheduled`
- [~] Category/attribute map APIs exist; rich mapping UI / live E2E still open
- **Prod:** set `CREDENTIALS_ENCRYPTION_KEY` (falls back to derived material — prefer explicit key)
- See [ops-runtime.md](ops-runtime.md) for SMTP, mailhooks, encryption, billing

### Docs
- [x] Schema map, cutover runbook, features checklist

---

## What is missing or incomplete

### Critical for any real use (P0)

1. **Feed sync leftovers** (URL+CSV/XML done; not full parity)
   - [x] Download URL, parse CSV/XML, mappings -> upsert `raw_products`, chunked progress
   - [ ] FTP|FTPS / Excel / uploaded-file paths as in legacy

2. **Export generation** — **mostly done** (`internal/feeds/export.go`)
   - [x] Build XML/CSV from processed products + export template (streamed)
   - [x] On-demand generate; public XML/CSV stream real rows
   - [ ] Schedule/cron generation; persist files to object storage

3. **AI processing pipeline** — **mostly done**
   - [x] Categorize -> attributes -> enhance; OpenAI + optional Pinecone; credits/retries
   - [~] River client deferred (DB claim + NOTIFY); tune prompts / enable Pinecone index

4. **Public `/api/v1` surface** — **mounted**
   - [x] Bearer / `X-API-Key`; products/categories/attributes/feeds/export-feeds/process
   - [x] OpenAPI YAML at `/api/v1/openapi.yaml` (Redoc/Scalar UI optional)
   - [ ] Live DB integration test for API-key round-trip (**BLOCKER:** needs `DATABASE_URL` harness)

5. **Migrator production readiness** — **live dry-run + staging load done; cutover follow-ups open**
   - [x] Fixture offline path (`-dry-run -fixture ./cmd/migrator/testdata/fixture.json`); id-map + validation artifacts gitignored
   - [x] `admin_users` -> `is_platform_admin`; feed mappings; count/orphan report; set-password hooks
   - [x] Live MySQL dry-run + staging PG load (2026-08-03) — orphans **16/16 pass**; evidence in [migration-run-log.md](migration-run-log.md)
   - [ ] **BLOCKER (cutover):** synthetic `@legacy.local` emails — export Clerk emails and patch before invites
   - [ ] **BLOCKER (cutover):** all memberships imported as `role=member` — promote company admins
   - [ ] **BLOCKER (cutover):** SMTP + set-password login smoke (this load used `-skip-post-import`)
   - [ ] Data: 2 `company_plans` skipped (`plan_id=6`); api_keys / blobs / full company_settings still not migrated
   - Ops: local PG on host port **5433**; see [cutover.md](cutover.md) / [go-live-checklist.md](go-live-checklist.md)

### Important product gaps (P1)

6. **Catalog leftovers** (WS6 API largely done)
   - [x] Category tree API/CSV + attr links + formula JSON; product search/edit/CSV; ListFilter pagination
   - [ ] Drag-drop formula builder UI (legacy editors)
   - [ ] S3/object storage (local `UPLOAD_DIR` only today)
   - [ ] Merge-by-GTIN beyond CSV import path; category trees >2000 rows

7. **Feeds UX / advanced**
   - Mapping UI (schema extraction, XPath, transforms)
   - Preview APIs (input + export)
   - River (or real queue) instead of poll stub in `jobs/river.go`
   - Schema extraction tasks worker
   - Feed tags, templates, priorities, vendor IDs as in legacy

8. **Billing / usage completeness**
   - [x] Plans / assign / add-credits admin APIs + token packs + billing cycles cron (WS7)
   - [ ] Usage metrics / daily stats APIs polish
   - [ ] Low-credits UX polish
   - Stripe still out of scope unless requested

9. **Platform polish**
   - [x] `/readyz` (DB ping)
   - [x] Profile name update (`PATCH /api/auth/me`)
   - [x] Invite revoke + list pending invites
   - [x] SMTP + `cmd/mailhooks` (operator: `SMTP_ENABLED` + host; run mailhooks after migrator)
   - Session idle policy documented in [ops-runtime.md](ops-runtime.md)

10. **Admin**
    - [x] Platform admin gate on routes (`is_platform_admin`)
    - [x] Admin APIs + `/admin` UI (WS7); polish leftovers
    - [x] Migrate `admin_users` in migrator

11. **WooCommerce**
    - [x] Real REST client + connection test + batched product push (MVP)
    - [x] Worker sync claim hook + 15m `EnqueueDueScheduled`
    - [ ] Rich category/attribute mapping UI / auto-map
    - [ ] Live E2E against a real store
    - **Prod:** set `CREDENTIALS_ENCRYPTION_KEY`

### Engineering / ops gaps (P2)

12. **`packages/migrator` layout** — migrator lives in `apps/api/cmd/migrator` (fine); plan mentioned `packages/migrator` (optional move)
13. **Handlers still raw pgx** — sqlc generated but not fully wired
14. **CI** — [~] Makefile test/vet/check-web + go test ./...; no GitHub Actions yet
15. **Automated tests** — [~] unit auth/CSRF/v1/tenant/migrator; live API-key DB round-trip blocked
16. **Object storage** — uploads/exports files on disk/S3
17. **Observability** — structured logs, request IDs (chi has RequestID), metrics
18. **Deploy** — reverse proxy, TLS, env secrets, worker process supervision
19. **WebSocket / live feed updates** — legacy had WS; not in v2
20. **Cron equivalents** — worker 15m: billing cycles, stuck reset, Woo `EnqueueDueScheduled`; feed sync schedule still open

---

## Stubs to replace (code pointers)

| Stub | File | Replace with |
|------|------|----------------|
| Feed sync | `internal/feeds` `Sync` (was SyncStub) | Done for URL+CSV/XML; FTP/Excel leftover |
| Export XML/CSV | `internal/feeds/export.go` public generators | Done: template XML/CSV stream + generate hook; schedule/disk still open |
| Processing | `internal/processing` pipeline (OpenAI/heuristic + optional Pinecone) | Tune prompts / enable Pinecone index; River client optional later |
| Job queue | `internal/jobs/river.go` | Real River client or keep DB claim + document |
| Woo test/sync | `internal/woocommerce` | Partial MVP done; rich maps UI / live E2E leftover |

---

## Recommended work order

Do not start cutover until P0 items for the phases you actually use in production are green.

```text
1. P0 Clerk email patch + set-password invites + SMTP login smoke (staging data already loaded)
2. P0 promote membership roles; fix skipped plan_id=6 company_plans
3. P0 feed sync leftovers (FTP/Excel) if required for cutover tenants
4. P0 export schedule/disk if required; AI prompt/index tuning
5. P0 /api/v1 API-key re-issue runbook for customers
6. P1 catalog formula UI + S3 leftovers
7. P1 billing / Woo polish if required for cutover
8. P2 CI/tests/deploy
9. Execute cutover.md only after gates green
```

### Phase “done when” (practical gates)

- **Phase A done for cutover:** invites email, `/readyz`, admin flag migrated, API keys work with `/api/v1`
- **Phase B done:** CSV import + product edit/search parity for daily ops
- **Phase C done:** sync + export non-stub for all active feed types
- **Phase D done:** processing quality acceptable vs legacy sample set; credits match expectations
- **Phase E done:** Woo sync if used; admin stuck-job tools; cutover checklist executed

---

## Parity snapshot vs legacy Descrybe

| Legacy capability | v2 today |
|-------------------|----------|
| Clerk auth / orgs | Replaced by first-party auth |
| Dashboard products table + filters | List + ListFilter SQL pagination/search |
| Title/description formula builders | API formulas; no drag-drop UI |
| XML/CSV/Excel/FTP ingest | URL CSV/XML sync done; FTP/Excel open |
| Field mapping + schema extract | Mapping JSON store; no extractor |
| Export feeds builder | Streaming generate + public XML/CSV |
| AI pipeline + queue/cron | OpenAI/heuristic + DB claim worker |
| Pinecone auto-categorize | Optional; needs index config |
| WooCommerce sync | Partial MVP (REST + worker + schedule enqueue) |
| Billing plans/cycles/admin | Plans/credits/token packs + 15m cycles |
| Public v1 API + OpenAPI docs | Mounted + OpenAPI YAML |
| Admin analytics / stuck cleanup | APIs + `/admin` UI (polish open) |
| Org export/import | Missing |
| WebSocket live updates | Missing |

---

## How to run what exists

```bash
cd f:/laragon/www/_MY/descrybe-v2
docker compose up -d
cd apps/api && go run ./cmd/api
# optional worker
go run ./cmd/worker
# web
cd ../web && npm install && npm run dev
```

Migrator: live staging load already done — see [migration-run-log.md](migration-run-log.md). For login rehearsal:

```bash
cd apps/api
export DATABASE_URL="postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
go run ./cmd/migrator -issue-set-password-invites -postgres "$DATABASE_URL" -maps-dir ../../artifacts
# Dev-only bootstrap one user:
go run ./cmd/migrator -set-password "email@example.com:YourPass123" -postgres "$DATABASE_URL"
```

---

## Tracking

Update this file when a gap closes. Keep [features.md](features.md) checkboxes aligned. Do not mark cutover complete until emails/roles/SMTP login and P0 for in-use phases are verified on staging.