Initial commit of Descrybe v2 without local scratch artifacts.

Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
2026-08-09 22:47:43 +02:00
commit 8580c996c3
1285 changed files with 325780 additions and 0 deletions
@@ -0,0 +1,129 @@
+# Migration run log — legacy MySQL → v2 Postgres
+
+**Date:** 2026-08-03 (local)  
+**Status:** **SUCCESS** (dry-run + live load, exit 0)  
+**Elapsed (live):** ~11 minutes
+
+Secrets are masked below. Do not commit `artifacts/` maps or hook tokens.
+
+## Environment
+
+| Item | Value |
+|---|---|
+| Legacy MySQL source | `f:/laragon/www/_MY/descrybe/.env.local` → `DATABASE_URL` |
+| MySQL (masked) | `root:****@tcp(localhost:3306)/descrybe_v2` |
+| MySQL also present | `descrybe_v1`, `descrybe_21_05_2025` (not used this run) |
+| Postgres | Docker Compose service `postgres` (`descrybe-v2-postgres`) |
+| Postgres URL (masked) | `postgres://descrybe:****@localhost:5433/descrybe?sslmode=disable` |
+| Goose | migrations **001–007** applied (see below) |
+| Migrator | `apps/api/cmd/migrator` — `go build ./cmd/migrator` **PASS** |
+
+### Goose notes
+
+- `docker compose up -d` — container healthy on host port **5433**.
+- `scripts/migrate.ps1` failed on this machine: `goose@latest` needs Go ≥ 1.25.7 (runner had 1.25.5); `sqlc` not on PATH.
+- Worked: `go run github.com/pressly/goose/v3/cmd/goose@v3.24.3 -dir sql/schema postgres "$DATABASE_URL" up`
+- Fixed **UTF-8 BOM** on `apps/api/sql/schema/006_feed_sync.sql` (goose parse error `\ufeff-- +goose Up`).
+
+## Migrator fixes applied this session
+
+1. **`profiles.role` missing** — memberships default to `member` (Clerk held org roles).
+2. **`profiles.email` / no `users` table** — synthetic `…@legacy.local` emails; enrich from `admin_users` when present.
+3. **`companies.language` missing** — load from `company_settings` / default `en`.
+4. **MySQL DSN** — accept `mysql://…` URLs; normalize to `user:pass@tcp(host:port)/db?parseTime=true`; clearer ping/open errors with Laragon hints (passwords masked in logs).
+5. **`raw_products.feed_ids`** — optional on MySQL (column absent); PG still gets `NULL` / jsonb.
+6. **GTIN unique index** — dedupe `(company_id, gtin)` before insert; report `raw_products_gtin_deduped`.
+7. **Dry-run `planMap`** — populate with legacy plan IDs so `company_plans` linkage is counted.
+
+## Dry-run
+
+```text
+go run ./cmd/migrator -mysql "root:****@tcp(localhost:3306)/descrybe_v2?parseTime=true" \
+  -postgres "postgres://descrybe:****@localhost:5433/descrybe?sslmode=disable" \
+  -dry-run -maps-dir ../../artifacts -id-map ../../artifacts/id-map.json
+```
+
+| Entity | Count |
+|---|---:|
+| companies | 27 |
+| users | 21 |
+| memberships | 15 |
+| admin_users (flagged) | 1 |
+| plans | 5 |
+| company_plans | 3 (+2 skipped: orphan `plan_id=6`) |
+| credit_balances | 5 (+1 skipped: orphan company `…-0`) |
+| categories | 8278 |
+| attributes | 95292 |
+| category_attributes | 74377 |
+| custom_variables | 36 |
+| input_feeds (xml) | 31 |
+| feed_mappings | 24 |
+| raw_products | 126516 |
+| processed_products | 7219 |
+| export_feeds | 8 |
+| files | 1 |
+| Validation | `ok=true` (orphan checks skipped in dry-run) |
+
+## Live load
+
+Same command without `-dry-run` (used `-skip-post-import` this run). Artifacts under `artifacts/` (gitignored).
+
+### Migration report (attempts / written)
+
+| Entity | Report | Notes |
+|---|---:|---|
+| companies | 27 | PG total 29 (includes 2 pre-existing seed rows) |
+| users | 21 | PG total 23 (seed + migrate); **no password hashes imported** |
+| memberships | 15 | 6 legacy profiles have `company_id` NULL → skipped |
+| admin_users | 1 | → `users.is_platform_admin` |
+| plans | 5 | |
+| company_plans | 3 | 2 skipped (`plan_id=6` missing in `plans`) |
+| credit_balances | 5 | 1 skipped (orphan company id) |
+| categories | 8278 | PG 8279 (+1 seed) |
+| attributes | 95292 | PG **57630** = MySQL `COUNT(DISTINCT company_id, attribute_key)` (unique constraint collapse) |
+| category_attributes | 74377 | |
+| custom_variables | 36 | PG 37 (+1 seed) |
+| input_feeds | 31 | PG 33 (+2 seed) |
+| feed_mappings | 24 | |
+| raw_products | 118784 | + **7732** GTIN-deduped (= 126516 MySQL rows) |
+| processed_products | 7219 | |
+| export_feeds | 8 | PG 9 (+1 seed) |
+| files | 1 | metadata only; blobs not copied |
+
+### Validation (live)
+
+- `mode: live ok=true`
+- Orphan FK checks: **16 passed / 0 failed** (companies, memberships, feeds, products, mappings, platform admin, etc.)
+
+## Blockers / follow-ups
+
+| Severity | Item | Action needed |
+|---|---|---|
+| **Data** | Most users have synthetic emails (`user_…@legacy.local`) — Clerk emails are not in MySQL | Export emails from Clerk (or another source) and patch `users.email` before set-password invites |
+| **Data** | All memberships imported as `role=member` (`profiles.role` absent) | Promote company admins post-cutover |
+| **Data** | `company_plans` rows with `plan_id=6` skipped | Create missing plan or remap those companies |
+| **Data** | 1 orphan credit balance (`company_id` suffix `-0`) | Ignore or repair source row |
+| **Ops** | Attribute / GTIN collapses are intentional under v2 unique constraints | Accept PG counts; ID map still covers remapped FKs |
+| **Ops** | Goose pin / BOM / sqlc PATH | Prefer pinned goose ≤ Go toolchain; keep schema files BOM-free; install sqlc if regenerating queries |
+| **Ops** | Set-password hooks | Re-run without `-skip-post-import` (or `-issue-set-password-invites`) when ready to email invites |
+
+## CLI verification
+
+```text
+cd apps/api
+go build ./cmd/migrator          # PASS
+go test ./cmd/migrator -count=1  # PASS
+```
+
+Offline smoke (no MySQL):
+
+```text
+go run ./cmd/migrator -dry-run -fixture ./cmd/migrator/testdata/fixture.json -maps-dir ../../artifacts
+```
+
+## How to re-run
+
+1. `docker compose up -d` in `descrybe-v2`
+2. Apply goose (`goose@v3.24.3` or newer matching local Go)
+3. Set DSN from legacy `.env.local` (convert `mysql://` → go-sql-driver form, or pass URL — migrator converts)
+4. Dry-run, then live; reuse `artifacts/id-map.json` for idempotent remaps