descrybe/docs/portable-mysql-pg-migration.md

# Portable MySQL → PostgreSQL migration

Clerk-free ETL from legacy Descrybe MySQL into Descrybe v2 Postgres. Extends `apps/api/cmd/migrator` in place (no `_v2` duplicate).

## What is migrated

| Domain (`-domains`) | Sources → targets |
|---|---|
| `identity` | companies, users (email only), memberships, platform admins from `admin_users` |
| `billing` | plans, company_plans, credit_balances |
| `catalog` | categories (+ title/description templates), attributes, category_attributes, custom_variables |
| `feeds` | xml_feeds → input_feeds + feed_mappings, export_feeds |
| `products` | raw_products (COPY batches), processed_products |
| `files` | files **metadata only** (blobs resynced separately) |
| `settings` | company_settings → `company_settings.settings` JSONB |
| `formulas` | field_groups, standard_fields, structured_description_fields |
| `tags` | feed_tags, feed_tag_mappings |
| `woo` | `wc_*` custom_fields → woocommerce_configs (when present) |
| `usage` | usage_metrics / usage_limits → `settings._legacy_usage*` (no v2 usage tables) |
| `jobs` | `processing_jobs` (+ best-effort `processing_job_products`), `tasks` — tagged `ai_provider_mode=migrated` so 30-day retention keeps them |

### Mapping / standard-field notes (A1)

- Feed `field_mappings` object maps are stored as `[{key, mapping}, …]`; migrator also rewrites legacy `fieldName` values (`name`→`title`, `purchasePrice`→`purchase_price`, …).
- Migrated company standard fields keep dump keys; the mapping UI merges the full ecommerce catalog (price/sku/…) and resolves targets against company keys so required chips / dropdowns stay correct.
- After import, open **Standard Fields → Enable recommended** (or open any feed Map page) once so `EnsureEcommerceCatalog` fills gaps for mapping/forms.

**Also excluded:** Clerk API / Clerk user sync, legacy password hashes, API key secrets, file **blob bytes** (metadata only via domain `files`). Job history is **optional**: requires domain `jobs` (included in `all`); cutover docs treat empty Processing history as the accepted default unless ops explicitly ran `jobs`.

Users are matched by **email**. Synthetic `…@legacy.local` addresses may appear when profiles lacked email. Local/dev: set a password with migrator `-set-password email:password`, or Admin → Users → **Set local password** / **Switch to user** (non-production only).

## Demo account

After import, create/login as:

| Field | Value |
|-------|--------|
| Email | `demo@descrybe.local` |
| Password | `DemoPass123!` |

(`seed-demo` historically used `demo@descrybe.test` with the same password — both work if seeded.)

```powershell
cd apps/api
$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
go run ./cmd/migrator -postgres $env:DATABASE_URL -ensure-demo `
  -demo-email demo@descrybe.local -demo-password 'DemoPass123!'
# Or full seed (Enterprise plan + demo API key):
go run ./cmd/seed-demo -postgres $env:DATABASE_URL `
  -email demo@descrybe.local -password 'DemoPass123!'
```

`-ensure-demo` makes the user platform admin, admin of **every** company, and renames the richest catalog tenant to **Local Demo Co**.

## Prerequisites

1. Goose migrations applied on Postgres (`001`+).
2. MySQL reachable (Laragon) **or** use `-fixture` offline dry-run.
3. Do **not** commit DSNs, `artifacts/`, id-maps, or invite tokens.

## CLI flags

| Flag | Env | Purpose |
|------|-----|---------|
| `-mysql` | `MIGRATE_MYSQL_DSN` | MySQL DSN (`user:pass@tcp(host:3306)/db?parseTime=true` or `mysql://…`) |
| `-postgres` | `DATABASE_URL` | Postgres URL |
| `-dry-run` | | Count/remap without writes |
| `-resume` | | Reuse `id-map.json` UUIDs (default **true**) |
| `-company` | | Comma-separated **legacy** company ids |
| `-domains` | | `all` or subset listed above (includes `jobs`) |
| `-maps-dir` | | id-map + validation artifacts (gitignored) |
| `-report-dir` | | JSON reports (default = maps-dir); also copies into `docs/migration-reports/` when that folder exists |
| `-id-map` | | Unified id-map path |
| `-fixture` | | Offline dry-run JSON |
| `-ensure-demo` | | Upsert demo user |
| `-demo-email` / `-demo-password` | | Demo credentials (local only) |
| `-skip-post-import` | | Skip set-password invite generation |
| `-set-password` | | `email:password` bootstrap (Postgres only) |
| `-issue-set-password-invites` | | Re-issue invites (Postgres only) |
| `-list-legacy-emails` | | List users still on `…@legacy.local` (Postgres only; read-only) |
| `-export-legacy-emails` | | Write inventory + emails stub map (`-emails-out` or `<maps-dir>/legacy-emails.json`) |
| `-patch-emails` | | Apply Clerk/real emails from `-emails-file` (requires `-dry-run` or `-confirm`) |
| `-emails-file` | | JSON/CSV map for `-patch-emails` (Clerk id → email) |
| `-emails-out` | | Optional export path for `-export-legacy-emails` |
| `-confirm` | | Required for live mutating repair flags (plans / memberships / email patch) |

## Clerk / `@legacy.local` email repair (cutover data hygiene)

Migrated users without a MySQL email become `user_<clerkId>@legacy.local`. Set-password invites **skip** those addresses (`IsSyntheticLegacyEmail`). Repair **before** `-issue-set-password-invites`.

Safety:

- List/export are read-only.
- Patch requires `-dry-run` (preview) or `-confirm` (live write) — same gate as other repair flags.
- Live `UPDATE` only touches rows whose **current** email still ends with `@legacy.local` (real A1/live emails are never overwritten).
- Skips target collisions, empty/placeholder emails, and mappings with no matching synthetic user.
- Do **not** commit Clerk exports, `legacy-emails.json`, or DSNs.

```powershell
cd apps/api
$pg = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
$maps = "..\..\artifacts"

# 1) Find
go run ./cmd/migrator -postgres $pg -list-legacy-emails

# 2) Export inventory + empty emails{} stub (fill from Clerk Dashboard export; no secrets in git)
go run ./cmd/migrator -postgres $pg -export-legacy-emails -maps-dir $maps
# → artifacts/legacy-emails.json

# 3) Dry-run patch (Clerk JSON/CSV or filled emails map)
go run ./cmd/migrator -postgres $pg -patch-emails -emails-file $maps\clerk-users.json -dry-run

# 4) Live patch (explicit confirm)
go run ./cmd/migrator -postgres $pg -patch-emails -emails-file $maps\clerk-users.json -confirm

# 5) Re-issue invites only after emails are real
go run ./cmd/migrator -postgres $pg -issue-set-password-invites -maps-dir $maps
```

Accepted `-emails-file` shapes (no API secrets):

- `{ "user_xxx": "real@example.com" }` or inventory `{ "emails": { … } }`
- JSON array: `{ "id", "email" | "primary_email_address" }` (Clerk-style `email_addresses` OK)
- CSV with `id`/`legacy_user_id` + `email`/`primary_email_address` columns

Then continue with [migration-readiness.md](migration-readiness.md) / [cutover.md](cutover.md).

## Local cutover rehearsal

```powershell
cd f:\laragon\www\_MY\descrybe-v2\apps\api

# Build
go build -o $env:TEMP\descrybe-migrator.exe ./cmd/migrator
go build ./...

# Resolve MySQL from legacy .env.local WITHOUT printing secrets
# Preferred: set MIGRATE_MYSQL_DSN yourself from Laragon.

$pg = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
$maps = "f:\laragon\www\_MY\descrybe-v2\artifacts"

# Offline dry-run (no MySQL)
go run ./cmd/migrator -fixture ./cmd/migrator/testdata/fixture.json -dry-run -maps-dir $maps

# Live dry-run
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg -dry-run `
  -maps-dir $maps -report-dir $maps -resume

# Gap fill only (after a prior full load)
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -domains settings,formulas,tags,woo,usage `
  -maps-dir $maps -report-dir $maps -resume -skip-post-import -ensure-demo

# Full live load (idempotent upserts; products use COPY batches)
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -maps-dir $maps -report-dir $maps -resume -ensure-demo
```

Single-tenant rehearsal:

```powershell
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -company 97e1a309-3d23-4aa2-b518-8e8d7afdfec7 `
  -maps-dir $maps -resume -ensure-demo
```

## Reports

After each run:

- `<report-dir>/migration-report.json` — latest counts, domains, validation, demo email (password **not** stored)
- `<report-dir>/migration-report-<UTC>.json` — stamped copy
- `docs/migration-reports/migration-report-latest.json` — when the docs folder exists
- `<maps-dir>/validation-report.json`, `id-map.json`, per-entity maps

## Postgres optimizations

- **FK order:** companies/users → catalog/feeds → products → export → gap domains
- **Batches:** attributes & category_attributes via `pgx.Batch`; raw_products via `COPY` with per-row upsert fallback
- **Transactions:** per product batch; usage merge uses a single batch transaction
- **Idempotent:** natural-key `ON CONFLICT` upserts + `-resume` id-map reuse
- **Filters:** `-company` pushes `IN (...)` into MySQL SELECTs (no N+1)


## Id-map hygiene

- Use a dedicated -maps-dir for live loads (e.g. rtifacts/). **Do not** point fixture dry-runs at the same directory afterward — fixture remaps can drift UUIDs.
- On every live run the migrator **reconciles** companies.legacy_company_id → Postgres id so gap-only domains (-domains settings,formulas,...) still resolve FKs even if the id-map drifted.
- Rebuild maps from Postgres if needed: SELECT legacy_company_id, id FROM companies WHERE legacy_company_id IS NOT NULL.

## Security notes

- Never import Clerk sessions or password hashes (`must_set_password=true` for migrated users).
- Do not commit `.env`, DSNs, `artifacts/`, or invite tokens.
- Woo consumer secrets are copied when present in legacy custom fields — treat Postgres as secret-bearing after woo migration.

## Related docs

- [migration-readiness.md](migration-readiness.md)
- [migration-run-log.md](migration-run-log.md)
- [demo-user.md](demo-user.md)
- [cutover.md](cutover.md)