descrybe/docs/portable-mysql-pg-migration.md

Portable MySQL → PostgreSQL migration

Clerk-free ETL from legacy Descrybe MySQL into Descrybe v2 Postgres. Extends apps/api/cmd/migrator in place (no _v2 duplicate).

What is migrated

Domain (-domains)	Sources → targets
identity	companies, users (email only), memberships, platform admins from admin_users
billing	plans, company_plans, credit_balances
catalog	categories (+ title/description templates), attributes, category_attributes, custom_variables
feeds	xml_feeds → input_feeds + feed_mappings, export_feeds
products	raw_products (COPY batches), processed_products
files	files metadata only (blobs resynced separately)
settings	company_settings → company_settings.settings JSONB
formulas	field_groups, standard_fields, structured_description_fields
tags	feed_tags, feed_tag_mappings
woo	wc_* custom_fields → woocommerce_configs (when present)
usage	usage_metrics / usage_limits → settings._legacy_usage* (no v2 usage tables)
jobs	processing_jobs (+ best-effort processing_job_products), tasks — tagged ai_provider_mode=migrated so 30-day retention keeps them

Mapping / standard-field notes (A1)

• Feed field_mappings object maps are stored as [{key, mapping}, …]; migrator also rewrites legacy fieldName values (name→title, purchasePrice→purchase_price, …).
• Migrated company standard fields keep dump keys; the mapping UI merges the full ecommerce catalog (price/sku/…) and resolves targets against company keys so required chips / dropdowns stay correct.
• After import, open Standard Fields → Enable recommended (or open any feed Map page) once so EnsureEcommerceCatalog fills gaps for mapping/forms.

Also excluded: Clerk API / Clerk user sync, legacy password hashes, API key secrets, file blob bytes (metadata only via domain files). Job history is optional: requires domain jobs (included in all); cutover docs treat empty Processing history as the accepted default unless ops explicitly ran jobs.

Users are matched by email. Synthetic …@legacy.local addresses may appear when profiles lacked email. Local/dev: set a password with migrator -set-password email:password, or Admin → Users → Set local password / Switch to user (non-production only).

Demo account

After import, create/login as:

Field	Value
Email	demo@descrybe.local
Password	DemoPass123!

(seed-demo historically used demo@descrybe.test with the same password — both work if seeded.)

cd apps/api
$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
go run ./cmd/migrator -postgres $env:DATABASE_URL -ensure-demo `
  -demo-email demo@descrybe.local -demo-password 'DemoPass123!'
# Or full seed (Enterprise plan + demo API key):
go run ./cmd/seed-demo -postgres $env:DATABASE_URL `
  -email demo@descrybe.local -password 'DemoPass123!'

-ensure-demo makes the user platform admin, admin of every company, and renames the richest catalog tenant to Local Demo Co.

Prerequisites

1. Goose migrations applied on Postgres (001+).
2. MySQL reachable (Laragon) or use -fixture offline dry-run.
3. Do not commit DSNs, artifacts/, id-maps, or invite tokens.

CLI flags

Flag	Env	Purpose
-mysql	MIGRATE_MYSQL_DSN	MySQL DSN (user:pass@tcp(host:3306)/db?parseTime=true or mysql://…)
-postgres	DATABASE_URL	Postgres URL
-dry-run		Count/remap without writes
-resume		Reuse id-map.json UUIDs (default true)
-company		Comma-separated legacy company ids
-domains		all or subset listed above (includes jobs)
-maps-dir		id-map + validation artifacts (gitignored)
-report-dir		JSON reports (default = maps-dir); also copies into docs/migration-reports/ when that folder exists
-id-map		Unified id-map path
-fixture		Offline dry-run JSON
-ensure-demo		Upsert demo user
-demo-email / -demo-password		Demo credentials (local only)
-skip-post-import		Skip set-password invite generation
-set-password		email:password bootstrap (Postgres only)
-issue-set-password-invites		Re-issue invites (Postgres only)
-list-legacy-emails		List users still on …@legacy.local (Postgres only; read-only)
-export-legacy-emails		Write inventory + emails stub map (-emails-out or <maps-dir>/legacy-emails.json)
-patch-emails		Apply Clerk/real emails from -emails-file (requires -dry-run or -confirm)
-emails-file		JSON/CSV map for -patch-emails (Clerk id → email)
-emails-out		Optional export path for -export-legacy-emails
-confirm		Required for live mutating repair flags (plans / memberships / email patch)

Clerk / @legacy.local email repair (cutover data hygiene)

Migrated users without a MySQL email become user_<clerkId>@legacy.local. Set-password invites skip those addresses (IsSyntheticLegacyEmail). Repair before -issue-set-password-invites.

Safety:

• List/export are read-only.
• Patch requires -dry-run (preview) or -confirm (live write) — same gate as other repair flags.
• Live UPDATE only touches rows whose current email still ends with @legacy.local (real A1/live emails are never overwritten).
• Skips target collisions, empty/placeholder emails, and mappings with no matching synthetic user.
• Do not commit Clerk exports, legacy-emails.json, or DSNs.

cd apps/api
$pg = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
$maps = "..\..\artifacts"

# 1) Find
go run ./cmd/migrator -postgres $pg -list-legacy-emails

# 2) Export inventory + empty emails{} stub (fill from Clerk Dashboard export; no secrets in git)
go run ./cmd/migrator -postgres $pg -export-legacy-emails -maps-dir $maps
# → artifacts/legacy-emails.json

# 3) Dry-run patch (Clerk JSON/CSV or filled emails map)
go run ./cmd/migrator -postgres $pg -patch-emails -emails-file $maps\clerk-users.json -dry-run

# 4) Live patch (explicit confirm)
go run ./cmd/migrator -postgres $pg -patch-emails -emails-file $maps\clerk-users.json -confirm

# 5) Re-issue invites only after emails are real
go run ./cmd/migrator -postgres $pg -issue-set-password-invites -maps-dir $maps

Accepted -emails-file shapes (no API secrets):

• { "user_xxx": "real@example.com" } or inventory { "emails": { … } }
• JSON array: { "id", "email" | "primary_email_address" } (Clerk-style email_addresses OK)
• CSV with id/legacy_user_id + email/primary_email_address columns

Then continue with migration-readiness.md / cutover.md.

Local cutover rehearsal

cd f:\laragon\www\_MY\descrybe-v2\apps\api

# Build
go build -o $env:TEMP\descrybe-migrator.exe ./cmd/migrator
go build ./...

# Resolve MySQL from legacy .env.local WITHOUT printing secrets
# Preferred: set MIGRATE_MYSQL_DSN yourself from Laragon.

$pg = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
$maps = "f:\laragon\www\_MY\descrybe-v2\artifacts"

# Offline dry-run (no MySQL)
go run ./cmd/migrator -fixture ./cmd/migrator/testdata/fixture.json -dry-run -maps-dir $maps

# Live dry-run
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg -dry-run `
  -maps-dir $maps -report-dir $maps -resume

# Gap fill only (after a prior full load)
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -domains settings,formulas,tags,woo,usage `
  -maps-dir $maps -report-dir $maps -resume -skip-post-import -ensure-demo

# Full live load (idempotent upserts; products use COPY batches)
go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -maps-dir $maps -report-dir $maps -resume -ensure-demo

Single-tenant rehearsal:

go run ./cmd/migrator -mysql $env:MIGRATE_MYSQL_DSN -postgres $pg `
  -company 97e1a309-3d23-4aa2-b518-8e8d7afdfec7 `
  -maps-dir $maps -resume -ensure-demo

Reports

After each run:

• <report-dir>/migration-report.json — latest counts, domains, validation, demo email (password not stored)
• <report-dir>/migration-report-<UTC>.json — stamped copy
• docs/migration-reports/migration-report-latest.json — when the docs folder exists
• <maps-dir>/validation-report.json, id-map.json, per-entity maps

Postgres optimizations

• FK order: companies/users → catalog/feeds → products → export → gap domains
• Batches: attributes & category_attributes via pgx.Batch; raw_products via COPY with per-row upsert fallback
• Transactions: per product batch; usage merge uses a single batch transaction
• Idempotent: natural-key ON CONFLICT upserts + -resume id-map reuse
• Filters: -company pushes IN (...) into MySQL SELECTs (no N+1)

Id-map hygiene

• Use a dedicated -maps-dir for live loads (e.g. rtifacts/). Do not point fixture dry-runs at the same directory afterward — fixture remaps can drift UUIDs.
• On every live run the migrator reconciles companies.legacy_company_id → Postgres id so gap-only domains (-domains settings,formulas,...) still resolve FKs even if the id-map drifted.
• Rebuild maps from Postgres if needed: SELECT legacy_company_id, id FROM companies WHERE legacy_company_id IS NOT NULL.

Security notes

• Never import Clerk sessions or password hashes (must_set_password=true for migrated users).
• Do not commit .env, DSNs, artifacts/, or invite tokens.
• Woo consumer secrets are copied when present in legacy custom fields — treat Postgres as secret-bearing after woo migration.

Related docs

• migration-readiness.md
• migration-run-log.md
• demo-user.md
• cutover.md