greeneclipse/descrybe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial commit of Descrybe v2 without local scratch artifacts.

Browse Source 
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
This commit is contained in:
greeneclipse
2026-08-09 22:47:43 +02:00
commit 8580c996c3
1285 changed files with 325780 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/migrate-from-descrybe-new.md
+220
View File
@@ -0,0 +1,220 @@
# Migrate `descrybe_new.sql` → v2 Postgres (A1 only)
**Date:** 2026-08-08 (local reseed from `descrybe_new (1).sql`)
**Status:** **PASS**
**Scope:** Single tenant **A1 Slovenija** only (all other dump tenants skipped / cleaned)
Secrets are masked below. Do not commit `.env`, MySQL passwords, `artifacts/`, or invite tokens.
## What “a1” resolved to
| Field | Value |
|-------|--------|
| MySQL company name | `A1 Slovenija` |
| MySQL / legacy `company_id` | `97e1a309-3d23-4aa2-b518-8e8d7afdfec7` |
| Match used | `companies.name` LIKE `%A1%` → exact row above (only A1 hit) |
| Postgres after migrate + demo | **A1 Slovenija** (dump name kept) |
| Postgres `companies.id` | `604f23a8-b66e-4b21-8b45-0d72b68f4790` (stable via `artifacts/descrybe-new-a1/id-map.json`) |
| `legacy_company_id` | `97e1a309-3d23-4aa2-b518-8e8d7afdfec7` |
**Company naming:** Keep dump name **A1 Slovenija**. Do not rename to “Local Demo Co”. Wallet/plan come from MySQL `credit_balances` / plan **A1** (2500 total / 163 used / 2337 remaining on 2026-08-08 dump) — not a fake 1M demo pack.
## Inputs
| Item | Value |
|------|--------|
| Dump | `C:\Users\Green Eclipse\Downloads\descrybe_new (1).sql` (~549 MB Adminer MySQL 8 dump) |
| Temp MySQL DB | Laragon `descrybe_new` (left intact; did **not** overwrite `descrybe_v2`) |
| Postgres | Docker `descrybe-v2-postgres` host port **5433** |
| Postgres DSN (local default) | `postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable` |
| Migrator | `apps/api/cmd/migrator` |
| Maps / report | `artifacts/descrybe-new-a1/` (gitignored); copy `docs/migration-reports/migration-report-a1-latest.json` |
## Commands run
### 1. Wipe v2 Postgres only (keep Docker volume)
```powershell
docker exec descrybe-v2-postgres psql -U descrybe -d postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'descrybe' AND pid <> pg_backend_pid();"
docker exec descrybe-v2-postgres psql -U descrybe -d postgres -c "DROP DATABASE IF EXISTS descrybe;"
docker exec descrybe-v2-postgres psql -U descrybe -d postgres -c "CREATE DATABASE descrybe OWNER descrybe;"
```
### 2. Goose migrations (001–024)
```powershell
cd f:\laragon\www\_MY\descrybe-v2\apps\api
$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
go run github.com/pressly/goose/v3/cmd/goose@v3.24.3 -dir sql/schema postgres $env:DATABASE_URL up
```
Note: `024_ai_prompts.sql` had a UTF-8 BOM (`\ufeff`); stripped so goose could parse (same class of fix as earlier `006_feed_sync.sql`).
### 3. Import dump into temp MySQL (streamed; ~2 min)
```powershell
# create empty DB, then stream file into mysql client (do not load 862MB into PowerShell)
mysql -uroot -p**** -e "DROP DATABASE IF EXISTS descrybe_new; CREATE DATABASE descrybe_new CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
# stdin stream of descrybe_new.sql → descrybe_new
```
Dump-wide MySQL counts (before filter): companies 31, categories 8465, attributes 198319, xml_feeds 33, raw_products 159070, processed_products 7953, export_feeds 11.
### 4. A1-only migrator
```powershell
cd f:\laragon\www\_MY\descrybe-v2\apps\api
$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
$env:MIGRATE_MYSQL_DSN = "root:****@tcp(localhost:3306)/descrybe_new?parseTime=true"
$maps = "f:\laragon\www\_MY\descrybe-v2\artifacts\descrybe-new-a1"
$a1 = "97e1a309-3d23-4aa2-b518-8e8d7afdfec7"
go build -o $env:TEMP\descrybe-migrator.exe ./cmd/migrator
# dry-run then live (filter at migrate time)
& $env:TEMP\descrybe-migrator.exe -mysql $env:MIGRATE_MYSQL_DSN -postgres $env:DATABASE_URL `
-company $a1 -dry-run -maps-dir $maps -report-dir $maps -resume=false
& $env:TEMP\descrybe-migrator.exe -mysql $env:MIGRATE_MYSQL_DSN -postgres $env:DATABASE_URL `
-company $a1 -maps-dir $maps -report-dir $maps -resume `
-ensure-demo -demo-email demo@descrybe.local -demo-password 'DemoPass123!' `
-skip-post-import
```
Live elapsed ≈ **32s**. Validation: `mode: live ok=true`, orphan checks **17/17 PASS**.
### 5. Seed demo (Enterprise + API key + `.test` alias)
```powershell
go run ./cmd/seed-demo -postgres $env:DATABASE_URL `
-email demo@descrybe.local -password 'DemoPass123!'
```
### 6. Post-cleanup (junk from global user import)
Migrator still upserts **all** MySQL profile→user rows even with `-company` (memberships are filtered). Removed users with **no** membership (kept demo emails):
```sql
DELETE FROM users u
WHERE NOT EXISTS (SELECT 1 FROM memberships m WHERE m.user_id = u.id)
AND lower(u.email) NOT IN ('demo@descrybe.local','demo@descrybe.test');
-- deleted 22 orphan users
```
### 6b. Import A1 job history (optional follow-up)
Job history was skipped in the initial cutover. Re-run with domain `jobs` (rows tagged `ai_provider_mode=migrated` so 30-day retention does not purge them):
```powershell
& $env:TEMP\descrybe-migrator.exe -mysql $env:MIGRATE_MYSQL_DSN -postgres $env:DATABASE_URL `
-company $a1 -domains jobs -maps-dir $maps -report-dir $maps -resume -skip-post-import
```
Local/dev (**outdated — do not follow**): demo no longer owns A1; use **Platform Demo** + `a1-primary@` per [safe-test-fixtures.md](safe-test-fixtures.md). Historical logins:
| Role | Email | Password |
|------|-------|----------|
| Demo admin | `demo@descrybe.local` (alias `.test`) | `DemoPass123!` |
| Primary A1 | `a1-primary@descrybe.local` | `DemoPass123!` |
Primary maps to Clerk `user_30AqqJ8uepxvPUzDSqy81U5w6Ll`. Other A1 MySQL profiles were removed after migrate (jobs reassigned to primary). To reset the primary password:
```powershell
go run ./cmd/migrator -postgres $env:DATABASE_URL `
-set-password 'a1-primary@descrybe.local:DemoPass123!'
```
### 6c. Local Demo Co user cleanup (2026-08-05)
MySQL A1 (`97e1a309-…`) had **5** `profiles.user_id` values. Preference: keep **demo + primary only**.
| Action | Detail |
|--------|--------|
| Kept | `demo@descrybe.local`, `demo@descrybe.test`, `a1-primary@descrybe.local` (`user_30AqqJ…`) |
| Deleted (4) | `user_2tJxu…`, `user_2yaUP…`, `user_2zzh…`, `user_3005r…` (`@legacy.local`) |
| Reassigned | 3 `processing_jobs.user_id` → primary (280 jobs on primary; 51 still null) |
| Catalog | Unchanged — **4378** processed products, 331 jobs |
**Do not restore** those four `@legacy.local` members. A later agent re-inserted them for switcher labeling; they were deleted again. Local Demo Co stays at **3 users** / **3 memberships**. UserSwitcher labels: **Demo admin** (`.local` + `.test`) and **A1 · …81U5w6Ll** (`is_primary_a1`).
## Row counts — kept vs skipped
### Migrator written (A1 filter)
| Entity | Written / kept | Notes |
|--------|---------------:|-------|
| companies | 1 | Only A1; renamed → Local Demo Co |
| memberships | 5 → **3** after cleanup | A1 primary + 2 demo admins |
| users (after cleanup) | **3** | demo `.local` + demo `.test` + `a1-primary@descrybe.local` |
| plans | seed + migrated catalog | Global plan rows OK for local billing |
| company_plans | 1 → Enterprise via seed-demo | Other tenants’ plans skipped |
| credit_balances | 1 | 1M Enterprise credits after seed |
| categories | 119 | |
| attributes | 303 | |
| category_attributes | 307 | |
| custom_variables | 3 | |
| input_feeds (xml) | 12 | |
| feed_mappings | 11 | ComTrade has 0 mappings in source too |
| raw_products | 23748 | MySQL A1 had 56132; GTIN-deduped under `(company_id, gtin)` |
| processed_products | 4330 | |
| export_feeds | 2 | |
| files | 0 | Dump had 1 file metadata row for another context / skipped |
| processing_jobs | 179 | via `-domains jobs` (`ai_provider_mode=migrated`) |
### Skipped (other tenants / noise)
| Item | Approx. skipped |
|------|----------------:|
| Other companies | 30 of 31 |
| Non-A1 categories / attributes / feeds / products / exports | Rest of dump (see MySQL totals above) |
| Orphan users after cleanup | 22 |
| `company_plans` for non-A1 | 4 skipped in migrator log |
| `credit_balances` for non-A1 | 7 skipped |
| API keys / Clerk passwords / job history | Never imported |
### Final Postgres snapshot (A1 Slovenija — 2026-08-08)
| Metric | Count |
|--------|------:|
| companies | 1 (+ `__platform_settings__`) |
| users | **3** (demo.local, demo.test, a1-primary) |
| memberships | **3** |
| categories | 119 (108 with AI prompts overlay) |
| attributes | 303 |
| category_attributes | 307 |
| input_feeds | 12 |
| feed_mappings | 11 |
| raw_products | 23748 |
| processed_products | 4330 |
| export_feeds | 2 |
| processing_jobs | 179 |
| custom_variables | 3 |
| wallet | 2500 / 163 used / 2337 remaining |
## Demo admin (local only)
| Field | Value |
|-------|--------|
| Email | `demo@descrybe.local` (alias `demo@descrybe.test`) |
| Password | `DemoPass123!` (already documented in `docs/demo-user.md`) |
| Role | Platform admin + company **admin** on Local Demo Co |
| Plan | Enterprise — 1_000_000 AI credits, unlimited SKUs |
| API key | `dk_demo_local_descrybe_test_key_v1` (hash only in DB) |
Web: http://localhost:5174/login · API: http://localhost:8080
## Verification (this run)
| Check | Result |
|-------|--------|
| `GET /healthz` | ok |
| `GET /readyz` | database ok |
| `POST /api/auth/login` (CSRF cookie + Origin) | **200** → company Local Demo Co |
| `GET /api/v1/products?limit=1` (demo API key) | **200**, `meta.total=4378` |
| `GET /api/v1/feeds` | **200**, 12 feeds |
## Related docs
- [portable-mysql-pg-migration.md](portable-mysql-pg-migration.md) — migrator flags / domains
- [demo-user.md](demo-user.md) — demo credentials & Enterprise wallet
- [migration-run-log.md](migration-run-log.md) — prior full-tenant rehearsal
- Report: [migration-reports/migration-report-a1-latest.json](migration-reports/migration-report-a1-latest.json)