Initial commit of Descrybe v2 without local scratch artifacts.
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
# Performance notes (descrybe-v2 Go API)
|
||||
|
||||
Date: 2026-08-04
|
||||
|
||||
Stack: Go API (`apps/api`) + Postgres (goose migrations under `apps/api/sql/schema`).
|
||||
Not the legacy Next.js / Drizzle repo (`descrybe`).
|
||||
|
||||
## Goals
|
||||
|
||||
Bound round-trips and memory on hot paths for large catalogs (Local Demo Co scale): product lists, feed sync, export feeds.
|
||||
|
||||
## Review vs legacy TS wins
|
||||
|
||||
| Concern | Legacy TS (`descrybe`) | Go v2 (`descrybe-v2`) |
|
||||
|---------|------------------------|------------------------|
|
||||
| Product list count + page | Parallel `Promise.all` | **Added** `parallelCountAndList` in `internal/catalog` |
|
||||
| Slim list columns | Drizzle relation trim | Already selects list columns (not `SELECT *`) on HTTP list path |
|
||||
| Feed sync re-parse every chunk | Bug in `syncFeedChunk` loop | **Already OK** — one download/parse per `Sync`, row callback + `upsertChunk` |
|
||||
| XML/CSV export streaming | Added stream + disk cache | **Already OK** — `streamExport` / `streamXML` / `streamCSV` over `pgx.Rows` + flush |
|
||||
| List indexes | Drizzle `0028_…` MySQL | **Added** goose `018_list_hotpath_indexes.sql` (Postgres) |
|
||||
|
||||
## Changes landed on v2
|
||||
|
||||
### 0. Scale small wins (2026-08-09)
|
||||
|
||||
- **Keyset enforcement** — `normalizeProductListFilter` caps product `limit` at 200 and rejects deep OFFSET (`>5000`) unless `cursor`/`after_id` is set.
|
||||
- **pgx pool jitter** — `DB_MAX_CONN_LIFETIME_JITTER` (default 6m) + sizing docs in `docs/ops-runtime.md`.
|
||||
- **Sync concurrency cap** — `jobs.SyncSlots` / `ClampSyncWorkers` (default 1, max 2) in `cmd/worker` for feed/Woo/Shopify claims.
|
||||
|
||||
### 1. Parallel product list (`apps/api/internal/catalog/service.go`)
|
||||
|
||||
- `ListRawProducts`, `ListProcessedProducts`, `ListProcessedProductsDetailed` run **count** and **page** queries concurrently via `parallelCountAndList`.
|
||||
- Arg slices are copied so limit/offset appends do not race the count query.
|
||||
|
||||
### 2. Feed sync (`apps/api/internal/feeds/sync.go` + `parse.go`)
|
||||
|
||||
No code change required for the legacy re-parse bug:
|
||||
|
||||
- `runSync` loads the feed once, parses once (`parseXMLItems` / `parseCSV` with per-row callback).
|
||||
- Upserts in `upsertChunkSize` (100) with job progress updates.
|
||||
- Content-hash short-circuit skips parse when the feed body is unchanged.
|
||||
|
||||
### 3. Export streaming (`apps/api/internal/feeds/export.go`)
|
||||
|
||||
No code change required:
|
||||
|
||||
- Public and generate paths use `streamExport` → `streamXML` / `streamCSV`.
|
||||
- Rows come from a single scoped SQL query (`LIMIT` capped by `exportMaxProducts`); writers flush every `exportChunkHint` (500) rows.
|
||||
- Content is not buffered as a full XML string for the public stream path.
|
||||
|
||||
### 4. Indexes (goose `018_list_hotpath_indexes.sql`)
|
||||
|
||||
- `processed_products_company_updated_idx` — `(company_id, updated_at DESC)`
|
||||
- `processed_products_company_status_updated_idx` — `(company_id, status, updated_at DESC)`
|
||||
- `raw_products_company_updated_idx` — `(company_id, updated_at DESC)`
|
||||
- `raw_products_company_processed_updated_idx` — `(company_id, is_processed, updated_at DESC)`
|
||||
- `categories_company_parent_idx` — `(company_id, parent_unique_id)`
|
||||
|
||||
Apply:
|
||||
|
||||
```powershell
|
||||
cd f:\laragon\www\_MY\descrybe-v2
|
||||
$env:DATABASE_URL = "postgres://descrybe:descrybe@localhost:5433/descrybe?sslmode=disable"
|
||||
.\scripts\migrate.ps1
|
||||
```
|
||||
|
||||
## Smoke (Local Demo Co)
|
||||
|
||||
Repeatable low-concurrency harness (healthz + v1 feeds/products list), RPS assumptions, and how to scale up: [`scripts/api-load-smoke/README.md`](../scripts/api-load-smoke/README.md). Default API base is `http://127.0.0.1:28471` (`HTTP_ADDR`).
|
||||
|
||||
```powershell
|
||||
cd f:\laragon\www\_MY\descrybe-v2\scripts\api-load-smoke
|
||||
go run . # -c 2 -d 5s -rate 6; ~6 aggregate RPS
|
||||
```
|
||||
|
||||
One-off Measure-Command (API key from `docs/demo-user.md`):
|
||||
|
||||
```powershell
|
||||
$h = @{ Authorization = "Bearer dk_demo_local_descrybe_test_key_v1" }
|
||||
Measure-Command { Invoke-RestMethod -Headers $h "http://127.0.0.1:28471/api/v1/products?limit=25&offset=0" }
|
||||
Measure-Command { Invoke-RestMethod -Headers $h "http://127.0.0.1:28471/api/v1/products?kind=raw&limit=25&offset=0" }
|
||||
```
|
||||
|
||||
Restart the API process after deploying Go changes so handlers pick up `parallelCountAndList`.
|
||||
|
||||
## API docs (/docs) performance (2026-08-04)
|
||||
|
||||
Viewer: **RapiDoc** (OSS web component; no Scalar cloud). Disk sizes:
|
||||
|
||||
| Asset | Raw | Gzip |
|
||||
|-------|-----|------|
|
||||
| RapiDoc `rapidoc-min.js` | **~843 KB** | **~213 KB** |
|
||||
| Scalar ESM (previous) | 691 KB | ~154 KB (+ lazy chunks; ~3.8 MB total vendor tree) |
|
||||
| Scalar IIFE (older) | 3661 KB | ~1052 KB |
|
||||
| OpenAPI YAML (`v1_openapi.go`) | ~74 KB | ~12 KB |
|
||||
|
||||
Changes:
|
||||
|
||||
1. **RapiDoc vendor** — `scripts/copy-rapidoc-ui.mjs` copies `dist/rapidoc-min.js` and writes `.gz`. Single file, no cloud registry/agent chunks.
|
||||
2. **hooks.server.ts** — long-lived `Cache-Control` for `/vendor/rapidoc/*`; serves precompressed `.gz` when `Accept-Encoding: gzip`.
|
||||
3. **/docs page** — `type=module` load, `modulepreload`, early `prefetchOpenApiSpec` + `loadSpec(blobUrl)` (no double fetch), theme via attribute (no remount).
|
||||
4. **OpenAPI handler** — gzip body when requested (`Vary: Accept-Encoding`); coordinates with sibling Cache-Control / ETag / 304 work.
|
||||
|
||||
Verify: `node apps/web/scripts/copy-rapidoc-ui.mjs`, then load `/docs` and confirm Network shows ~843 KB (or ~213 KB gzip) for `rapidoc-min.js` and `Content-Encoding: gzip` on `/api/v1/openapi.yaml`.
|
||||
|
||||
|
||||
## Benchmark / smoke (this run)
|
||||
|
||||
API was up at `http://127.0.0.1:8080`. Goose `018` applied (indexes live immediately). Go parallel list is in source — restart `cmd/api` to load it.
|
||||
|
||||
Local Demo Co via demo API key (3 probes each; cold then warm):
|
||||
|
||||
| Probe | total | page | ms (3 runs) | avg |
|
||||
|-------|------:|-----:|-------------|----:|
|
||||
| GET /api/v1/products?limit=25&offset=0 (processed) | 4337 | 25 | 90,19,10 | ~40 |
|
||||
| GET /api/v1/products?limit=25&offset=1000 | 4337 | 25 | 14,10,9 | ~11 |
|
||||
| GET /api/v1/products?kind=raw&limit=25&offset=0 | 23775 | 25 | 12,8,8 | ~9 |
|
||||
| GET /api/v1/products?kind=raw&limit=25&offset=1000 | 23775 | 25 | 35,29,26 | ~30 |
|
||||
|
||||
`go build ./...` OK. Tests: `./internal/catalog` `./internal/feeds` `./internal/httpapi` OK.
|
||||
|
||||
## Million-SKU backlog (prioritized)
|
||||
|
||||
Operator/dev map from web research → concrete paths. Status notes reflect current tree (2026-08-04); do not treat “partial” as done at 1M SKU.
|
||||
|
||||
### P0 — do first (blocks million-SKU)
|
||||
|
||||
1. **Keyset pagination** — replace deep `OFFSET` with `(updated_at, id)` / `(created_at, id)` cursors.
|
||||
- **Enforced (2026-08-09):** product lists reject `offset > 5000` without `cursor`/`after_id` (`MaxOffsetWithoutCursor`); product page limit capped at 200 (`MaxProductPageLimit`) in `normalizeProductListFilter`.
|
||||
- API: `apps/api/internal/catalog/service.go` + `cursor.go`, handlers expose `next_cursor` / `next_after_id`.
|
||||
- Web: `apps/web/src/lib/list.ts` (`pageOffset`), `apps/web/src/lib/components/products/ProductPagination.svelte` — still prefer cursor over deep offset.
|
||||
2. **Set-based feed upserts** — today `upsertChunk` uses `pgx.Batch` of per-row INSERT/UPDATE (`apps/api/internal/feeds/sync.go`). Move to set-based `INSERT … SELECT` / `UNNEST` (or COPY + merge) for chunk apply.
|
||||
3. **Kill JSONB ILIKE (full cast)** — `CAST(mapped_data/raw_data AS text) ILIKE` is already avoided; search uses keyed `->>'…'` + `gtin`/`feed` ILIKE in `appendRawProductFilters` (`apps/api/internal/catalog/service.go`). Remaining P0: measure leading-wildcard ILIKE; add `pg_trgm`/expression indexes or constrain search to prefix/FTS before 1M.
|
||||
4. **Stream feed download + parse** — `downloadFeed` still `io.ReadAll` into a size-capped buffer (`apps/api/internal/feeds/download.go`); wired via `loadFeedSource` (`source.go`) into `Sync` (`sync.go`). Goal: stream HTTP → parser without holding the full body when possible.
|
||||
5. **AI Batch API + content-hash skip** — **Hash-skip landed** (`HashEnhanceInput` + `field_sources.enhance_input_hash` in `apps/api/internal/processing`; loaded in `processOne` before `RunSteps` / `ConsumeCredits`). Sync `Completer` / `CompleteWithOptions` is still per-SKU. **OpenAI Batch deferred:** needs a job-level coordinator (JSONL submit -> `batch_id` -> reconcile -> debit) that does not fit the current sync completer interface; ~50% OpenAI discount remains a follow-up.
|
||||
6. **Tenant-leading indexes** — composites landed in `018_list_hotpath_indexes.sql` / `020_raw_list_created_indexes.sql`, but single-column leftovers from `apps/api/sql/schema/002_catalog.sql` remain (e.g. `raw_products_company_id_idx`, `raw_products_gtin_idx`, `raw_products_feed_id_idx`, `raw_products_processing_status_idx`, `processed_products_company_id_idx`, `processed_products_product_id_idx`). Drop or replace with `(company_id, …)` leading keys where planners still pick the narrow indexes.
|
||||
|
||||
### P1 — next (fairness, scale edges, ops)
|
||||
|
||||
- **Per-tenant job fairness** — `apps/api/internal/jobs/river.go` (+ processing claim path in `pipeline.go`): avoid one company starving the queue.
|
||||
- **Export beyond 50k** — `exportMaxProducts = 50000` in `apps/api/internal/feeds/export.go`; raise via keyset/chunked export or a read model (see P2).
|
||||
- **CSV import batching** — `apps/api/internal/catalog/import_csv.go` + `httpapi/catalog_import_handlers.go`: per-row `Exec` today; batch/COPY.
|
||||
- **Covering indexes** — extend list/export SELECT lists to match index INCLUDE / composite keys (build on `018`/`020`).
|
||||
- **Observability** — request/query timing on list, sync, process, export; slow-query + job duration dashboards.
|
||||
|
||||
### P2 — later (architecture)
|
||||
|
||||
- **Partition plan** — `raw_products` / `processed_products` by `company_id` (or time) once single-tenant size dominates vacuum/autovacuum.
|
||||
- **Export read model** — denormalized export rows so public/generate streams avoid heavy joins at read time.
|
||||
- Stream XML via `encoding/xml` Token writer if payload construction grows heavy.
|
||||
- Optional NDJSON disk cache for multi-worker resume of a single sync job (in-process `Sync` already avoids re-parse).
|
||||
Reference in New Issue
Block a user