descrybe/docs/category-formulas.md

# Category formulas

A category formula is what makes enhance *generate* copy instead of reshuffling the
feed. The feed supplies facts; the formula supplies the shape.

Legacy Descrybe (`descrybe-legacy/generator.php`) did this by making the category's
`Prompt` column the entire user message — a "GPT predloga" with `<name>{…}</name>`,
`<metaDescription>{…}</metaDescription>` and an HTML body of `<H2>`/`<p>` pairs
closing with a technical-specification list. v2 keeps that behaviour and adds
structured `title_template` / `description_template` columns on top of it.

## Two formula sets

| | A1 Slovenija (legacy plan) | Everyone else |
|---|---|---|
| Source | `scripts/seed/a1-category-prompts.json` (+ `wp_product_categories.sql`) | `apps/api/internal/aiprompts/defaults/default-category-prompts.json` |
| Applied by | `cmd/seed-a1`, `cmd/repair-category-prompts`, Admin → Sync A1 | category create + enhance-time fallback |
| Language of the formula text | Slovenian (the tenant's own copy) | English |
| Prompt scaffolding | Slovenian (`GPT predloga:`, `Staro_ime_izdelka:`) | English (`Product template:`, `Old_product_name:`) |

Both drive the same code path. The scaffolding language follows the formula's own
language (`reSlovenianFormulaCue` in `processing/a1_enhance_prompt.go`) so A1 keeps
the exact frame `generator.php` used. Output language is always `{{language}}` from
the company's content language — an English formula still produces Slovenian copy
for a Slovenian tenant.

## The English defaults

`default-category-prompts.json` is the A1 seed translated: 116 categories, same
naming formulas and body structure, generated by

```
make seed-defaults      # regenerate
make check-defaults     # fail if stale (CI guard)
```

Translation is phrase-level over a closed vocabulary (55 title slots, 39 body
blocks, 1 meta rule) and the generator **fails** if any Slovenian text survives, so
a new legacy phrase can never ship untranslated.

Where they apply:

1. **Category create** — `catalog.ApplyDefaultCategoryFormulas` fills `prompt`,
   `title_template` and `description_template` on new categories (API create and
   CSV taxonomy import). Only empty columns are written, so nothing existing is
   overwritten.
2. **Enhance-time fallback** — `processing.withDefaultCategoryFormula` supplies the
   default for a categorised product whose category still has no formula. It is
   skipped when the product has no category, when the category has its own
   prompt/formula, or when the tenant authored their own enhance template.

A category outside the seeded taxonomy gets `GenericDefaultCategoryPrompt` — same
structure, no category-specific naming slots.

## Proving it locally

`cmd/formula-e2e` builds the pipeline the way `cmd/worker` does, runs a real job and
checks the output is not a feed copy:

```
make up                                          # Postgres :5433
cd apps/api && go run ./cmd/mock-llm &            # formula-aware local LLM
make formula-e2e
```

`cmd/mock-llm` reads the template out of the prompt and answers in its shape using
only the supplied name/description/category/attrs. That proves two things a static
stub cannot: the formula reached the model, and a compliant reply survives the
pipeline's formula gate.

Useful flags:

```
go run ./cmd/formula-e2e -company "A1 Slovenija" -gtin 8022068075495 -llm-base http://127.0.0.1:18767/v1
go run ./cmd/formula-e2e -new-tenant -llm-base http://127.0.0.1:18767/v1
```

`-llm-base` pins the provider for the run; without it the harness uses whatever
`platformsettings.ResolveOpenAI` returns (admin DB setting first, then env).

## What Review compares

Review shows Original vs Enriched for name, description, category, attributes and
SEO meta. Each "Original" must come from a source the pipeline never writes, which
is less obvious than it sounds:

| field | Original comes from | trap |
|---|---|---|
| name / description | `original_name` / `original_description` | `name`/`description` from the API are display-preferred and resolve to the ENRICHED copy |
| category | `feed_category` | `mapped_data.category` is written back by the pipeline; `category_source` says who chose it |
| attributes | `mapped_data.specifications` (feed specs) | `processed_products.attributes` is written to the same value as `processed_attributes` |
| | | EPREL writes one fact under several names — `product-attributes.ts` folds the aliases so it is listed once |
| SEO meta | `mapped_data.meta_*` | most feeds have none, so Enriched shows as **Added** |

Attributes are grouped **Added / Changed / Unchanged**, with unchanged collapsed —
a review should lead with what the run actually did. Alias keys are folded into one
row: enrichment stores `eprel_pdf` and `eprel_pdf_url`, `energy_class` and
`eprel_energy_class`, which otherwise listed the same fact two or three times.
Folding needs a related name AND an identical value, so width and height are never
merged just because they match (`$lib/attribute-diff`, unit-tested).

The product detail endpoint deliberately does **not** apply the SEO-meta omit rule.
That rule keeps meta out of list views for cohorts that do not sell on it, but the
detail payload feeds review, and meta the pipeline actually generated has to be
reviewable — the panel hides the block itself when both sides are empty.

Anything with no Original is labelled **Added** and shown as "not in feed", not
"Matched" — saying Matched about a field the supplier never sent reads as if the
feed had one.

Two things behave this way by design, not by accident:

- **A1 Slovenija and Platform Demo do not GENERATE SEO meta**
  (`processing.CompanyOmitsSEOMetaID`). Meta already stored on their rows is still
  shown — the read path only hides empty values, so Review and Content display what
  the AI actually produced.
- **Rows processed before `category_source` existed** cannot say whether their
  `mapped_data.category` came from the feed or from an earlier AI run, so Review
  still treats it as feed-supplied. Reprocessing the product records the real
  provenance.

## Low-confidence categorisation fails the product

The categorize model always answers with *some* id from the list, so a reply like
`{"categoryId":"1","confidence":0.08}` is the model saying "I do not know". Taking
it at face value files a camping chair under Generators and then drives that
category's formula, producing confidently wrong copy.

Below `processing.DefaultMinCategorizeConfidence` (**0.75**) the product fails with
`ErrCategoryNotFound` and processing stops there — enhance is never called, so no
credits and no provider cost are spent writing copy for a category nobody believes
in. The same applies when the model returns an id outside the taxonomy.

```
processing: category rejected uid=1 confidence=0.08 min=0.75 reason=low_confidence
processing: item failed ... err=product category could not be determined:
            best match "1" scored 0.08, below the 0.75 minimum
```

The product lands in the job as `failed` with that message on
`processing_job_products.error`, and `raw_products.processing_status = 'failed'`.

Not gated: a category that came from the feed, vector or a prior run (categorize
never runs), a model that reports no confidence at all, and AI-disabled plans.
Override the floor per job with `StepPolicy.MinCategorizeConfidence`.

## Why enhance can still return feed copy

In order of how often it bites:

1. **The product has no category.** No category → no formula. Roughly 83% of the A1
   catalog has no `mapped_data.category` (see `scripts/seed/README.txt`); those rely
   on the `categorize` step, which needs a working LLM or Pinecone.
2. **The AI provider is unreachable.** Enhance falls back to supplier copy and marks
   `field_sources.name = ai_enhance_failed`. Check the admin OpenAI setting — it
   overrides the `OPENAI_*` env values.
3. **The plan/credits gate.** `ai_enhance skip reason=entitlement_can_use_ai` or
   `insufficient credits`.
4. **The model ignored the formula.** One retry is issued
   (`descriptionFormulaRetrySuffix` / `titleRewriteRetrySuffix`); if it still fails,
   the result is marked `synthesized`/`refused` and no `enhance_input_hash` is
   stored, so a reprocess tries again rather than caching the bad copy.

`processing: ai_enhance outcome=… category_uid=… title_formula=… desc_formula=…
formula_override=…` in the worker log tells you which of these happened — and
**Admin → AI calls** shows the exact prompt and reply for the call itself
(docs/ai-call-inspector.md).