descrybe/docs/local-llm-tuning.md

# Local LLM tuning (weak / 8k-class models)

Defaults for OpenAI-compatible local servers such as Green Chat Gemma 12B (`overloaded-local`, ~8k context).

## Env

| Variable | Typical local value |
|----------|---------------------|
| `OPENAI_BASE_URL` | `http://<host>:8767/v1` |
| `OPENAI_MODEL` | id from `GET /v1/models` (e.g. `overloaded-local`) |
| `OPENAI_API_KEY` | required non-empty for `Enabled()` |
| `PROCESSING_RPM` | `60` |
| `PROCESSING_MAX_RETRIES` | `3` |

Restart **both** `api` and `worker` after changing `OPENAI_*`. Worker logs should say `OpenAI enabled base=… model=…`, not `heuristic completer`.

## Product enhance model choice

Admin AI role for **product enhance** should prefer a **fast non-reasoning chat model** that emits `message.content` (JSON) under budget.

Reasoning / “code-fast” style models often spend the entire `max_tokens` budget in `reasoning_content`, then return `finish_reason=length` with empty or truncated JSON. Descrybe retries once at `MaxTokensEnhanceRetry` and may synthesize formula HTML as salvage — that is **not** live LLM formula compliance.

If enhance logs show `finish_reason=length` with empty/truncated content even after the retry, switch Admin enhance to a non-reasoning model.

## Hardening defaults (code)

| Constant | Value | Where |
|----------|------:|-------|
| `DefaultStructuredTemp` | `0.2` (capped ≤0.3) | `processing/llm_json.go`, `openai.go` |
| `MaxTokensEnhance` | `16384` | product title/description (reasoning + formula HTML JSON) |
| `MaxTokensEnhanceRetry` | `32768` | one-shot bump on `finish_reason=length` empty/truncated JSON |
| `MaxTokensSEO` | `180` | meta title/description |
| `MaxTokensCampaign` | `650` | email JSON |
| `MaxProductDescRunes` | `400` | user context |
| `MaxAttrKeys` / `MaxAttrValueRunes` | `10` / `60` | attrs in enhance prompt |
| `MaxBrandInjectRunes` | `500` | brand kit block |
| `MaxCampaignProducts` | `8` | campaign product list |

OpenAI-compatible chat completions here use a single `max_tokens` completion budget (no separate reasoning vs content split on typical gateways). Length-cap handling lives in `OpenAIClient.doComplete` / `CompleteWithOptions`.

## Prompt style

- Short system prompts with **bullet rules** + explicit JSON schema
- **One** few-shot example max
- Brand kit as compact bullets: `Brand:\n- tone: …\n- do: …`
- Product context: category + name + truncated desc + priority attrs only

## JSON reliability

`processing.CompleteJSON`:

1. Call Completer with `max_tokens` + low temperature
2. `StripJSONFences` / isolate `{…}`
3. On parse fail → **one retry** with `INVALID. Reply with ONLY one JSON object…`
4. Call sites fall back to originals/templates instead of storing garbage prose as titles

`OpenAIClient` additionally treats `finish_reason=length` with empty or unparseable JSON as failure (not success) and retries once at `MaxTokensEnhanceRetry`.

## Ops tips

- Prefer LAN TCP reachability checks (`Test-NetConnection host -Port 8767`) over ICMP
- Smoke: [ai-full-smoke.md](ai-full-smoke.md), [green-chat-llm.md](green-chat-llm.md)
- CI / no real model: [mock-llm.md](mock-llm.md) (`OPENAI_BASE_URL=http://127.0.0.1:18767/v1`) — includes **translation + processing verification**
- Locale lists + commented `OPENAI_*` / `MOCK_LLM_*` keys: root [`.env.example`](../.env.example)
- Do not commit real keys or LAN secrets