descrybe/docs/local-llm-tuning.md

# Local LLM tuning (weak / 8k-class models)

Defaults for OpenAI-compatible local servers such as Green Chat Gemma 12B (`overloaded-local`, ~8k context).

## Env

| Variable | Typical local value |
|----------|---------------------|
| `OPENAI_BASE_URL` | `http://<host>:8767/v1` |
| `OPENAI_MODEL` | id from `GET /v1/models` (e.g. `overloaded-local`) |
| `OPENAI_API_KEY` | required non-empty for `Enabled()` |
| `PROCESSING_RPM` | `60` |
| `PROCESSING_MAX_RETRIES` | `3` |

Restart **both** `api` and `worker` after changing `OPENAI_*`. Worker logs should say `OpenAI enabled base=… model=…`, not `heuristic completer`.

## Product enhance model choice

Admin AI role for **product enhance** should prefer a model that emits `message.content` (JSON) under budget. Recommended OpenAI: **`gpt-5.6-luna`** (high-volume / cost-efficient GPT-5.6 tier).

Reasoning / “code-fast” style models often spend the entire completion budget in `reasoning_content`, then return `finish_reason=length` with empty or truncated JSON. Descrybe retries once at `MaxTokensEnhanceRetry` and may synthesize formula HTML as salvage — that is **not** live LLM formula compliance.

For GPT-5 / o-series Chat Completions, the client sends `max_completion_tokens` (not `max_tokens`), omits custom `temperature`, and sets `reasoning_effort` (`none` for probes, `low` for long formula-HTML enhance) so visible HTML is not eaten by deep reasoning.

If enhance logs show `finish_reason=length` with empty/truncated content even after the retry, switch Admin enhance to a lower-reasoning or classic chat model (e.g. `gpt-4o-mini`).

## Hardening defaults (code)

| Constant | Value | Where |
|----------|------:|-------|
| `DefaultStructuredTemp` | `0.2` (capped ≤0.3; omitted on GPT-5/o-series) | `processing/llm_json.go`, `openai.go` |
| `MaxTokensEnhance` | `24576` | product title + multi-section HTML description JSON |
| `MaxTokensEnhanceRetry` | `40960` | one-shot bump on `finish_reason=length` empty/truncated JSON |
| `MaxTokensSEO` | `180` | meta title/description |
| `MaxTokensCampaign` | `650` | email JSON |
| `MaxProductDescRunes` | `3500` | source description in enhance user context |
| `MaxAttrKeys` / `MaxAttrValueRunes` | `10` / `60` | attrs in enhance prompt |
| `MaxBrandInjectRunes` | `500` | brand kit block |
| `MaxCampaignProducts` | `8` | campaign product list |

Classic OpenAI-compatible chat completions use `max_tokens`. GPT-5 / o-series use `max_completion_tokens` (budget includes hidden reasoning tokens). Length-cap handling lives in `OpenAIClient.doComplete` / `CompleteWithOptions`.

## Prompt style

- Short system prompts with **bullet rules** + explicit JSON schema
- **One** few-shot example max
- Brand kit as compact bullets: `Brand:\n- tone: …\n- do: …`
- Product context: category + name + truncated desc + priority attrs only

## JSON reliability

`processing.CompleteJSON`:

1. Call Completer with completion budget + low temperature (classic) / GPT-5.6 param mapping
2. `StripJSONFences` / isolate `{…}`
3. On parse fail → **one retry** with `INVALID. Reply with ONLY one JSON object…`
4. Call sites fall back to originals/templates instead of storing garbage prose as titles

`OpenAIClient` additionally treats `finish_reason=length` with empty or unparseable JSON as failure (not success) and retries once at `MaxTokensEnhanceRetry`.

## Ops tips

- Prefer LAN TCP reachability checks (`Test-NetConnection host -Port 8767`) over ICMP
- Smoke: [ai-full-smoke.md](ai-full-smoke.md), [green-chat-llm.md](green-chat-llm.md)
- CI / no real model: [mock-llm.md](mock-llm.md) (`OPENAI_BASE_URL=http://127.0.0.1:18767/v1`) — includes **translation + processing verification**
- Locale lists + commented `OPENAI_*` / `MOCK_LLM_*` keys: root [`.env.example`](../.env.example)
- Do not commit real keys or LAN secrets