66 lines
3.3 KiB
Markdown
66 lines
3.3 KiB
Markdown
# Local LLM tuning (weak / 8k-class models)
|
|
|
|
Defaults for OpenAI-compatible local servers such as Green Chat Gemma 12B (`overloaded-local`, ~8k context).
|
|
|
|
## Env
|
|
|
|
| Variable | Typical local value |
|
|
|----------|---------------------|
|
|
| `OPENAI_BASE_URL` | `http://<host>:8767/v1` |
|
|
| `OPENAI_MODEL` | id from `GET /v1/models` (e.g. `overloaded-local`) |
|
|
| `OPENAI_API_KEY` | required non-empty for `Enabled()` |
|
|
| `PROCESSING_RPM` | `60` |
|
|
| `PROCESSING_MAX_RETRIES` | `3` |
|
|
|
|
Restart **both** `api` and `worker` after changing `OPENAI_*`. Worker logs should say `OpenAI enabled base=… model=…`, not `heuristic completer`.
|
|
|
|
## Product enhance model choice
|
|
|
|
Admin AI role for **product enhance** should prefer a **fast non-reasoning chat model** that emits `message.content` (JSON) under budget.
|
|
|
|
Reasoning / “code-fast” style models often spend the entire `max_tokens` budget in `reasoning_content`, then return `finish_reason=length` with empty or truncated JSON. Descrybe retries once at `MaxTokensEnhanceRetry` and may synthesize formula HTML as salvage — that is **not** live LLM formula compliance.
|
|
|
|
If enhance logs show `finish_reason=length` with empty/truncated content even after the retry, switch Admin enhance to a non-reasoning model.
|
|
|
|
## Hardening defaults (code)
|
|
|
|
| Constant | Value | Where |
|
|
|----------|------:|-------|
|
|
| `DefaultStructuredTemp` | `0.2` (capped ≤0.3) | `processing/llm_json.go`, `openai.go` |
|
|
| `MaxTokensEnhance` | `16384` | product title/description (reasoning + formula HTML JSON) |
|
|
| `MaxTokensEnhanceRetry` | `32768` | one-shot bump on `finish_reason=length` empty/truncated JSON |
|
|
| `MaxTokensSEO` | `180` | meta title/description |
|
|
| `MaxTokensCampaign` | `650` | email JSON |
|
|
| `MaxProductDescRunes` | `400` | user context |
|
|
| `MaxAttrKeys` / `MaxAttrValueRunes` | `10` / `60` | attrs in enhance prompt |
|
|
| `MaxBrandInjectRunes` | `500` | brand kit block |
|
|
| `MaxCampaignProducts` | `8` | campaign product list |
|
|
|
|
OpenAI-compatible chat completions here use a single `max_tokens` completion budget (no separate reasoning vs content split on typical gateways). Length-cap handling lives in `OpenAIClient.doComplete` / `CompleteWithOptions`.
|
|
|
|
## Prompt style
|
|
|
|
- Short system prompts with **bullet rules** + explicit JSON schema
|
|
- **One** few-shot example max
|
|
- Brand kit as compact bullets: `Brand:\n- tone: …\n- do: …`
|
|
- Product context: category + name + truncated desc + priority attrs only
|
|
|
|
## JSON reliability
|
|
|
|
`processing.CompleteJSON`:
|
|
|
|
1. Call Completer with `max_tokens` + low temperature
|
|
2. `StripJSONFences` / isolate `{…}`
|
|
3. On parse fail → **one retry** with `INVALID. Reply with ONLY one JSON object…`
|
|
4. Call sites fall back to originals/templates instead of storing garbage prose as titles
|
|
|
|
`OpenAIClient` additionally treats `finish_reason=length` with empty or unparseable JSON as failure (not success) and retries once at `MaxTokensEnhanceRetry`.
|
|
|
|
## Ops tips
|
|
|
|
- Prefer LAN TCP reachability checks (`Test-NetConnection host -Port 8767`) over ICMP
|
|
- Smoke: [ai-full-smoke.md](ai-full-smoke.md), [green-chat-llm.md](green-chat-llm.md)
|
|
- CI / no real model: [mock-llm.md](mock-llm.md) (`OPENAI_BASE_URL=http://127.0.0.1:18767/v1`) — includes **translation + processing verification**
|
|
- Locale lists + commented `OPENAI_*` / `MOCK_LLM_*` keys: root [`.env.example`](../.env.example)
|
|
- Do not commit real keys or LAN secrets
|