greeneclipse
8580c996c3
Drop one-shot tmp/axe scripts and agent i18n scratch so the Gitea tree is deployable.
53 lines
2.2 KiB
Markdown
53 lines
2.2 KiB
Markdown
# Local LLM tuning (weak / 8k-class models)
|
|
|
|
Defaults for OpenAI-compatible local servers such as Green Chat Gemma 12B (`overloaded-local`, ~8k context).
|
|
|
|
## Env
|
|
|
|
| Variable | Typical local value |
|
|
|----------|---------------------|
|
|
| `OPENAI_BASE_URL` | `http://<host>:8767/v1` |
|
|
| `OPENAI_MODEL` | id from `GET /v1/models` (e.g. `overloaded-local`) |
|
|
| `OPENAI_API_KEY` | required non-empty for `Enabled()` |
|
|
| `PROCESSING_RPM` | `60` |
|
|
| `PROCESSING_MAX_RETRIES` | `3` |
|
|
|
|
Restart **both** `api` and `worker` after changing `OPENAI_*`. Worker logs should say `OpenAI enabled base=… model=…`, not `heuristic completer`.
|
|
|
|
## Hardening defaults (code)
|
|
|
|
| Constant | Value | Where |
|
|
|----------|------:|-------|
|
|
| `DefaultStructuredTemp` | `0.2` (capped ≤0.3) | `processing/llm_json.go`, `openai.go` |
|
|
| `MaxTokensEnhance` | `350` | product title/description |
|
|
| `MaxTokensSEO` | `180` | meta title/description |
|
|
| `MaxTokensCampaign` | `650` | email JSON |
|
|
| `MaxProductDescRunes` | `400` | user context |
|
|
| `MaxAttrKeys` / `MaxAttrValueRunes` | `10` / `60` | attrs in enhance prompt |
|
|
| `MaxBrandInjectRunes` | `500` | brand kit block |
|
|
| `MaxCampaignProducts` | `8` | campaign product list |
|
|
|
|
## Prompt style
|
|
|
|
- Short system prompts with **bullet rules** + explicit JSON schema
|
|
- **One** few-shot example max
|
|
- Brand kit as compact bullets: `Brand:\n- tone: …\n- do: …`
|
|
- Product context: category + name + truncated desc + priority attrs only
|
|
|
|
## JSON reliability
|
|
|
|
`processing.CompleteJSON`:
|
|
|
|
1. Call Completer with `max_tokens` + low temperature
|
|
2. `StripJSONFences` / isolate `{…}`
|
|
3. On parse fail → **one retry** with `INVALID. Reply with ONLY one JSON object…`
|
|
4. Call sites fall back to originals/templates instead of storing garbage prose as titles
|
|
|
|
## Ops tips
|
|
|
|
- Prefer LAN TCP reachability checks (`Test-NetConnection host -Port 8767`) over ICMP
|
|
- Smoke: [ai-full-smoke.md](ai-full-smoke.md), [green-chat-llm.md](green-chat-llm.md)
|
|
- CI / no real model: [mock-llm.md](mock-llm.md) (`OPENAI_BASE_URL=http://127.0.0.1:18767/v1`) — includes **translation + processing verification**
|
|
- Locale lists + commented `OPENAI_*` / `MOCK_LLM_*` keys: root [`.env.example`](../.env.example)
|
|
- Do not commit real keys or LAN secrets
|