descrybe/docs/local-llm-tuning.md

Local LLM tuning (weak / 8k-class models)

Defaults for OpenAI-compatible local servers such as Green Chat Gemma 12B (overloaded-local, ~8k context).

Env

Variable	Typical local value
OPENAI_BASE_URL	http://<host>:8767/v1
OPENAI_MODEL	id from GET /v1/models (e.g. overloaded-local)
OPENAI_API_KEY	required non-empty for Enabled()
PROCESSING_RPM	60
PROCESSING_MAX_RETRIES	3

Restart both api and worker after changing OPENAI_*. Worker logs should say OpenAI enabled base=… model=…, not heuristic completer.

Hardening defaults (code)

Constant	Value	Where
DefaultStructuredTemp	0.2 (capped ≤0.3)	processing/llm_json.go, openai.go
MaxTokensEnhance	350	product title/description
MaxTokensSEO	180	meta title/description
MaxTokensCampaign	650	email JSON
MaxProductDescRunes	400	user context
MaxAttrKeys / MaxAttrValueRunes	10 / 60	attrs in enhance prompt
MaxBrandInjectRunes	500	brand kit block
MaxCampaignProducts	8	campaign product list

Prompt style

• Short system prompts with bullet rules + explicit JSON schema
• One few-shot example max
• Brand kit as compact bullets: Brand:\n- tone: …\n- do: …
• Product context: category + name + truncated desc + priority attrs only

JSON reliability

processing.CompleteJSON:

1. Call Completer with max_tokens + low temperature
2. StripJSONFences / isolate {…}
3. On parse fail → one retry with INVALID. Reply with ONLY one JSON object…
4. Call sites fall back to originals/templates instead of storing garbage prose as titles

Ops tips

• Prefer LAN TCP reachability checks (Test-NetConnection host -Port 8767) over ICMP
• Smoke: ai-full-smoke.md, green-chat-llm.md
• CI / no real model: mock-llm.md (OPENAI_BASE_URL=http://127.0.0.1:18767/v1) — includes translation + processing verification
• Locale lists + commented OPENAI_* / MOCK_LLM_* keys: root .env.example
• Do not commit real keys or LAN secrets