84 lines
3.5 KiB
Markdown
84 lines
3.5 KiB
Markdown
|
# AI call inspector (Admin → AI calls)
|
|||
|
|
|
|||
|
|
**Admin → AI calls** (`/admin/ai-calls`, platform admin only) shows the exact
|
|||
|
|
system prompt, user prompt and raw response of every LLM call, per tenant and per
|
|||
|
|
user.
|
|||
|
|
|
|||
|
|
Before this existed the prompt was only ever in the worker log, truncated to 500
|
|||
|
|
runes from each end, and `processed_products.gpt_response` kept response metadata
|
|||
|
|
without the prompt at all — so "what exactly did we send for this job" had no
|
|||
|
|
answer.
|
|||
|
|
|
|||
|
|
## What is captured
|
|||
|
|
|
|||
|
|
Everything resolved through `aiprovider.Service`, tagged by role:
|
|||
|
|
|
|||
|
|
| role | where it comes from |
|
|||
|
|
|---|---|
|
|||
|
|
| `processing` | product enhance (title / description / meta / attrs) |
|
|||
|
|
| `categorize` | taxonomy pick when the product has no category |
|
|||
|
|
| `seo_meta` | standalone SEO meta generation |
|
|||
|
|
| `campaign` | marketing email generation |
|
|||
|
|
| `support` | support draft assist |
|
|||
|
|
|
|||
|
|
Each row carries company, the user who started the work, job id, raw product id,
|
|||
|
|
model, provider mode, finish reason, token counts, duration, and the provider error
|
|||
|
|
when the call failed.
|
|||
|
|
|
|||
|
|
## How it works
|
|||
|
|
|
|||
|
|
- `internal/aiaudit` is a leaf package: context helpers plus the writer. Callers
|
|||
|
|
attach detail with `aiaudit.WithCall(ctx, …)`; `ProcessJob` sets company/user/job
|
|||
|
|
once and `processOne` narrows it to the product.
|
|||
|
|
- `aiprovider` wraps every Completer it hands out (`audit.go`). One capture point,
|
|||
|
|
so a new caller cannot forget to log. The wrapper forwards
|
|||
|
|
`CompleterWithOptions` when the client has it — otherwise every enhance call
|
|||
|
|
would silently lose `MaxTokens` / `Temperature` / `ReasoningEffort`.
|
|||
|
|
- Capture never breaks the call it observes: write failures are swallowed after one
|
|||
|
|
log line, and the insert uses `context.WithoutCancel` so a cancelled job still
|
|||
|
|
records the call that was in flight.
|
|||
|
|
|
|||
|
|
## Retention
|
|||
|
|
|
|||
|
|
Rows are large and high volume — one per product per language, a few KB each. They
|
|||
|
|
are a **debugging buffer, not an audit trail**:
|
|||
|
|
|
|||
|
|
- `aiaudit.RetentionDays` = 7.
|
|||
|
|
- The worker prunes on its retention tick (`aiaudit.CleanupExpired`).
|
|||
|
|
- Each body is capped at 60k runes.
|
|||
|
|
- Never build billing or reporting on this table.
|
|||
|
|
|
|||
|
|
For scale: a 25k-product job writes roughly 50–150 MB, which ages out within a
|
|||
|
|
week.
|
|||
|
|
|
|||
|
|
## API
|
|||
|
|
|
|||
|
|
Platform admin only (`RequirePlatformAdmin`; 401 unauthenticated, 403 non-admin).
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
GET /api/admin/ai-calls?company_id=&user_id=&job_id=&raw_product_id=&role=&outcome=&q=&limit=&before=
|
|||
|
|
GET /api/admin/ai-calls/{id}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The list returns lengths plus 240-rune previews — one unfiltered request must never
|
|||
|
|
stream a tenant's whole prompt corpus. Full bodies come from the detail endpoint,
|
|||
|
|
one row at a time. `q` searches system prompt, user prompt and response.
|
|||
|
|
`before` is the RFC3339 cursor returned as `next_before`.
|
|||
|
|
|
|||
|
|
## Reading a row
|
|||
|
|
|
|||
|
|
Start from the outcome, then the prompt:
|
|||
|
|
|
|||
|
|
- `outcome=failed` — the `error` field holds the provider error. If it is a dial or
|
|||
|
|
5xx error the model never answered and enhance fell back to supplier copy.
|
|||
|
|
- `role=processing` with a short `user_length` — the category formula probably did
|
|||
|
|
not key. Check that the prompt contains `GPT predloga:` / `Product template:` and
|
|||
|
|
a `<name>{` block.
|
|||
|
|
- Response present but the product still looks like the feed — compare the response
|
|||
|
|
against the formula sections; a reply that ignores them is rejected by the
|
|||
|
|
formula gate and marked `synthesized`/`refused` (see docs/category-formulas.md).
|
|||
|
|
|
|||
|
|
A job with no rows at all made no LLM calls: it was either hash-skipped
|
|||
|
|
(`ai_enhance outcome=skip reason=unchanged_hash`), gated by plan/credits, or the
|
|||
|
|
provider was unset.
|