146 lines
6.1 KiB
Markdown
146 lines
6.1 KiB
Markdown
# AI call inspector (Admin → AI calls)
|
||
|
||
**Admin → AI calls** (`/admin/ai-calls`, platform admin only) shows the exact
|
||
system prompt, user prompt and raw response of every LLM call, per tenant and per
|
||
user.
|
||
|
||
Before this existed the prompt was only ever in the worker log, truncated to 500
|
||
runes from each end, and `processed_products.gpt_response` kept response metadata
|
||
without the prompt at all — so "what exactly did we send for this job" had no
|
||
answer.
|
||
|
||
## What is captured
|
||
|
||
Everything resolved through `aiprovider.Service`, tagged by role:
|
||
|
||
| role | where it comes from |
|
||
|---|---|
|
||
| `processing` | product enhance (title / description / meta / attrs) |
|
||
| `categorize` | taxonomy pick when the product has no category |
|
||
| `seo_meta` | standalone SEO meta generation |
|
||
| `campaign` | marketing email generation |
|
||
| `support` | support draft assist |
|
||
|
||
Each row carries company, the user who started the work, job id, raw product id,
|
||
model, provider mode, finish reason, token counts, duration, and the provider error
|
||
when the call failed.
|
||
|
||
## How it works
|
||
|
||
- `internal/aiaudit` is a leaf package: context helpers plus the writer. Callers
|
||
attach detail with `aiaudit.WithCall(ctx, …)`; `ProcessJob` sets company/user/job
|
||
once and `processOne` narrows it to the product.
|
||
- `aiprovider` wraps every Completer it hands out (`audit.go`). One capture point,
|
||
so a new caller cannot forget to log. The wrapper forwards
|
||
`CompleterWithOptions` when the client has it — otherwise every enhance call
|
||
would silently lose `MaxTokens` / `Temperature` / `ReasoningEffort`.
|
||
- Capture never breaks the call it observes: write failures are swallowed after one
|
||
log line, and the insert uses `context.WithoutCancel` so a cancelled job still
|
||
records the call that was in flight.
|
||
|
||
## Cost (Admin → AI costs)
|
||
|
||
`/admin/ai-costs` answers "what does each user cost me": spend by user, by company
|
||
and by model, with a grand total across all tenants.
|
||
|
||
It reads `ai_usage_daily`, a rollup written next to every captured call — one row
|
||
per day / company / user / model / role. That is why it can be kept for
|
||
`aiaudit.UsageRetentionDays` (**3 years**) while the prompt bodies expire in a week:
|
||
a busy tenant adds tens of rows a day, not one per product.
|
||
|
||
Money is **micro-USD integers** end to end (`cost_micros`). Summing millions of
|
||
fractional-cent calls as floats drifts; integers add up exactly. `cost_usd` in the
|
||
API is a display convenience — never re-sum it.
|
||
|
||
Prices live in `ai_model_prices`, seeded with the OpenAI GPT-5.6 list prices
|
||
effective 2026-07-30 (per 1M tokens):
|
||
|
||
| model | input | cached input | output |
|
||
|---|---|---|---|
|
||
| gpt-5.6-luna | $0.20 | $0.02 | $1.20 |
|
||
| gpt-5.6-terra | $2.00 | $0.20 | $12.00 |
|
||
| gpt-5.6-sol | $5.00 | $0.50 | $30.00 |
|
||
|
||
Update the table when list prices change (`UPDATE ai_model_prices …`). Recorded
|
||
cost is never recalculated, so history keeps the rate that applied at the time.
|
||
`aiaudit.defaultModelPrices` mirrors these as a code fallback for a model with no
|
||
row; an unpriced model records usage at zero cost rather than guessing.
|
||
|
||
Cached prompt tokens are read from `usage.prompt_tokens_details.cached_tokens` and
|
||
billed at the cached rate — they are a *subset* of prompt tokens, so they are
|
||
discounted, never added on top.
|
||
|
||
```
|
||
GET /api/admin/ai-costs?days=30&from=&to=&company_id=
|
||
```
|
||
|
||
### History from before capture existed
|
||
|
||
`cmd/backfill-ai-usage` recovers cost for products processed before this shipped,
|
||
reading the provider usage blocks still stored in `processed_products.gpt_response`.
|
||
|
||
```
|
||
go run ./cmd/backfill-ai-usage # dry run
|
||
go run ./cmd/backfill-ai-usage -apply
|
||
```
|
||
|
||
It is best effort, and the limits are real:
|
||
|
||
- **Only rows that still carry a usage block.** `gpt_response` is overwritten on
|
||
every reprocess and absent on older or failed rows.
|
||
- **No per-user attribution** — `processed_products.user_id` is null on those rows,
|
||
so recovered spend lands under "Unattributed" in the by-user view. By-company and
|
||
the grand total are complete.
|
||
- **The day is `updated_at`**, the last write to the row, not necessarily when the
|
||
call was made.
|
||
- **Cached-token discounts cannot be recovered**, so recovered input is priced as
|
||
fully fresh — an over-estimate, never an under-estimate.
|
||
|
||
Rows are written with `source='backfill'`, so a re-run replaces its own rows and can
|
||
never double-count live capture. By default the scan stops at the first day live
|
||
capture recorded; `-before YYYY-MM-DD` overrides that.
|
||
|
||
## Retention
|
||
|
||
Rows are large and high volume — one per product per language, a few KB each. They
|
||
are a **debugging buffer, not an audit trail**:
|
||
|
||
- `aiaudit.RetentionDays` = 7.
|
||
- The worker prunes on its retention tick (`aiaudit.CleanupExpired`).
|
||
- Each body is capped at 60k runes.
|
||
- Never build billing or reporting on this table.
|
||
|
||
For scale: a 25k-product job writes roughly 50–150 MB, which ages out within a
|
||
week.
|
||
|
||
## API
|
||
|
||
Platform admin only (`RequirePlatformAdmin`; 401 unauthenticated, 403 non-admin).
|
||
|
||
```
|
||
GET /api/admin/ai-calls?company_id=&user_id=&job_id=&raw_product_id=&role=&outcome=&q=&limit=&before=
|
||
GET /api/admin/ai-calls/{id}
|
||
```
|
||
|
||
The list returns lengths plus 240-rune previews — one unfiltered request must never
|
||
stream a tenant's whole prompt corpus. Full bodies come from the detail endpoint,
|
||
one row at a time. `q` searches system prompt, user prompt and response.
|
||
`before` is the RFC3339 cursor returned as `next_before`.
|
||
|
||
## Reading a row
|
||
|
||
Start from the outcome, then the prompt:
|
||
|
||
- `outcome=failed` — the `error` field holds the provider error. If it is a dial or
|
||
5xx error the model never answered and enhance fell back to supplier copy.
|
||
- `role=processing` with a short `user_length` — the category formula probably did
|
||
not key. Check that the prompt contains `GPT predloga:` / `Product template:` and
|
||
a `<name>{` block.
|
||
- Response present but the product still looks like the feed — compare the response
|
||
against the formula sections; a reply that ignores them is rejected by the
|
||
formula gate and marked `synthesized`/`refused` (see docs/category-formulas.md).
|
||
|
||
A job with no rows at all made no LLM calls: it was either hash-skipped
|
||
(`ai_enhance outcome=skip reason=unchanged_hash`), gated by plan/credits, or the
|
||
provider was unset.
|