# Kevo Flagging, Review & Knowledge Correction — Implementation Plan

Status: awaiting approval · 12 Nov 2026
Companion: [API_NOTES.md](./API_NOTES.md) (verified ElevenLabs endpoints, 12 Nov 2026)

## What this builds

Any mistake, knowledge gap or unfulfillable request in a Kevo (ElevenLabs agent) conversation
gets flagged — by the agent itself in-call, by post-call analysis, or manually by a reviewer —
and lands in one Flags queue in the Console. Reviewers approve corrections; a publisher pushes
them into one managed knowledge-base document on the agent. Data-change requests and escalations
become ClickUp tasks. Human-in-the-loop only.

## Decisions already made (with Shiju)

- The Console's **Chatbot section is replaced by the Kevo module**. The legacy prompts/agents/
  history pages (UI over the retired Python chatbot) are not reused.
- UI follows the **promo-code module's design system** (PromoThemeShell + Mantine), with the
  accent set to Karma bronze `#8c7451` and the Karma DS content rules (red = problem only,
  green = done only, plain language, serif-style big numbers).
- Backend is **native core** (Hono + kysely + Postgres + BullMQ) — nothing goes through the
  Python proxy client.

## Key facts from API verification that shaped the plan

- KB document **content is updatable in place** (`PATCH /v1/convai/knowledge-base/{id}` with
  `content`) and re-chunks/re-embeds automatically → publish is create-once then update,
  no create-swap-delete dance.
- Tools are standalone (`/v1/convai/tools` CRUD) and attached via
  `conversation_config.agent.prompt.tool_ids` (inline tools deprecated). Secret headers come
  from workspace secrets (`/v1/convai/secrets`).
- Webhook signature: `ElevenLabs-Signature: t=<ts>,v0=<hex>`, HMAC-SHA256 over
  `"{timestamp}.{raw_body}"`, 30-minute staleness window. Retries are opt-in, 5 attempts.
- `simulate-conversation` is **deprecated with removal 31 Oct 2026 (already past)** → the
  post-publish smoke test uses the replacement agent-testing API
  (`/v1/convai/agent-testing/create` + `/v1/convai/agents/{id}/run-tests`), as an optional phase.
- Not documented anywhere: per-conversation dashboard deep-link format (we'll use
  the dashboard's conversation-history URL pattern and keep it in one helper), and REST
  rate-limit numbers.

## Repo conventions this follows

- Backend template: the **account-deletion** module (route → controller →
  `internal/kevo/{service,repository}.ts`, `BaseRepository`, response envelope from
  `lib/response.ts`).
- Migrations: `apps/core/src/internal/datastore/migrations/YYYYMMDDHHMMSS_*.ts`, idempotent,
  text status columns (no PG enums), seeds with `onConflict doNothing`.
- Async with retry/backoff: **BullMQ** (same options pattern as `notification-target.queue.ts`).
- Inbound secret checks reuse the `timingSafeEqual` style from `middlewares/apiKeyAuth.ts`;
  public endpoints mount outside the session-gated `/v1/admin-console`, like edm-internal.
- Per-domain audit table (the `dashboard_audit_log` pattern): best-effort, never rolls back
  the mutation it describes.
- Console: central `routes.ts` registration, `page.tsx` loader + `_client.tsx`,
  `lib/features/kevo/{query,action,types}.ts` on `createCoreAPIClient`, `@mantine/form` +
  `zod4Resolver`, `createRoleProtectedLoader` + `FEATURE_SLUGS` gating.

## Roles (reusing the existing RBAC matrix — no new mechanism)

Two new `console_features` seeds, plus both slugs added to the `AdminAccessFeatureType` union
and `FEATURE_SLUGS` / `NAV_ACCESS_ORDER`:

| Feature slug | read | create | update | delete |
|---|---|---|---|---|
| `kevo` | view flags/knowledge/stats (Viewer) | manual flag, write correction | approve/reject/route (Reviewer) | retire knowledge entries |
| `kevo-publish` | see publish preview | — | publish to agent (Publisher) | — |

Super admin bypasses all, as everywhere else.

## Environment variables (all server-side, core)

`ELEVENLABS_API_KEY`, `ELEVENLABS_AGENT_ID`, `ELEVENLABS_WEBHOOK_SECRET`,
`KEVO_TOOL_SHARED_SECRET`, `CLICKUP_API_TOKEN`, `CLICKUP_LIST_ID`,
optional `ELEVENLABS_BASE_URL` (default `https://api.elevenlabs.io`).
Runtime state that isn't config — the managed KB document id, last publish hash — lives in the
existing `console_settings` table, not env.

## Database (one migration + one RBAC seed migration)

- `kevo_flags`: id uuid, `reference` text unique (`FLG-` + sequence), `source`
  (IN_CALL|POST_CALL|MANUAL), `category`, `status`
  (NEW|IN_REVIEW|APPROVED|REJECTED|PUBLISHED|ROUTED|RESOLVED), `conversation_id`, `member_id`
  (reference only — no raw PII), `member_question`, `agent_answer`, `member_claim`, `entity`,
  `requested_change`, `reviewer_id` FK console_users (set null), `reviewer_notes`,
  `clickup_task_id`, timestamps.
  Dedupe: unique index on `(conversation_id, category, coalesce(entity,''))`.
- `kevo_knowledge_entries`: id, `flag_id` nullable FK, `entity`, `aliases text[]`, `property`,
  `category` (amenity|policy|faq|correction), `verified_text`, `status`
  (DRAFT|APPROVED|PUBLISHED|RETIRED), `approved_by`, `approved_at`, `published_at`, `version`.
- `kevo_conversations` (lightweight cache): `conversation_id` PK, `member_id`, `summary`,
  `analysis_json` jsonb, `created_at`. Transcripts are never stored — fetched on demand.
- `kevo_publish_log`: id, `kb_document_id`, `entries_count`, `content_hash`, `published_by`,
  `published_at`, `result`.
- `kevo_audit_log` (dashboard_audit_log pattern): `flag_id`/`entry_id` nullable, `action`
  (`flag.created`, `flag.status_changed`, `entry.approved`, `publish.succeeded`, …),
  `actor_email`, `metadata` jsonb, `created_at`. Every status change writes here.

## Core backend

New ElevenLabs client `apps/core/src/lib/elevenlabs/client.ts`: thin typed fetch wrapper
(xi-api-key header, JSON errors surfaced); retries come from BullMQ job attempts, not the
client, per repo convention. ClickUp client `apps/core/src/lib/clickup/client.ts`, same shape.

Public routes (new `routes/kevo/kevo-public.route.ts`, mounted at `/v1/kevo`, NOT session-gated):

1. `POST /v1/kevo/tools/flag-issue` — the agent's server tool.
   - Auth: `X-Kevo-Secret` vs `KEVO_TOOL_SHARED_SECRET`, timing-safe.
   - Rate limit: fixed-window Redis INCR (the repo's first limiter — kept minimal).
   - zod-validated body per spec; inserts the flag inline (single insert, well under 1.5 s)
     and returns `{"status":"flagged","reference":"FLG-1042"}`.
   - Dedupe hits the unique index; on conflict returns the existing reference.
2. `POST /v1/kevo/webhooks/post-call`
   - Verifies `ElevenLabs-Signature` (HMAC over `"{t}.{raw_body}"`, reject >30 min old),
     dedupes on `(conversation_id, event type)`, responds 200 immediately, enqueues a BullMQ
     job (`kevo-postcall` queue, attempts 6, fixed backoff — same options as the existing queue).
   - The job stores the `kevo_conversations` row and auto-creates POST_CALL flags from failed
     evaluation criteria / populated data-collection fields, unless an in-call flag already
     covers that (conversation, topic).
   - The global activity-log middleware gets a skip for `/v1/kevo/*` so webhook payloads
     (member data) are never written to `admin_activity_log` — required by the no-PII rule.

Admin routes (`/v1/admin-console/kevo/*`, `withConsoleUserSession` + `withConsoleUserRole`):
flags list (POST with filters, promo-bookings style) / flag detail / status transitions with
reviewer notes; on-demand conversation transcript (proxied from ElevenLabs, relevant turn
located by matching the flagged question); knowledge entries CRUD + approve; publish preview
(rendered markdown + diff vs last published hash) and publish; ClickUp task creation
(enqueued, retried); manual flag creation from a conversation id; stats (flags/week by
category, top entities, time-to-resolution — plain SQL aggregates).

Publish flow (service, `kevo-publish` RBAC):
1. Render all APPROVED+PUBLISHED entries to markdown grouped by property; SHA-256 hash;
   skip if unchanged.
2. First publish: `POST /v1/convai/knowledge-base/text`, save document id in
   `console_settings`, then merge-safe agent PATCH — read
   `conversation_config.agent.prompt.knowledge_base`, append our locator, write back,
   never touching other entries.
3. Later publishes: `PATCH /v1/convai/knowledge-base/{id}` with new content (re-indexes
   automatically per docs).
4. Mark entries PUBLISHED, write `kevo_publish_log`, flip linked flags, audit.

Setup script `apps/core/src/cmd/kevo-setup.ts` (idempotent, run with tsx): upserts the
`flag_issue` tool in `/v1/convai/tools` (secret header via workspace secret), merges its
tool_id into the agent's `prompt.tool_ids`, sets `platform_settings.evaluation.criteria`
(`answered_correctly_without_gaps`, `member_disputed_information`, `unresolved_request`) and
`platform_settings.data_collection` (`knowledge_gap_topic`, `disputed_fact`,
`data_change_requested`). Post-call webhook registration is workspace-level → documented as
manual steps in the README.

## Console UI (new `/admin/kevo`, replaces Chatbot in the nav)

`lib/features/kevo/` + `routes/kevo/`:

1. **Flags queue** — stat tiles (open flags, needs review, routed, avg resolution) + filter
   drawer (category/status/source/date/property) + table: reference, category, entity,
   member question, source, age. Weekly-by-category chart on the same page (promo chart kit).
2. **Flag detail** (`$reference`) — flag fields; "View conversation" panel fetching the
   transcript on demand with the relevant turn highlighted and tool calls shown; deep link to
   the ElevenLabs dashboard; actions by category (correction form with Approve/Reject/
   Duplicate, or Create-ClickUp-task with status shown).
3. **Knowledge** — list/edit/retire entries; **Publish** button (kevo-publish only) opens the
   diff preview first.
4. **Manual flag** — paste a conversation id, fetch transcript, pick the turn, flag it.

System-prompt addition text goes in `docs/kevo-flagging/system-prompt-addition.md` (pasted
into the agent by hand, per spec).

## Tests & docs

- Add **vitest to apps/core** (first test setup in core — dev-dep only, no turbo changes
  beyond a `test` script). Units: HMAC verification (valid/invalid/stale), flag dedupe,
  KB markdown generation, merge-safe agent PATCH (mocked fetch). Integration: flag-issue and
  webhook endpoints against mocked ElevenLabs.
- `docs/kevo-flagging/README.md`: setup, env vars, tool registration, webhook configuration
  in ElevenLabs settings, runbook (rotate secrets, re-publish, replay a webhook).

## Order of work (after approval)

2. Migrations + RBAC seeds + feature-slug wiring. 3. flag-issue endpoint + setup script.
4. Post-call webhook + BullMQ job + analysis config. 5. Console UI. 6. Publishing + optional
smoke test via agent-testing API. 7. ClickUp. 8. Tests + README.
Numbers match the task spec; each lands as its own reviewable commit on a feature branch.

## Deliberately out of scope (v1)

- ClickUp status sync-back (poll/webhook) — task id + link stored; sync added later if needed.
- Chat-history browsing beyond what flag review needs (ElevenLabs dashboard owns that).
- Autonomous publishing — every path to the agent goes through a human approve + publish.
