bullitt/trackbox

[WSJF-8] Add evaluator step in CI that verifies extraction quality #2

New issue

Open

opened 2026-06-27 10:45:03 +02:00 by bullitt · 0 comments

bullitt commented

2026-06-27 10:45:03 +02:00

Owner

Context

Generator-evaluator loop: separate the agent doing work (ingest) from the agent judging it (quality checker).

Problem

Our smoke test only checks 'did it respond' not 'did it extract correctly.' A parser could return garbage and CI would still pass.

Proposal

Maintain a set of 5-10 golden test emails with expected extractions
CI step after deploy: POST each golden email, verify extracted fields match expected values
Fail pipeline if extraction accuracy drops below threshold

WSJF: 8

## Context Generator-evaluator loop: separate the agent doing work (ingest) from the agent judging it (quality checker). ## Problem Our smoke test only checks 'did it respond' not 'did it extract correctly.' A parser could return garbage and CI would still pass. ## Proposal - Maintain a set of 5-10 golden test emails with expected extractions - CI step after deploy: POST each golden email, verify extracted fields match expected values - Fail pipeline if extraction accuracy drops below threshold ## WSJF: 8

bullitt added the

label

2026-06-27 10:45:03 +02:00

bullitt referenced this issue

2026-06-27 10:45:28 +02:00

[WSJF-7] Structured artifact handoff: save emails as golden test fixtures #6

bullitt referenced this issue

2026-06-27 15:41:14 +02:00

Dashboard: Truncated card titles should wrap to 2 lines instead of clipping #23

bullitt referenced this issue from a commit

2026-06-27 17:09:53 +02:00

security: implement all 10 expert findings — auth, key protection, input sanitization, rate limiting, error scrubbing, non-root container

bullitt referenced this issue from a commit

2026-06-27 17:09:53 +02:00

refactor: implement architect findings — auth, scrape_log retention, settings cache, DB indexes, async ingest, metrics, retention consolidation

bullitt referenced this issue from a commit

2026-06-28 16:10:38 +02:00

security: implement all 10 expert findings — auth, key protection, input sanitization, rate limiting, error scrubbing, non-root container

bullitt referenced this issue from a commit

2026-06-28 16:10:38 +02:00

refactor: implement architect findings — auth, scrape_log retention, settings cache, DB indexes, async ingest, metrics, retention consolidation

Sign in to join this conversation.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

bullitt/trackbox#2

No description provided.