bullitt/trackbox

[WSJF-8] Parser quality self-test: verify new parsers against source email #3

New issue

Open

opened 2026-06-27 10:45:03 +02:00 by bullitt · 0 comments

bullitt commented

2026-06-27 10:45:03 +02:00

Owner

Context

Self-evaluation bias: AI 'confidently praises' its own field_maps even when quality is mediocre.

Problem

AI generates a field_map and we store it without verification. The parser might not actually work on the email that created it.

Proposal

After AI generates field_map, immediately apply it to the same email body
Compare deterministic extraction result with AI's direct extraction
If they disagree on >1 field, mark parser as low-confidence or discard it
Log mismatches for debugging

WSJF: 8

## Context Self-evaluation bias: AI 'confidently praises' its own field_maps even when quality is mediocre. ## Problem AI generates a field_map and we store it without verification. The parser might not actually work on the email that created it. ## Proposal - After AI generates field_map, immediately apply it to the same email body - Compare deterministic extraction result with AI's direct extraction - If they disagree on >1 field, mark parser as low-confidence or discard it - Log mismatches for debugging ## WSJF: 8

bullitt added the

label

2026-06-27 10:45:03 +02:00

bullitt referenced this issue

2026-06-27 15:41:14 +02:00

Dashboard: Group sort-field select and sort-direction button visually #24

bullitt referenced this issue from a commit

2026-06-27 17:09:53 +02:00

security: implement all 10 expert findings — auth, key protection, input sanitization, rate limiting, error scrubbing, non-root container

bullitt referenced this issue from a commit

2026-06-27 17:09:53 +02:00

refactor: implement architect findings — auth, scrape_log retention, settings cache, DB indexes, async ingest, metrics, retention consolidation

bullitt referenced this issue from a commit

2026-06-27 17:09:53 +02:00

feat: implement PM findings — manual add, ETA field, stalled reset, webhook notifier, ETA highlight, bulk archive

bullitt referenced this issue from a commit

2026-06-28 16:10:38 +02:00

security: implement all 10 expert findings — auth, key protection, input sanitization, rate limiting, error scrubbing, non-root container

bullitt referenced this issue from a commit

2026-06-28 16:10:38 +02:00

refactor: implement architect findings — auth, scrape_log retention, settings cache, DB indexes, async ingest, metrics, retention consolidation

bullitt referenced this issue from a commit

2026-06-28 16:10:38 +02:00

feat: implement PM findings — manual add, ETA field, stalled reset, webhook notifier, ETA highlight, bulk archive

Sign in to join this conversation.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

bullitt/trackbox#3

No description provided.