[WSJF-3] Re-evaluate: test if gpt-4o-mini suffices for parser generation #18

New issue

Open

opened 2026-06-27 10:46:15 +02:00 by bullitt · 0 comments

bullitt commented

2026-06-27 10:46:15 +02:00

Owner

Context

Article: 'Revisit harness assumptions with each model release.'

Problem

We use gpt-4o for all AI calls. Parser generation might work fine with gpt-4o-mini at 1/15th the cost. Never tested.

Proposal

Run benchmark: 10 real emails through both gpt-4o and gpt-4o-mini extraction
Compare: extraction accuracy, field_map quality, cost
If gpt-4o-mini achieves >80% same quality, switch parser generation to mini
Keep gpt-4o for complex/failed cases only

WSJF: 3 (Medium value, Low urgency, Medium risk reduction, Small size)

## Context Article: 'Revisit harness assumptions with each model release.' ## Problem We use gpt-4o for all AI calls. Parser generation might work fine with gpt-4o-mini at 1/15th the cost. Never tested. ## Proposal - Run benchmark: 10 real emails through both gpt-4o and gpt-4o-mini extraction - Compare: extraction accuracy, field_map quality, cost - If gpt-4o-mini achieves >80% same quality, switch parser generation to mini - Keep gpt-4o for complex/failed cases only ## WSJF: 3 (Medium value, Low urgency, Medium risk reduction, Small size)