[WSJF-6] Stress-test parser assumptions on model upgrade #12

New issue

Open

opened 2026-06-27 10:45:50 +02:00 by bullitt · 0 comments

bullitt commented

2026-06-27 10:45:50 +02:00

Owner

Context

Article: 'Every harness component encodes assumption about model limitations. Assumptions may be quickly stale.'

Problem

When OpenAI releases a new model (or we switch from gpt-4o to gpt-4o-mini), existing parsers might produce different results. No mechanism to verify.

Proposal

Script: re-run all stored parsers against their original fingerprint emails (from fixtures)
Compare extraction results with stored expectations
Flag parsers that produce different results after model change
Run as part of model migration checklist

WSJF: 6

## Context Article: 'Every harness component encodes assumption about model limitations. Assumptions may be quickly stale.' ## Problem When OpenAI releases a new model (or we switch from gpt-4o to gpt-4o-mini), existing parsers might produce different results. No mechanism to verify. ## Proposal - Script: re-run all stored parsers against their original fingerprint emails (from fixtures) - Compare extraction results with stored expectations - Flag parsers that produce different results after model change - Run as part of model migration checklist ## WSJF: 6