FamilyBench: A Benchmark for Family-Coordination Extraction

Abstract

Consumer AI increasingly acts as an extraction agent: it reads a natural, messy input, such as a forwarded school email, a coach's group message, or a photographed flyer, and turns it into the structured records a user depends on: calendar events, reminders, and assignments to the right family member. This task resists existing evaluation paradigms. Unlike code generation it has no executable oracle, unlike question answering its correctness is multi-field, and unlike either, the correct output for the same input differs across users because it depends on private household state.

We present FamilyBench, a benchmark for family-coordination extraction. Each instance pairs a frozen family fixture (members, children with grades and schools, timezone, and the current date) with an input message and hand-written per-field gold outcomes. A hermetic harness runs a production extraction pipeline in-process with persistence, search, and notification dependencies faked out, so only the language-model calls remain live, and grades produced records against gold using a declarative vocabulary of named match types plus assertions on pipeline outcome state. Because extraction is stochastic, we adopt the reliability metric pass^k over n trials rather than single-shot accuracy.

We describe the task, the dataset design, and the methodology, and evaluate three triage models across two providers and three capability tiers at n = 5 trials. The evaluation separates the models under a drop-in prompt swap and exposes a capability cliff: a family-context filtering task the incumbent passes on every trial collapses to zero under a naive model swap. A neutral-prompt leaderboard shows that cliff is an artifact of prompt fit, not capability: the ranking flips. The study also reveals model-specific reliability defects, fields extracted non-deterministically, that single-shot evaluation hides. A prompt ablation isolates the value of the production system's heavily tuned prompt: pass^10 of 1.00 versus 0.40 for a generic baseline on tasks that exercise its rules, and no gap on tasks that need none. We release the harness, schema, and seed tasks as an extensible framework.

Citation

@misc{familybench2026,
  title  = {FamilyBench: A Benchmark for Family-Coordination Extraction},
  author = {{The Sense Team}},
  year   = {2026},
  url    = {https://familybench.com/paper}
}

Questions about the benchmark, or interest in running a model against the tasks: get in touch.