A benchmark for turning family messages into the right calendar records

The Task

Consumer AI increasingly acts as an extraction agent. It reads a natural, messy input and turns it into structured records a household depends on. This resists existing evaluation paradigms in three ways.

• No executable oracle. Unlike code generation, there is no test suite to run. Correctness is a judgment about dates, times, recurrence, and who the event is for.
• Multi-field correctness. Unlike question answering, an output is right only if every graded field is right. A correct date with the wrong child is a failure a parent notices.
• Household-dependent truth. The correct output for the same email differs across families. A grade-3 field trip is an event for one family and noise for another.

A FamilyBench task is therefore three things: a frozen family fixture (members, children with grades and schools, timezone, and today's date), an input (an email, or an ordered sequence of emails), and gold written by hand for each field that matters.

The gold for that task, as written in the manifest:

"evaluation": {
  "events": [
    { "title":     { "match": "fuzzy", "value": "Science Museum field trip" },
      "startDate": { "match": "exact", "value": "2026-03-19" },
      "members":   { "match": "set",   "value": ["Emma"] } },
    { "title":     { "match": "fuzzy", "value": "City Zoo field trip" },
      "startDate": { "match": "exact", "value": "2026-03-24" },
      "members":   { "match": "set",   "value": ["Noah"] } }
  ]
}

Tiers

• T1, clean. One dated event, one child, nothing to trip over.
• T2, digests. Several events in one message, recurrence, multi-day ranges, dismissal times buried in prose.
• T3, photos. Flyers and screenshots. Planned; no tasks yet.
• T4, adversarial. Filter to the family's own children, produce nothing for a promotional email, merge a re-forwarded message into the existing event instead of duplicating it, keep a recurring schedule as one object across re-descriptions.

How Grading Works

Grading is declared as data, not written as code per task. Each gold field names a match type, and a task passes only when every assertion holds.

• exact for dates, recurrence rules, and identifiers.
• time for clock times, tolerant of formatting.
• fuzzy for titles, token overlap rather than string equality.
• set for the members an event is assigned to, order-free and exact.
• present when a field must exist but its value is not graded.
• state for pipeline outcome counts, such as events_created = 0 on a re-forward that must merge rather than duplicate.

Omitting a key leaves it ungraded. A task that lists "reminders": [] asserts that no reminder is produced; a task that omits the key does not penalize a helpful extra reminder.

Reliability, not accuracy

Extraction is stochastic. The same model on the same email can extract a dismissal time on one run and drop it on the next. A single trial hides that. FamilyBench runs each task n times and reports pass^k, the probability that all k of k trials pass, estimated as in tau-bench, with bootstrap confidence intervals. The headline number is pass^n. Tasks are also bucketed as passed on every trial, flaky, or failed on every trial, because a flaky task is the one worth reading.

Hermetic except the model

The harness runs a system in-process with persistence, search, and notifications faked out. Only the language-model calls stay live. A change to a prompt is exactly what moves the score, which is what makes the benchmark useful as a regression gate on prompt edits.

Any system can be scored

A system is any function from a message and a fixture to produced records. Two ship with the harness: a full production pipeline, and a self-contained reference runner that calls a single model with one neutral prompt and parses its output. The reference runner is how models are compared fairly; the production pipeline is how a product measures itself. Two model-free baselines, one that produces nothing and one built on regular expressions, give the floor and probe whether any task is gameable.

Results

Three tables, each answering a different question. All are prototype-scale and the confidence intervals are wide. Model identifiers are as evaluated in August 2026.

Which model does the task best, under one neutral prompt?

The reference runner is single-shot with no reconciliation, so every model fails the re-forward dedup task by construction. The ranking here is what a model-fair leaderboard looks like when prompt engineering is held constant.

What happens when a product swaps its model without changing its prompt?

The incumbent's prompt was tuned against one provider. Under a drop-in swap, the own-kids filter task collapses from every trial passing to none for both alternative models. Under the neutral prompt above, that cliff disappears and the ranking flips. The cliff measures prompt fit, not capability.

How much is the tuned prompt worth?

Prompt value is concentrated. On tasks that stress a tuned rule, such as fortnightly recurrence or a sent-home-in-backpacks reminder, the generic prompt fails outright. On tasks that need no rule, the two are indistinguishable. An ablation on easy tasks would have reported noise.

Tasks

Gold is small and hand-curated, following the SWE-bench Verified philosophy: fifty trusted tasks beat five thousand machine-labeled ones. Every task carries a shared canary string so a leaked task is detectable in training data. Tasks marked sequence feed an ordered set of emails into one evolving fixture and grade the final state, which is how multi-message reconciliation is tested.

Release and Access

Tasks embed family and children's data, de-identified but still sensitive, so there will not be a fully open dump. Following GAIA, the plan is a small public development split with gold, for iteration and for exercising the reference runner, and a larger held-out test split whose gold is withheld and scored through a submission endpoint.

The current seed tasks are synthetic, realistic patterns rather than real forwards, and were authored alongside the pipeline they test. Growing the set from real de-identified messages, with gold written by annotators who have not seen any pipeline's rules, is the work that turns this from a methodology into a benchmark. The annotation protocol is described in the paper.

To run a model against the seed tasks, or to contribute de-identified tasks, get in touch.

Citation

@misc{familybench2026,
  title  = {FamilyBench: A Benchmark for Family-Coordination Extraction},
  author = {{The Sense Team}},
  year   = {2026},
  url    = {https://familybench.com/paper}
}