“Voice agent evaluation” covers several different problems that are easy to conflate. A benchmark may test speech recognition without testing task completion, or test tool calls without a live spoken interaction. The Voice Agent Benchmark Landscape exists to make those boundaries explicit. This post is about how it was built.
One row per benchmark, one question per field
The dataset has 13 rows in the 1 September 2026 release. For each benchmark it records the primary focus and, as separate fields, whether the benchmark covers spoken input, spoken output, multi-turn interaction, tool use, goal completion, computer or browser action, meeting or long-form material, and real-time operation. It also records whether the data is public, the code licence and the data licence separately, the repository, dataset card and paper addresses, the date last verified, and evidence notes.
Every capability value is one of four words: yes, no, partial or unclear. “Partial” means the capability appears in part of a suite or is evaluated indirectly; it says nothing about quality. “No” means the capability is outside the published scope or absent from the public materials, not that the project could never support it. “Unclear” is used when the public material was insufficient to classify, and it stays until a first-party source settles it.
Sources are first-party, and licences are recorded twice
Each row cites the maintainers' own repository, dataset card or paper; nothing is classified from a secondary write-up. The source audit in the repository lists, per row, the pages that supported the capability values and the pages that state the licences. Code and data licences are recorded separately because they differ more often than people assume: one entry's code and data are under a community licence that is not an open-source licence and restricts commercial use; another's code is Apache-2.0 while its data is CC BY-NC 4.0 and gated. A reader deciding whether to reuse a benchmark needs both facts.
Why there are no scores
A leaderboard needs a shared task and a shared metric. The thirteen projects were designed for thirteen different evaluation questions, so a single ranking would compare things that were not built to be compared. The landscape therefore records what each benchmark measures and nothing about how any system performs on it. Inclusion does not imply endorsement, and the dataset does not contain Cue's private 227-sample production benchmark described in Google DeepMind's case study.
What the map shows is missing
Laid out this way, the public landscape is strongest in conversational customer service, spoken tool selection, voice-assistant content evaluation and speech recognition. It is thin at the intersection of continuous desktop voice typing in arbitrary applications, long-form meeting transcription with structured notes, cross-application computer action, confirmation and reversibility of side effects, and any single evaluation that connects transcription quality to user outcomes. The repository states that as a gap in what exists, not as a defect in any project.
Keeping it honest
- The data file has a documented schema and a validator; a row that breaks the field contract fails the check.
- Every source link is re-checked on a schedule. A definitive 404 or 410 fails the audit; rate limits and transient errors are reported separately so they are not mislabelled as broken links.
- Releases are archived on Zenodo with a DOI per version, and the record carries the dataset's SHA-256, so a citation points at an exact file.
- Corrections name the row, the field, the proposed value and a first-party source, and ship as a new version.
The repository is at github.com/Sophon-LLC/voice-agent-benchmark-landscape; the citation format and version list are on the record page.