A voice-editing test corpus for names, numbers and code

Command versus content boundary

Every utterance should be tagged before any transcription comparison. Tag each sample as command, content, or mixed. Mixed cases matter most, because a spoken phrase can contain both intent and literal text.

Define the exact token that switches modes. Whether it is a wake phrase or a gesture, keep that marker in the audio metadata rather than inferring it only from text output. This prevents a command from silently becoming part of the dictated document.

Identifier preservation

A test row should store the canonical identifier and compare it byte for byte. Names, email handles, product codes, and file paths each need their own expected form. Diacritics, capitalization, hyphenation, and spacing must be explicit in the expected transcript, because a correct scheme can still differ in presentation.

Include pairs that differ by one character, such as invoice numbers with swapped digits. These expose normalization errors that broad accuracy scores hide. Also include identifiers containing rare punctuation so the evaluator checks whether punctuation came from speech or from cleanup rules.

Numerals and code tokens

Decide the canonical numeral form for each domain before collection. Spoken cardinal numbers, ordinal markers, versions, and addresses often map to different written forms. The corpus should store both the audio transcript and the intended target form so evaluators can see conversion rules rather than guess them.

Code tokens are a distinct category. Function names, flags, operators, and delimiters should be tested as literal strings. Add samples where spoken modifiers sit beside code tokens, so boundary detection is measured directly. The expected output must state delimiters exactly.

Corrections and undo

Each correction sample needs three layers: initial utterance, correction command, and resulting expected text. Track whether the tool should replace a word, a span, or the last phrase. Distinguish semantic correction from formatting correction, because both alter output but need different validation.

Undo tests require ordered event history. Record the full edit sequence before the undo trigger, then the expected restored state. Test single undo, repeated undo, and undo after a correction that itself changed punctuation. Include invalid undo calls, such as undo with nothing to revert, to confirm safe no-op behavior.

Coverage matrix

Build coverage from dimensions, not random prompts. Dimensions include speaker clarity, accent variation, background noise, speaking rate, command vocabulary, identifier length, numeral style, code syntax, and correction depth. Cross at least two levels of each dimension across hundreds of samples rather than thousands of easy repeats.

Balance command frequency with content frequency. If commands dominate, metrics overstate control usability while hiding dictation quality. Reserve a held-out set with unseen names and code tokens so results reflect generalization rather than memorization.

Capture permissions and session state

Web-based test harnesses that capture audio need explicit user permission before recording. The permission state affects data collection and must be logged per session. Tests should not assume granted access, and any capture pipeline should handle denial or revocation gracefully.

Store device and permission metadata alongside each sample. This keeps corpus comparisons fair when rerunning evaluations on different machines or browsers.

Scoring and failure tracking

Report separate metrics for command recognition, content fidelity, identifier exact match, numeral conversion, code token fidelity, correction success rate, and undo state accuracy. A single blended score would hide failures in high-stakes tokens like names and numbers.

Also log confidence calibration. For correction and undo, measure whether the system only applies an action when confidence exceeds a documented threshold. Store failure categories so regressions can be traced to parsing, transcription, or policy layers.

Checklist

• Tag every sample as command, content, or mixed before scoring.
• Store canonical identifiers with exact capitalization and punctuation.
• Define numeral and code token target forms in the dataset schema.
• Record ordered edit events for correction and undo tests.
• Keep metrics separate and retain a held-out generalization set.