Building AI voice applications that people actually enjoy

In a text interface, a two second wait reads as thoughtfulness. In a phone call it reads as a dropped line. The person on the other end starts talking again, the model hears the interruption as new input, and the conversation collapses into two people apologising to each other.

So the first thing we build for any voice product is a latency budget, written down before any model is chosen. Speech capture and endpointing, transport, transcription, model inference, speech synthesis, and playback each get a millisecond allowance. The total needs to land near the rhythm of human conversation — roughly the length of a natural pause, not the length of a page load.

Once that budget exists, most architecture decisions make themselves. Streaming everywhere. No batching. No step that waits for a complete utterance when it could start on a partial one. A cheaper model that answers immediately beats a smarter model that answers late, because the user cannot hear intelligence they never waited around for.

Humans negotiate turns with breath, pitch and micro-pauses. A voice pipeline has to approximate that with endpointing thresholds, and every threshold is a trade-off. Cut in too early and you interrupt a child mid-sentence. Wait too long and the silence feels like the system has died.

The setting that worked for us was not one number but a set of them, chosen by context. During an open question, wait longer — people think out loud. After a yes/no prompt, cut in fast. When the caller is a child, extend everything, because children pause in places adults do not.

Barge-in matters just as much. If the user starts speaking while the system is talking, synthesis must stop within a couple of hundred milliseconds. A voice that talks over you is instantly, viscerally unpleasant in a way no transcript ever shows.

A voice persona is a constraint system. Tone, vocabulary ceiling, sentence length, what the character knows, what the character must never claim, and how it recovers when it does not understand. We treat those as testable properties rather than vibes in a prompt.

Sentence length in particular is a technical lever. Short sentences synthesise faster, stream sooner, and survive interruption gracefully. A model instructed to answer in one or two sentences feels dramatically more responsive than the same model given no length guidance, with no change to the underlying inference speed.

Recovery lines are worth writing by hand. When transcription fails or the model stalls, a warm in-character fallback keeps the illusion intact. A generic error message ends it permanently.

Voice products collect the most personal data there is, often from the people least able to consent to it. Our position is to hold as little as possible and to hold it for as short a time as possible.

In practice that means personalisation context is assembled just before a call and discarded after, audio is not retained by default, and nothing spoken in a session is used to train anything. Where a transcript is needed for support or abuse review, it is scoped, time-limited and explicitly disclosed.

Content safety has to run inside the latency budget too. Guardrails that add a second of round-trip are guardrails that will eventually be turned off under pressure, so we prefer constraint at the prompt and persona layer, plus fast classifiers, over heavyweight post-hoc review.

Prototype on the phone, not in a browser tab. The telephony path has its own codecs, jitter and echo characteristics, and a demo that sounds perfect over a laptop microphone can be unusable over a real call.

Test with the actual audience early. Adults speak in complete sentences to computers; children do not, and neither do people in a hurry, in a car, or with a strong regional accent.

Finally, decide what the product does when the AI is unavailable. A voice product with no graceful degradation is a product that occasionally, publicly, does nothing at all.