Telephony meets streaming AI: wiring a phone call to a model

A call arrives at a cloud telephony service. That service exposes the media as a bidirectional stream — typically 8 kHz mono audio in small frames, over a websocket. Our service sits in the middle: it receives inbound frames, forwards them to speech recognition, feeds recognised text plus session context to a model, and pushes synthesised audio back down the same socket.

Everything in that loop is a stream. Nothing waits for completion. The moment recognition emits a stable partial, the model can start; the moment the model emits its first tokens, synthesis can start; the moment synthesis emits its first frames, they go on the wire.

Telephony audio is narrowband and usually µ-law or A-law encoded. Modern speech models generally want linear PCM at 16 kHz or higher. Every hop between those formats costs conversion time and, more importantly, quality.

Resampling artefacts show up as recognition errors on exactly the words that matter — names, numbers, place names. We found more accuracy improvement from cleaning up the audio path than from swapping recognition models.

The same applies in reverse. A beautifully synthesised 24 kHz voice downsampled clumsily to narrowband sounds thin and robotic. Doing that conversion deliberately, with a decent filter, is audibly better than letting a default do it.

Telephony streams are real-time and unforgiving: frames arrive on a fixed clock whether or not your service is ready. If any downstream stage slows — a model queue, a synthesis cold start — audio backs up somewhere.

The wrong fix is a large buffer, which converts a latency spike into a permanently delayed conversation that never recovers. The right fix is bounded buffers with an explicit drop or truncate policy, plus a watchdog that ends the turn cleanly rather than letting it drift.

We instrument every stage with timestamps on the same clock, so a slow call can be attributed to a specific hop after the fact instead of guessed at.

A call is not a request. It has a lifecycle: ringing, answered, greeting, listening, thinking, speaking, interrupted, wrapping up, hung up, failed. Modelling that explicitly, as a state machine with allowed transitions, removed a whole class of bugs where two subsystems both believed it was their turn to speak.

It also gives you somewhere sensible to put the hard cases. Caller hangs up mid-synthesis. Recognition returns nothing for fifteen seconds. Model errors twice in a row. Each of those is a transition with a defined behaviour rather than an unhandled promise rejection.

Voice workloads are spiky in a way that punishes naive autoscaling. A seasonal product can sit near zero and then take a large share of its annual traffic in a few evenings.

We size for the peak evening rather than the average, keep a warm pool so cold starts never land inside a call, and make the expensive stages independently scalable — recognition, inference and synthesis rarely bottleneck at the same time.

Per-minute cost is worth tracking as a first-class product metric from day one. It shapes what the product can be: a two minute call and a twenty minute call are different businesses, not different settings.