How OpenAI rebuilt WebRTC for real-time voice

When you talk to a voice AI and it answers in what feels like real conversational rhythm, you stop, it starts, you interrupt, it yields, there's a surprisingly old piece of plumbing doing most of the work underneath. It's called WebRTC, it was designed in 2011 to let two browsers do a video call, and it was emphatically not designed to let millions of people simultaneously talk to a model running in a datacentre several thousand kilometres away.

OpenAI just published a long engineering post explaining how they rebuilt their WebRTC stack to make this work at scale. I want to walk through what they actually did, because it's a clean example of an architectural pattern, the relay-plus-transceiver model with geo-steered signalling, that's going to show up in a lot of real-time AI products over the next year. Once you see the shape of it, you'll recognise it everywhere.

Why WebRTC at all. WebRTC is the protocol stack browsers use to send audio and video peer-to-peer with very low latency. The reason voice-AI builders reach for it instead of, say, plain WebSockets is that WebRTC has decades of engineering baked in around the things that matter for live audio: jitter buffers, packet loss concealment, echo cancellation, adaptive bitrate, and, crucially, UDP transport so a single dropped packet doesn't stall the whole stream the way it would over TCP. If you want a model to feel like it's talking to you rather than messaging you, you need all of that. WebRTC gives it to you for free.

The catch is that WebRTC was built for a peer-to-peer world. Two browsers, one connection, roughly symmetric. A voice-AI service is not that. It's millions of clients, each talking to a model that lives on a GPU somewhere specific, with the constraint that the perceived latency, the gap between you finishing your sentence and the model starting its reply, has to stay under roughly 300 milliseconds for the conversation to feel natural. Above that, the magic dies.

The problem the pattern solves

The naive approach breaks in two places. If you just point every client's WebRTC connection straight at the GPU server running the model, you hit two walls. First, the GPU server now has to terminate thousands of WebRTC sessions itself, doing all the codec work, the encryption, the jitter buffering, which is a colossal waste of expensive accelerator-adjacent CPU. Second, and worse, the client's connection has to traverse the public internet all the way to wherever the GPU happens to be. If you're in Sydney and the GPU is in Virginia, you're paying 200ms of round-trip time before the model has even started thinking.

Both problems are about the same underlying mistake: treating the client-to-model path as a single connection. The pattern OpenAI landed on splits it into two.

The shape of the pattern

There are three components, and the trick is in how they hand off to each other.

The edge relay. This is a fleet of small servers deployed in many geographic regions, close, in network terms, to where users actually are. The client's WebRTC connection terminates here, not at the GPU. The relay handles all the WebRTC machinery: the encrypted transport, the codec negotiation, the jitter buffer, the packet recovery. From the client's point of view, this is the server it's talking to.

The transceiver. Behind the relay, there's a separate connection from the relay to the actual model server, the GPU machine doing inference. This connection runs over OpenAI's internal network, which is fast, predictable, and not subject to the chaos of the open internet. The transceiver's job is to take decoded audio frames from the relay, hand them to the model, take the model's generated audio back out, and pipe it to the relay for delivery. The model server never sees a WebRTC connection. It sees a clean stream of audio frames.

Geo-steered signalling. When a client first tries to connect, it doesn't know which relay to talk to. The signalling layer, the thing that does the initial "hello, I want to start a call" handshake, looks at where the client is coming from (by IP geolocation and by measured latency to candidate relays) and steers the client to the nearest healthy one. This is the part that makes the whole thing globally scalable: a user in Sydney connects to a Sydney relay, a user in Frankfurt connects to a Frankfurt relay, and only the relay-to-GPU leg crosses long distances, over a network that's been engineered to handle it.

That's the budget the whole architecture is built around. Everything, the choice of relay, the codec, the buffer sizes, the path between relay and model, is tuned to keep the round trip from end-of-your-speech to start-of-model's-reply inside that window.

A worked example

Imagine you're in London, opening a voice session with a model that happens to be running on a GPU in Iowa.

Your browser hits OpenAI's signalling endpoint. The signalling layer sees you're in London, checks which relays are healthy and underloaded, and tells your browser to establish its WebRTC connection with a relay in, say, Dublin. That's a roughly 15ms round trip, short enough to feel like nothing.

Your microphone audio gets encoded (Opus codec, typically), packetised, encrypted, and streamed to the Dublin relay over UDP. The relay decrypts and decodes, producing a clean stream of audio frames. It hands those frames over its internal connection to the model server in Iowa.

The Iowa server runs inference, generates an audio response, streams it back to Dublin. Dublin re-encodes it, packages it as WebRTC media, and sends it to your browser. Your browser plays it.

The slow part of all this, Dublin to Iowa and back, happens over OpenAI's own backbone, not the public internet. The fast part, your house to Dublin, happens over a short, well-understood link. The total budget stays inside 300ms.

Trade-offs

You're paying for an extra hop. Every audio packet now travels client → relay → model and back, instead of going direct. In the best case (client and GPU in the same region) this adds latency you didn't strictly need. The bet is that the worst case, client far from GPU, gets dramatically better, and the best case is still fast enough.

The relay becomes a critical piece of infrastructure. It has to be highly available, geographically redundant, and capable of failing over without dropping the call. If a relay goes down mid-conversation, you need the client to migrate to another one fast enough that the user notices a hiccup, not a disconnection.

Operational complexity goes up. You now run two distinct fleets, relays and model servers, with different scaling characteristics, different failure modes, and a protocol between them that you own end to end. That protocol is internal, which means you can make it as efficient as you like, but it's also yours to maintain forever.

Adjacent patterns

This pattern is close to a couple of others worth knowing about.

The CDN model, content delivery networks like Cloudflare or Fastly do something architecturally similar for static assets: terminate the user's connection at the edge, fetch from origin behind the scenes. The relay-plus-transceiver pattern is the real-time, bidirectional cousin of that idea.

The SFU (Selective Forwarding Unit), used in group video calls, where a server in the middle takes one participant's media stream and forwards it to the other participants. OpenAI's relay is doing something subtler: it's not forwarding to other humans, it's translating between WebRTC and an internal protocol so the model server can stay simple.

When to reach for it

You want this pattern any time you're building a real-time AI service where the client is on the open internet, the model is on expensive specialised hardware that lives in a small number of locations, and the interaction is tight enough that latency budget actually matters, voice, live video understanding, real-time translation. If your latency budget is multiple seconds, you don't need this; a plain WebSocket will do. If it's hundreds of milliseconds and your users are global, this is the shape your architecture is going to grow into whether you plan it or not.

The interesting thing about OpenAI publishing this is that they've now made the playbook legible. Expect to see a lot of it.

Footnotes and links

Further reading

• WebRTC for the Curious, the best free book on how WebRTC actually works under the hood.
• Opus codec RFC 6716, the audio codec doing the heavy lifting on the wire.
• High Performance Browser Networking, Chapter 18, Ilya Grigorik's chapter on WebRTC, still the cleanest written explanation of the protocol stack.

¶