How a streaming speech API actually works

xAI shipped standalone Speech-to-Text and Text-to-Speech APIs this week, and the spec sheet is the kind of thing that rewards a closer look. TTS at $4.20 per million characters. STT with word-level timestamps, speaker diarization, a WebSocket endpoint for real-time streaming, and a REST endpoint for batch jobs. Twelve input audio formats. Five voices.

Most coverage will stop at "xAI now does voice." I want to use the launch as an excuse to explain what's actually going on inside an API like this, because once you understand the shape of it, every similar product from OpenAI, Google, Deepgram, ElevenLabs and the rest suddenly makes sense. They are all variations on the same small number of ideas.

I'm going to walk through the STT side in detail, then do TTS more briefly, because the interesting engineering is mostly on the listening side.

What speech-to-text is actually doing

At the bottom, speech is air pressure wobbling over time. A microphone turns those wobbles into numbers, typically 16,000 or 48,000 samples per second, each sample a small integer representing pressure at that instant. A one-minute clip at 16kHz is 960,000 numbers. That's the raw material.

A modern STT model doesn't read those numbers directly. It first converts them into a spectrogram, a picture, essentially, of which frequencies are loud at which moments. Imagine sheet music where the vertical axis is pitch and the horizontal axis is time, and the darkness of each spot tells you how much energy is at that pitch at that moment. Speech has a very characteristic visual signature in this form: vowels show as horizontal bands, consonants as sharp vertical smears, silence as empty space.

Imagine sheet music where the vertical axis is pitch and the horizontal axis is time, and the darkness of each spot tells you how much energy is at that pitch at that moment.

The model, almost always a transformer now, often an encoder-decoder like Whisper or a variant, reads this spectrogram and produces text. Along the way it's doing several things at once: recognising phonemes, assembling them into words, using language-model priors to pick the likely word when the audio is ambiguous ("recognise speech" vs "wreck a nice beach"), and emitting timing information about when each word occurred.

That's the core. Everything else in the API is plumbing around this core.

Why there are two endpoints

xAI, like most providers, offers two ways in: a batch REST endpoint and a real-time WebSocket. These exist because they solve genuinely different problems.

Batch REST is the simple case. You have a finished audio file, a recorded meeting, a podcast, a voicemail. You POST it to the endpoint, the server runs the whole file through the model, and you get back a transcript. The model sees the entire audio at once, which means it can use context from later in the recording to disambiguate earlier bits. If someone says "I work at Apple" at minute two and "I'm a software engineer" at minute five, the model can use the second sentence to be more confident about the first. Latency doesn't matter much; accuracy is the thing.

Real-time WebSocket is the hard case. The audio is still being spoken. You want transcription now, not when the speaker finishes. This is what powers voice assistants, live captioning, and anything where a human is waiting on the other end.

A WebSocket is a persistent two-way connection between your code and the server, unlike a normal HTTP request, which opens, sends, receives, and closes, a WebSocket stays open and both sides can send messages whenever they like. Your client streams audio chunks up (typically 20-100ms at a time); the server streams transcription back down, usually as a series of increasingly-confident guesses.

The guess-and-refine pattern matters. You'll have seen it if you've used live dictation: words appear, then change, then settle. That's the model emitting partial hypotheses as audio arrives, then committing to final ones once it has enough right-context to be sure. The technical name for this is streaming decoding with a sliding window, and the trade-off is exactly what it sounds like, smaller windows give lower latency but worse accuracy, because the model has less context to reason over.

Word-level timestamps and why they're not free

xAI's STT returns timestamps per word. This sounds like a small feature and is, in fact, a small miracle.

The model processes audio in fixed-size frames, say, every 20 milliseconds. Words don't line up with frames; they start and end whenever they want. To produce word-level timestamps, the system has to align the text it emitted back onto the audio it heard, working out which frames correspond to which word. The standard technique is called forced alignment and involves running a separate pass (or a shared attention mechanism) that matches each output token to the audio region that produced it.

Why you'd care: timestamps are what let you build a clickable transcript, generate subtitles, cut video on word boundaries, or search a recording for a specific phrase and jump to it. None of that is possible with a plain text blob.

Diarization, who's speaking

Speaker diarization is the answer to "who said what." The model, or, more commonly, a separate model that runs alongside, listens for changes in voice characteristics and labels segments by speaker. It doesn't know who the speakers are (it won't tell you it's Alice and Bob), but it will tell you "Speaker 1 said this, Speaker 2 said that, Speaker 1 said this other thing."

The mechanism: every short slice of speech is turned into a speaker embedding, a vector that captures voice identity the way face embeddings capture faces. The diarization system clusters those embeddings. Slices whose embeddings are close together get the same speaker label.

Where this breaks: overlapping speech (two people talking at once) is genuinely hard and most systems handle it poorly. Very short utterances ("yeah", "mhm") often get misattributed because there isn't enough audio to produce a stable embedding. And if someone coughs, laughs, or changes their tone dramatically, the system can spuriously introduce a new "speaker."

The twelve audio formats thing

The long format list, WAV, MP3, FLAC, Opus, and so on, is not about accuracy. Internally the server decodes all of them to the same raw sample stream before doing anything interesting. The list exists because clients produce audio in whatever format their hardware or library prefers, and making developers transcode before uploading is a friction tax nobody wants to pay. It's a thoughtful piece of API design dressed up as a feature bullet.

Text-to-speech, briefly

TTS runs the same machinery in reverse, roughly. A text string goes in; a neural model, typically a transformer that emits audio tokens, followed by a vocoder that turns those tokens into waveform samples, produces speech out. "Five expressive voices" means the model was trained with voice-identity conditioning: a small ID vector tells it which voice to use, and the same text comes out in different timbres depending on which ID you pass.

The pricing, $4.20 per million characters, is worth understanding. A million characters is roughly 150,000-200,000 words, or around 15-20 hours of speech. So roughly $0.25 per hour of generated audio, which is cheap enough that you can stop thinking about it for most applications and expensive enough that if you're generating audiobooks at scale you'll still notice.

What to watch

Two things.

First, latency floors. The interesting frontier in streaming STT isn't accuracy, accuracy is largely solved for clean audio in major languages. It's how quickly the first word appears after it's spoken. Sub-300ms is the threshold where conversation feels natural; most providers are fighting for that number now.

Second, the convergence of STT and LLM. Increasingly, speech models are being folded directly into language models, the model hears audio tokens and emits text tokens without a separate transcription step. When that fully arrives, the "STT API" as a category starts to dissolve into "the model also has ears." xAI's launch is still in the older, cleanly-separated architecture. That won't be true forever.

Footnotes and links

Further reading

• [Paper]: Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper), the clearest public description of a modern encoder-decoder STT system.
• [Docs]: xAI Speech API reference, the endpoint shapes and parameter lists.
• [Explainer]: Jurafsky & Martin, Speech and Language Processing, chapters on ASR and TTS, the textbook, freely available online, and still the best structural overview.

¶