Appearance
12.13.2 — Speech Recognition and Synthesis
Cartwise's phone support line received about 40,000 calls a day, many from customers who would rather speak than type, often in Hindi, Tamil, Bengali or a mix of Hindi and English. The old system was a menu: "press 1 for orders, press 2 for returns". Customers hated it. The new plan was a voice assistant that listens, understands, looks up the order, and answers in a natural voice, in the customer's own language, handing over to a person whenever it should.
That needs two technologies. Speech recognition, also called ASR for automatic speech recognition, turns spoken audio into text. Speech synthesis, also called TTS for text to speech, turns text back into spoken audio. In between sits everything earlier chapters built: the language model, the tools, the agent. This page explains both technologies from the physics of sound upwards, including why accents and dialects make recognition hard, how synthetic voices learned to sound human, and the markup that controls them.
1. Sound becomes numbers
Sound is air pressure rising and falling very quickly. A microphone turns the pressure into a voltage that rises and falls the same way, and a converter measures that voltage many times a second, each measurement a sample. The number of samples per second is the sample rate.
How many samples are needed? A rule from signal processing, the sampling theorem associated with Harry Nyquist and Claude Shannon, says that to capture a sound accurately you must sample at least twice as often as the highest frequency in it. Frequency is how many times per second the pressure rises and falls, measured in hertz (Hz). Most of what distinguishes one speech sound from another lies below about 8,000 Hz, so speech is usually recorded at 16,000 samples per second (16 kHz), which captures everything up to 8 kHz.
Telephone networks historically used 8 kHz, which captures only up to 4 kHz. That is why "s" and "f" are hard to tell apart on a phone call: the hissing energy that distinguishes them lies mostly above 4 kHz, and the phone line cuts it off. It is also why Cartwise's voice assistant, which receives phone audio, needed a recognition model trained on phone-quality speech; a model trained only on clean 16 kHz recordings does noticeably worse on calls.
From samples to a picture: the spectrogram
One second of speech is 16,000 numbers, and the individual numbers mean little. What carries information is which frequencies are present and how they change. So the audio is converted into a spectrogram: cut the sound into short overlapping windows of about 25 milliseconds, find how much energy each window has at each frequency using the Fourier transform (a mathematical method that splits a signal into the frequencies it is made of), and stack the results side by side.
The result is effectively an image: time across, frequency up, brightness for energy. Speech recognition becomes, mechanically, the problem of reading that picture, which is why the vision ideas of 12.13.1 and the convolutional and transformer networks of earlier chapters apply to it.
One more adjustment makes the picture match human hearing. People hear differences between low frequencies much better than between high ones: 200 Hz and 300 Hz sound clearly different, while 8,000 Hz and 8,100 Hz sound almost the same. The mel scale, from experiments on human pitch perception in the 1930s, captures this with the formula
m = 2595 \log_{10}\!\left(1 + \frac{f}{700}\right)
which converts a frequency f in hertz into "mels" m. Going from 200 to 300 Hz moves about 119 mels; going from 8,000 to 8,100 Hz moves about 13. A mel spectrogram spaces its frequency rows evenly in mels, so it spends its detail on the low frequencies where speech information is densest. It is the standard input to speech models.
2. Phonetics: the sounds of a language
To understand why recognition is hard, it helps to know a little phonetics, the study of speech sounds.
A phoneme is the smallest unit of sound that changes the meaning of a word in a particular language. In English, "pin" and "bin" differ by one phoneme, /p/ against /b/. English has about 44 phonemes, depending on the accent. Linguists write them in the International Phonetic Alphabet (IPA), which gives one symbol to each distinct sound, so that /ʃ/ is the "sh" in "ship" in every language.
Phonemes are categories, and languages draw the categories in different places. Say "pin" and "spin" with your hand in front of your mouth: the "p" in "pin" has a puff of air, called aspiration, and the one in "spin" does not. English speakers hear these as the same sound, because in English the difference never changes a word's meaning. In Hindi it does. पल (pal, a moment) and फल (phal, fruit) differ only by that puff of air, as do कल (kal, yesterday or tomorrow) and खल (khal, a villain). Hindi also distinguishes a "t" with the tongue against the teeth from one with the tongue curled back against the roof of the mouth, a retroflex sound, which English does not. A recognition model must learn the categories of each language it handles.
A second difficulty is coarticulation: sounds are not produced one after another like beads on a string. The mouth is always moving towards the next sound, so each sound is shaped by its neighbours. The "k" in "key" and the "k" in "car" are physically different, because the tongue is already preparing for the vowel. There is no clean boundary between sounds in the audio, which is one reason early systems that tried to cut speech into separate sounds first struggled, and why modern models learn from whole stretches of audio at once.
Accents and dialects
A dialect is a variety of a language spoken by a particular group, often a region, with its own vocabulary, grammar and pronunciation. An accent is the pronunciation part alone. Accents differ in several ways that matter to a recognition model:
- Different phoneme categories. Many Indian English speakers merge the English /v/ and /w/ into one sound, so "vest" and "west" sound alike; many Scottish speakers keep sounds other accents have merged.
- Different realisations of the same phoneme. Indian English often uses retroflex "t" and "d" where British and American English use other forms; vowels shift between regions.
- Different rhythm and stress. English stresses some syllables strongly and shortens the rest; many Indian languages give syllables more even weight, and this rhythm carries over into English.
- Different words and mixing. Customers say "the parcel is not reached", "kindly do the needful", or switch languages mid-sentence: "mera order abhi tak deliver nahi hua, can you check?". Switching between languages within a conversation is called code-switching, and Hinglish, the mixture of Hindi and English, is one of the most common examples in the world.
A model trained mostly on one accent does worse on others, not because those accents are less clear but because the model has seen less of them. This was measured in a 2020 study by Allison Koenecke and colleagues at Stanford, which found that five major commercial recognition systems made about twice as many errors on speech from Black American speakers as from white American speakers. The remedy is data: training and evaluating on the accents, dialects and mixtures your users actually speak, and measuring error rates separately for each, the slice evaluation of 12.12.4.
3. How speech recognition works
The old pipeline
For about thirty years, recognition systems were a pipeline of separate parts: an acoustic model that estimated which phoneme each slice of audio was, a pronunciation dictionary that mapped words to phonemes, and a language model that scored which word sequences were likely. Hidden Markov models, statistical models of sequences whose underlying states (here, phonemes) cannot be observed directly, tied the parts together. From around 2010, neural networks replaced the older statistical acoustic models and cut error rates sharply, but the pipeline remained, and each part had to be built and tuned separately.
End to end: CTC
The breakthrough that allowed one network to go straight from audio to text was Connectionist Temporal Classification, or CTC, introduced by Alex Graves and colleagues in 2006. It solves the problem of alignment. A two-second clip has about 200 frames of spectrogram, and its transcript might be 20 characters, but nobody has marked which frames belong to which character. CTC lets the network output, at every frame, either a character or a special blank symbol meaning "nothing new here". To read the output, you merge repeated characters and then remove the blanks:
text
frames: _ _ h h _ e e _ l l _ l o o _
merge repeats, drop blanks → h e l l o → "hello"The blank is what allows genuine double letters. The two separate runs of l with a blank between them become two l's; without the blank, l l l would merge into one. During training, CTC adds up the probability of every frame-by-frame path that collapses to the correct transcript, so the network learns where each character falls without anyone marking it. Later designs, such as the RNN-Transducer used in many streaming systems and attention-based encoder-decoders, built on the same end-to-end idea.
Large, weakly supervised models: Whisper
In 2022, Alec Radford and colleagues at OpenAI released Whisper, a transformer encoder-decoder (the architecture of 12.7.4) trained on 680,000 hours of audio collected from the internet with its existing transcripts. Those transcripts were imperfect, which is called weak supervision, but the sheer volume and variety of speakers, accents, languages, background noise and recording quality made the model far more robust than earlier ones trained on smaller, cleaner datasets. Whisper also translates into English and detects the language spoken. Its encoder reads 30-second windows of mel spectrogram; its decoder writes text token by token, like a language model listening as it writes.
Whisper and its successors made good recognition available to anyone, and they also illustrate the lesson of section 2: their accuracy varies widely by language, roughly in line with how much audio of each language was in their training data.
Measuring recognition: word error rate
The standard measure is the word error rate (WER). Line up the system's output against a correct reference transcript and count three kinds of error: substitutions S (a wrong word in place of a right one), deletions D (a word missed) and insertions I (a word added). Then
\text{WER} = \frac{S + D + I}{N}
where N is the number of words in the reference. If the customer said "I want to return my earphones" (6 words) and the system heard "I want return my ear phones", there is one deletion ("to"), one substitution ("earphones" heard as "ear"), and one insertion ("phones"), so WER = 3/6 = 50\%, although a person would understand the output perfectly. In the other direction, mishearing "not" in "my order has not arrived" is one error out of six words, a WER of about 17%, and it reverses the meaning. WER weighs every word equally, so it must be read alongside checks that matter for the task: whether order numbers, amounts and negations were recognised correctly.
4. How speech synthesis works
Synthesis runs the other way, from text to sound, in three stages.
Text normalisation turns written text into the words to be spoken. "Your order #4471 for ₹2,499 arrives on 20/10" contains a symbol, a number that should be read digit by digit, a currency amount and a date in a format that means 20 October in India and nothing at all in the United States. Each must be expanded correctly: "order four four seven one, for two thousand four hundred and ninety-nine rupees, arrives on the twentieth of October". Getting this wrong is the commonest reason synthetic speech sounds foolish, and it is a correctness problem, not a style one.
The acoustic model turns the words into a mel spectrogram: what the speech should sound like. Google's Tacotron 2 of 2017 showed that a neural network could do this directly from text, learning pronunciation and rhythm from recordings.
The vocoder turns the spectrogram into an actual waveform, the 16,000 or more samples a second that a speaker can play. DeepMind's WaveNet of 2016 was the first neural vocoder to produce speech that listeners found close to human, but it generated one sample at a time and was very slow. Later vocoders such as HiFi-GAN (2020) produce audio far faster than real time.
Prosody: what makes a voice sound human
Getting every sound right is not enough to sound human. What makes speech sound natural is prosody: the melody of the voice (pitch going up and down), its timing (how long each sound lasts, where the pauses fall, how fast it goes) and its loudness (which words are stressed). Prosody carries meaning that the words alone do not. "I never said she took the money" means something different depending on which word is stressed: stress she and someone else took it; stress said and perhaps you implied it. A question rises at the end in most accents; a list has a characteristic rhythm; bad news is delivered more slowly.
Older synthesis systems applied prosody by rules, which is why they sounded flat and mechanical. Neural systems learn it from recordings, and the most recent ones produce varied, context-sensitive prosody in three ways:
- Learning from expressive speech. Training on large amounts of natural, varied speech, including conversation, audiobooks and customer calls, rather than one speaker reading sentences evenly, teaches the model the full range of human delivery.
- Reading the meaning. Models that use a large language model's understanding of the text, or that are themselves built like language models, can infer from context which word deserves stress and what tone fits, rather than guessing from the spelling.
- Generating audio as tokens. A recent family of systems compresses audio into discrete audio tokens using a neural audio codec, a network that encodes sound into a short sequence of numbers and decodes it back, and then generates those tokens with a transformer, exactly as a language model generates text. Microsoft's VALL-E (2023) showed that such a model could continue speaking in a voice it had heard for only three seconds, copying its pitch, pace and emotion. The same approach powers speech-to-speech models that listen and reply in audio directly, without converting to text in between, which lets them respond to how something was said, not only to the words.
The ability to copy a voice from seconds of audio raises a serious problem: it can be used to impersonate people, for scams that "call" a family member in a cloned voice. Responsible providers require consent from the speaker whose voice is copied, add inaudible watermarks to generated audio, and block voices of public figures. Cartwise's assistant uses only licensed voices recorded with the speakers' agreement.
5. SSML: telling the synthesiser how to speak
When the text alone does not say how something should be spoken, applications use SSML, the Speech Synthesis Markup Language, a standard published by the World Wide Web Consortium (W3C) in 2004 and supported, in varying degrees, by the major speech services. It works like HTML: the text is wrapped in tags that the synthesis engine reads as instructions instead of speaking them.
xml
<speak>
Your order <say-as interpret-as="characters">4471</say-as>
for <say-as interpret-as="currency">₹2,499</say-as>
<emphasis level="moderate">has been refunded</emphasis>.
<break time="400ms"/>
<prosody rate="90%">The money will reach your card in three to five days.</prosody>
Thank you for shopping with <phoneme alphabet="ipa" ph="ˈkɑːtwaɪz">Cartwise</phoneme>.
</speak>Each tag solves one problem. <say-as interpret-as="characters"> reads 4471 as "four four seven one" rather than "four thousand four hundred and seventy-one", which matters because the customer may need to type it. interpret-as="currency" asks for the amount to be read as money. <emphasis> stresses the important part. <break> inserts a pause, giving the customer a moment to take in the news. <prosody> adjusts rate, pitch or volume; here it slows the sentence with the detail the customer must remember. <phoneme> gives the exact pronunciation of a word the engine might get wrong, written in the IPA from section 2, so the brand name is said the same way on every call.
Two practical points. Support differs between engines and voices, and the newest neural voices often treat prosody tags as suggestions rather than commands, because their own prosody comes from the model rather than from rules, so every important tag must be checked by listening to the specific voice. And much of what SSML was invented for, especially text normalisation, can also be done before synthesis in your own code, which makes it testable and the same across engines; Cartwise expands order numbers, amounts and dates in code and uses SSML mainly for pauses, pronunciations and emphasis.
6. The voice assistant, end to end
Putting the pieces together, a voice conversation flows through a chain where every stage streams, because waiting for each stage to finish before starting the next would make the conversation painfully slow.
Voice activity detection recognises when the customer is speaking, as opposed to background noise. Streaming recognition transcribes as they speak, so that when they stop, the text is already almost complete. End-of-turn detection decides when they have actually finished, which is harder than it sounds: a pause may be the end of a sentence or a moment of thought, and cutting in too early is as annoying as waiting too long. The agent is the support assistant of earlier chapters, with its tools and guardrails, prompted to reply in short spoken sentences rather than written paragraphs with bullet points. Streaming synthesis begins speaking as soon as the first sentence is ready. Barge-in stops the assistant mid-sentence the moment the customer starts talking again.
The latency target is set by human conversation: people expect a reply within about a second of finishing, and gaps much longer than that feel broken. The budget is spent across the chain, which is why each stage streams. Cartwise also fills unavoidable waits naturally, with "Let me check that order for you" spoken while a tool call runs, the spoken equivalent of the progress messages of 12.8.5.
Two design rules came from the first month of calls. Read back what matters: before acting on an order number or amount heard over a phone line, the assistant repeats it ("That's order four four seven one?"), because a single misheard digit sends everything after it wrong. And hand over early when recognition confidence is low or the customer repeats themselves, because nothing frustrates a caller more than an assistant that keeps mishearing them.
What the interviewer will push on
"How does speech recognition work?" From audio sampled at 16 kHz to a mel spectrogram (25 ms windows, Fourier transform, mel-spaced frequencies), then a neural network reads it. End-to-end models learn alignment with CTC (blank symbol, merge repeats, sum over paths) or transducers, or use encoder-decoders like Whisper, trained on 680,000 hours of weakly supervised audio.
"Why do accents and dialects make recognition harder?" Languages and accents draw phoneme categories differently, realise them differently, have different rhythm and vocabulary, and mix languages. Models do worse on varieties under-represented in training. Measure WER by accent and language, and train on the speech your users actually speak.
"What is WER and what are its limits?" (substitutions + deletions + insertions) ÷ reference words. It weighs every word equally, so a harmless split of "earphones" counts as heavily as a missed "not". Check task-critical items separately.
"How do modern TTS voices sound so natural?" Neural acoustic models and vocoders, trained on expressive speech, with prosody (pitch, timing, loudness) learned from context rather than rules; recent systems generate audio tokens with transformers and can copy a voice from seconds of audio, which requires consent and safeguards.
"What is SSML?" A W3C markup standard: tags such as say-as, break, emphasis, prosody and phoneme tell the engine how to read numbers, where to pause, what to stress and how to pronounce words. Neural voices may treat prosody tags as hints; test with the actual voice, and do text normalisation in code where you can.
"Design a voice support assistant." Streaming at every stage (speech detection, streaming recognition, end-of-turn detection, a streaming agent, sentence-by-sentence synthesis), barge-in, a latency target of about a second, reading back critical details, and early hand-over when recognition is poor.
Something to volunteer: telephone audio cuts off everything above 4 kHz, which removes much of what distinguishes "s" from "f". A recognition model for a phone line must be evaluated on phone-quality audio, not on clean recordings.
Recall
- Speech is sampled at 16 kHz to capture frequencies up to 8 kHz (sampling theorem); telephone audio stops at 4 kHz. A spectrogram shows energy by time and frequency from 25 ms windows; the mel scale, m = 2595\log_{10}(1 + f/700), matches human hearing.
- A phoneme is the smallest sound that changes meaning in a language; languages draw categories differently (Hindi पल/फल differ only by aspiration). Coarticulation blurs boundaries. Accents differ in categories, realisations, rhythm and vocabulary; code-switching (Hinglish) mixes languages. Models do worse on under-represented varieties; measure by slice.
- Recognition: from HMM pipelines to end-to-end networks. CTC (Graves, 2006) outputs characters or blanks per frame, merges repeats, drops blanks, and sums over alignments. Whisper (OpenAI, 2022): encoder-decoder on 680,000 hours of weakly supervised audio.
- WER = (S + D + I) / N; it weighs all words equally, so check numbers and negations separately.
- Synthesis: text normalisation, acoustic model to mel spectrogram (Tacotron 2), vocoder to waveform (WaveNet, HiFi-GAN). Prosody (pitch, timing, loudness) is learned from expressive data and context; audio-token models can copy voices from seconds of audio, so consent and watermarks matter.
- SSML (W3C, 2004):
say-as,break,emphasis,prosody,phoneme; neural voices may treat tags as hints. - Voice assistants stream every stage, support barge-in, aim for about one second, read back critical details, and hand over early.
Self-test: What sample rate captures frequencies up to 11 kHz? Compute the WER if "please cancel order 4471" is heard as "please cancel the order 4417". Which SSML tag would you use to make a synthesiser read an OTP digit by digit?
Next: 12.13.3 Image Generation and Diffusion — from understanding images and speech to creating them: how diffusion models turn random noise into a picture that matches a sentence, and what a retailer should and should not use them for.