12.7 — Speech, Voice and Translation

A voice assistant handles "play the news" perfectly and fails on a Glaswegian speaker saying their own surname. The microphone is fine, the network is fine, and the model is doing exactly what it was trained to do — which was mostly on a different accent.

Speech is where machine learning meets physics, human variation and real-time constraints at once, and it is worth understanding because the pipeline is visible in a way that a text model's is not.

1. From sound to numbers

Sound is pressure varying over time. A microphone turns it into voltage, and an analogue-to-digital converter samples it (Chapter 1.4).

16 kHz is the standard for speech. By the sampling theorem you can represent frequencies up to half the sample rate, and 8 kHz covers nearly everything that distinguishes one speech sound from another. Music uses 44.1 kHz because instruments carry meaningful energy far higher; telephones used 8 kHz, which is exactly why "s" and "f" are hard to tell apart on a phone call — the difference between them lives above 4 kHz.

Raw samples are a poor input. 16,000 numbers per second, and the information is in the frequencies present, not the individual values.

So the signal is converted to a spectrogram: cut the audio into overlapping windows of about 25 ms, take the Fourier transform of each (Chapter 1.4, and Volume III Part 4 develops it), and stack the results. The output is an image — time on one axis, frequency on the other, energy as brightness. Speech recognition is, mechanically, image recognition over that picture.

The frequency axis is then warped to the mel scale, which is roughly logarithmic, because human hearing is: the difference between 200 and 300 Hz is very audible, and between 8,000 and 8,100 Hz is not. A mel spectrogram spends its resolution where the information is, and it is the standard input to essentially every speech model.

MFCCs — a further compression of the mel spectrogram into about 13 numbers per frame — dominated before deep learning. Neural networks now take the mel spectrogram directly, because they can use the detail that MFCCs discarded.

2. Phonetics, and why accents break systems

A phoneme is a sound that distinguishes meaning in a language. English has roughly 44. "Pat" and "bat" differ by one.

Phonemes are categories, not fixed sounds. The p in "pin" is aspirated (a puff of air) and the p in "spin" is not — different sounds, one phoneme, and English speakers do not hear the difference. In some other languages that same difference distinguishes words, which is why a distinction that is obvious in one language is inaudible to speakers of another.

Coarticulation is the real difficulty. Sounds are not produced in sequence like beads; the mouth is already moving toward the next one. The k in "key" and in "caw" are physically different because the following vowel pulls the tongue to a different place. There is no clean boundary between one sound and the next, which is why segmenting speech into units is hard and why end-to-end models that never segment work better.

Accents and dialects vary in vowels, consonants, rhythm and vocabulary, and the practical consequence is measurable: recognition error rates are substantially higher for speakers whose accent, dialect or first language is under-represented in the training data, and studies have repeatedly found large gaps between demographic groups on commercial systems.

This is a data problem with a fairness consequence, not an inevitability. Diverse training data narrows it, and it is worth measuring per group rather than reporting one average error rate — a system with 5% average error can be at 4% for one population and 20% for another (Chapter 12.10).

3. How recognition works

The classical pipeline (1980s–2010s). An acoustic model gave the probability of each phoneme from the audio, a pronunciation dictionary mapped words to phoneme sequences, and a language model gave the probability of word sequences. A hidden Markov model handled the alignment between audio frames and phonemes, and a search found the most likely word sequence overall. Three separately trained components, each with its own failure modes and its own expertise.

The end-to-end shift replaced the whole stack with one model from audio to text.

CTC is the idea that made it possible. The difficulty is alignment: 300 audio frames should produce 12 characters, and nobody labelled which frame belongs to which character. CTC's answer is to let the model output a character or a special blank symbol at every frame, then collapse repeats and remove blanks:

frames:  _ _ h h _ e e l l _ l o o _
collapse repeats and drop blanks  →  "hello"

The blank is what allows a genuine double letter: l l _ l collapses to ll while l l l collapses to l. CTC sums the probability over every alignment that produces the target text, so the model learns alignment as a by-product of learning the transcription. No frame-level labels are needed, which is what made large-scale training possible.

Its limitation is that CTC assumes outputs are conditionally independent given the audio, so it has no internal language model — which is why CTC systems are usually combined with an external one at decoding time.

Attention-based encoder-decoder models treat it as translation from audio to text, with a decoder that does model the output sequence. Better accuracy, and it needs the whole utterance, which makes streaming awkward.

RNN-Transducer combines both and is what most production streaming recognisers use: it emits output incrementally like CTC while modelling output dependencies like an attention decoder.

The current generation — Whisper and its successors — are transformer encoder-decoders trained on very large amounts of weakly labelled audio scraped from the web, covering many languages and doing translation and language identification in the same model. The lesson matches Chapter 12.1's: scale plus a general objective beat carefully engineered components.

4. What a real recogniser has to do besides recognise

Voice activity detection and endpointing. Deciding when speech starts, and — harder — when the speaker has finished rather than paused to think. Endpointing is the main determinant of how responsive a voice assistant feels, and cutting people off is the most common complaint.

Streaming versus batch. Streaming emits words as they arrive, which requires committing to a word before hearing what follows, so early words sometimes get revised. Batch sees everything and is more accurate. Streaming trades accuracy for latency, and there is no way around that trade.

Punctuation and casing are usually a separate model applied to the raw transcript.

Inverse text normalisation turns "twenty five pounds fifty" into "£25.50" — nearly always rule-based, and nearly always the part that produces embarrassing errors.

Diarisation — who spoke when — is a separate clustering problem over speaker embeddings, and it degrades badly on overlapping speech.

Contextual biasing. Boosting the probability of a known vocabulary — a user's contacts, product names, medical terms — for the current request. This is the single most effective accuracy improvement available in a production system, because domain names and proper nouns are where general models fail.

Word error rate is the standard metric: insertions plus deletions plus substitutions, divided by the number of reference words. Read it critically. It weights every word equally, so misrecognising "not" and misrecognising "the" score the same while meaning entirely different things. Measure what matters to your application — entity accuracy, intent accuracy, or task success — alongside it.

5. Speech synthesis

Concatenative (1990s–2000s) — record a voice actor for many hours, cut it into units, and reassemble. Intelligible, and the joins are audible, which is the characteristic sound of older navigation systems.

Parametric — model the vocal tract and generate the waveform. Smooth, flexible, and noticeably synthetic.

Neural, in two stages — the architecture that changed the field:

1. Text to mel spectrogram — a model predicts what the audio should look like.
2. Vocoder to waveform — a second model turns the spectrogram into samples.

WaveNet (2016) was the breakthrough, generating raw audio sample by sample with quality close to human. It was far too slow for real time, and the work that followed — parallel and adversarial vocoders such as HiFi-GAN — made it fast enough to run in production.

Prosody is the hard part, and it is what "sounds human" actually means. Three components:

• Duration — how long each sound lasts, including pauses.
• Pitch (fundamental frequency) — the melody, which carries questions, emphasis and emotion.
• Energy — loudness variation.

A sentence's correct prosody depends on meaning, not spelling. "I never said she stole the money" has seven different meanings depending on which word is stressed, and nothing in the text says which. That is why prosody is the residual difficulty: the model must infer intent from context to produce the right melody.

How modern systems get it right: they are trained on large expressive datasets rather than flat read speech; they use a reference or style encoder that takes a sample of the desired delivery and conditions generation on it; some predict pitch and duration explicitly, which also makes them controllable; and the largest systems generate audio tokens directly from a language model, so the same context that determines the words also determines the delivery.

Zero-shot voice cloning — matching a voice from a few seconds of audio — now works well, and it is a genuine harms problem: impersonation, fraud and non-consensual synthetic speech. Consent, provenance marking and detection are engineering obligations here, not policy afterthoughts (Chapter 12.10).

6. SSML

SSML is XML for controlling synthesis:

<speak>
  Your order <say-as interpret-as="cardinal">1042</say-as> ships
  <emphasis level="strong">tomorrow</emphasis>.
  <break time="400ms"/>
  <prosody rate="slow" pitch="-2st">Thank you for waiting.</prosody>
  <sub alias="Doctor">Dr</sub> Ruiz will call you.
</speak>

• <say-as> — read as a number, date, time, telephone number or spelled-out characters. The most useful tag by far, because 1042 as "one thousand and forty-two" versus "one zero four two" is a real correctness question.
• <break> — an explicit pause.
• <emphasis> and <prosody> — stress, rate, pitch, volume.
• <phoneme> — an explicit pronunciation, for names and technical terms.
• <sub> — say something other than what is written.

The caveat that matters in practice: neural voices often interpret rather than obey. A modern model predicts prosody from context, so a <prosody rate="fast"> may be applied approximately or overridden. Support varies by voice and provider, <break> and <say-as> are the most reliably honoured, and you must test with the specific voice rather than assuming the tag worked.

7. Machine translation before and after

Rule-based (1950s–1980s). Grammars and dictionaries, written by linguists per language pair. Brittle, and the effort scaled with the square of the number of languages.

Statistical machine translation (1990s–2013). IBM's insight was to learn translation from parallel text — the same documents in two languages — rather than to encode rules. A translation model learned which phrases correspond, from word alignments discovered automatically; a language model scored fluency in the target language; and a decoder searched for the sequence maximising both.

Phrase-based SMT was the state of the art for two decades, and its characteristic failure was structural: it translated chunks well and reordered them poorly, which is why translations between languages with different word order — German's verb-final clauses, Japanese's subject-object-verb — were noticeably broken.

Neural machine translation (2014–2016) replaced the pipeline with one encoder-decoder network. The step change was real and rapid: fluency improved so much that the failure mode inverted. SMT produced obviously broken text; NMT produces fluent text that is occasionally confidently wrong — including omitting a clause entirely, which is much harder to notice.

Attention (Chapter 12.5.1) was invented here, to fix the fixed-size bottleneck in early sequence-to-sequence translation, before it was generalised into the transformer.

Today, large language models translate competitively and add what dedicated systems lacked: they can use surrounding context, follow instructions about tone and terminology, and keep a glossary consistent. Dedicated translation models remain cheaper, faster and often better for high-volume, low-context translation, so both are in use.

On measurement: BLEU compares n-gram overlap with a reference translation. It is convenient and weakly correlated with human judgement — it punishes a correct paraphrase and rewards a wrong translation that shares words. Learned metrics such as COMET, which score with a model trained on human ratings, correlate far better and are what current work uses.

8. Engineering notes

Latency is the product. In a voice interface the perceived delay is endpointing plus recognition plus your logic plus first audio out. Endpointing and time to first audio dominate, so stream both the recognition and the synthesis, and start speaking before the full response exists.

Handle recognition uncertainty explicitly. Most systems return alternatives and confidence. Confirm low-confidence values that matter — a name, an amount, an address — rather than acting on a guess.

Bias toward your vocabulary. Contacts, product names and identifiers, per request.

Fall back to text. Speech fails in noise, in accents and on unusual names. A voice-only path with no keyboard escape is an accessibility failure as well as a reliability one.

Measure by group. One average error rate hides a system that works for some users and not others, and that gap is the thing worth fixing.

Recall

• Speech uses 16 kHz because the distinguishing information sits below 8 kHz — telephony's 8 kHz is why "s" and "f" are confusable. Audio becomes a mel spectrogram (log-scaled like hearing), and recognition is image recognition over it.
• Phonemes are categories, not fixed sounds, and coarticulation means there is no clean boundary between them — which is why end-to-end models beat segmenting pipelines.
• Error rates are substantially worse for under-represented accents. It is a data problem with a fairness consequence — measure per group, not one average.
• CTC solved alignment by emitting a blank at every frame and collapsing repeats, summing over all alignments — so no frame-level labels are needed. It assumes output independence, hence external language models; RNN-T is the streaming production answer.
• Real recognisers also need endpointing (which decides how responsive it feels), punctuation, inverse text normalisation, diarisation, and contextual biasing — the most effective accuracy win available. Word error rate weights "not" and "the" equally, so measure task success too.
• Synthesis is text → mel spectrogram → vocoder → waveform; WaveNet made it human-quality and later vocoders made it fast. Prosody — duration, pitch, energy — is the hard part, because correct stress depends on meaning, not spelling.
• SSML's <say-as> and <break> are the reliable tags; neural voices interpret rather than obey the prosody tags, so test with the specific voice.
• Translation went rules → SMT (translation model + language model, poor at reordering) → NMT (fluent, and confidently wrong when wrong). Attention was invented for translation before the transformer. BLEU correlates weakly with human judgement; learned metrics such as COMET do far better.

Self-test: Why is 16 kHz the speech standard and 8 kHz a problem? · What does the blank symbol in CTC make possible? · Why is coarticulation an argument for end-to-end models? · Which SSML tags can you rely on, and why not the others? · What did NMT change about how translation fails? · What does a single average word error rate hide?