Appearance
12.13.4 — Machine Translation
Most of Cartwise's customers would rather shop in a language other than English: Hindi, Bengali, Marathi, Tamil, Telugu, Gujarati, Kannada, Malayalam, Odia, Punjabi or Assamese. Translating 400,000 product titles and descriptions, every policy page, every email template and every support reply into eleven languages by hand would take a team of translators years, and the catalogue changes every day. So translation is done by machine, with people checking where it matters.
Translation is also the oldest problem in artificial intelligence, older than the term itself, and its history is the history of the whole field in miniature: hand-written rules, then statistics, then neural networks, then language models. Each stage fixed the previous one's characteristic failure and introduced a new one. This page tells that story, because knowing how each method fails is what lets you use today's tools safely, and then builds Cartwise's translation pipeline, including how to measure whether a translation is any good.
1. Why translation is hard
Translating word by word fails immediately, for reasons any bilingual person knows.
Word order differs. English puts the verb in the middle: "The courier delivered the parcel." Hindi puts it at the end: "कूरियर ने पार्सल पहुँचाया", literally "courier parcel delivered". Linguists call these subject-verb-object and subject-object-verb orders. Most Indian languages are the second kind.
Words do not map one to one. English "you" is one word; Hindi chooses between आप (aap, respectful), तुम (tum, familiar) and तू (tu, very intimate or rude), and a shop that addresses customers with तुम sounds disrespectful. Hindi verbs and adjectives change with the gender of the noun; English mostly does not. Tamil builds long words by attaching pieces to a root, so one Tamil word can need a whole English phrase.
Meaning depends on context. "The order is cancelled" and "the order of the items" use "order" in different senses, and the translation must choose. "Returns" may be a noun (the returns page) or a verb (it returns tomorrow).
Some words should not be translated at all. Brand names, product model numbers, and terms customers already know in English, such as "cash on delivery" or just "COD", are better left alone. Another choice is to write them in the local script without translating them. That is called transliteration: writing the sound of a word in another alphabet, such as कैश ऑन डिलीवरी.
Each generation of translation technology handled these problems differently.
2. Rules: 1950s to 1980s
In 1949, the mathematician Warren Weaver wrote a memorandum suggesting that translation might be treated like code-breaking: a Russian text could be viewed as English "written in a strange code". On 7 January 1954, a demonstration by Georgetown University and IBM translated about sixty Russian sentences into English, using a vocabulary of 250 words and six grammar rules. The press reported that fully automatic translation was a few years away.
It was not. Translation systems of the following decades were built from hand-written rules: bilingual dictionaries plus rules for reordering and inflecting words, written by linguists, language pair by language pair. They worked on the sentences their rules anticipated and broke on everything else, and every new rule interacted with the old ones. In 1966, a United States government committee, the ALPAC report, concluded that machine translation was slower, less accurate and more expensive than human translation, and funding largely stopped. Rule-based systems survived in narrow uses. SYSTRAN, founded in 1968, translated documents for the European Commission and, from 1997, powered one of the first free web translation services.
The failure of rules is the lesson that runs through this whole chapter: language has too many exceptions to write down.
3. Statistics: learning from translated documents, 1990s to 2016
At IBM in the late 1980s, a group led by Peter Brown and Robert Mercer tried something different: instead of writing rules, learn translation from examples. Governments publish large amounts of professionally translated text. The Canadian parliament publishes its debates in both English and French, and the European Union and United Nations publish documents in many languages. Such collections of sentences with their translations are called parallel corpora.
Their 1993 paper, The Mathematics of Statistical Machine Translation, framed translation with Bayes' rule from 12.2.5. To translate a foreign sentence f into English, find the English sentence e that is most probable given f:
\hat e = \arg\max_e P(e \mid f) = \arg\max_e P(e)\, P(f \mid e)
Read \arg\max_e as "the e that makes this largest". The second form, from Bayes' rule, splits the problem in two. P(e) is a language model: how likely is this to be a fluent English sentence? It can be learned from any English text, of which there is plenty. P(f \mid e) is a translation model: how likely is it that this English sentence would be translated into this foreign one? It is learned from parallel corpora, by working out which words in one language tend to line up with which words in the other, called word alignment. The split is powerful because each half can be imperfect: the translation model proposes word choices, and the language model rejects candidates that are not fluent English.
Later systems translated phrases rather than single words, which captured idioms and local word order: phrase-based translation, developed by Philipp Koehn, Franz Och and Daniel Marcu in 2003. When Google launched Google Translate in April 2006, it was a phrase-based statistical system, led by Och and trained on the UN and EU documents among much else. For a decade it was how most of the world's machine translation was done.
Statistical translation's characteristic failure was broken structure. It translated short chunks well and stitched them together poorly, so long sentences came out in the wrong order, with verbs missing or doubled, especially between languages with very different word order such as English and Hindi. The output was obviously machine-made, which was at least a warning.
4. Neural networks: 2014 onwards
In 2014, two papers changed the field. Ilya Sutskever, Oriol Vinyals and Quoc Le at Google showed that one neural network, an encoder-decoder (sequence-to-sequence) model, could read a whole sentence into a vector and write its translation from that vector (12.5.6). And Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio added attention, letting the decoder look back at every word of the source sentence while writing each word of the translation, instead of relying on one compressed vector. Attention, the mechanism behind every model in Chapter 12.7, was invented for translation. So was the Transformer: the 2017 paper Attention Is All You Need introduced it as a translation model, measured on English-German and English-French.
In September 2016, Google switched Google Translate to a neural system and reported that it reduced translation errors by about 60% compared with its phrase-based system on major language pairs. A further discovery followed: one neural model trained on many language pairs at once could translate between pairs it had never seen together, such as Japanese to Korean after training only on Japanese-English and Korean-English. The model had built a shared internal representation of meaning, an idea Google called zero-shot translation. Large multilingual models since then, such as Meta's No Language Left Behind (2022, 200 languages) and, for Indian languages, AI4Bharat's open IndicTrans2 (2023, covering all 22 languages listed in India's constitution), extend the same approach.
Neural translation reversed the failure mode of statistics. Its output is fluent: grammatical, natural, readable. And that fluency hides its errors. A neural system may quietly omit a clause, add a plausible detail that was not in the source, or mistranslate with complete confidence, and because the result reads well, nobody notices. "Do not use with a fast charger" rendered fluently as "use with a fast charger" is the most dangerous kind of error, because the dropped word reverses the meaning and nothing looks wrong.
5. Language models as translators
Large language models translate well, especially between widely used languages, and add abilities dedicated translation models lack. They can follow instructions about tone, such as "be formal, and address the customer as आप". They can keep a glossary consistent, such as "always translate return window as वापसी अवधि". They can use context beyond the one sentence, such as the product category or the rest of the page. And they can explain their choices. For a support reply, where tone and context matter and volumes are moderate, a language model is usually the best translator available.
Their weaknesses are the neural ones, amplified. They can hallucinate content, the invented details of 12.7.9. Quality drops for languages less represented in their training data, and Odia or Assamese get noticeably worse results than Hindi. And they cost more per word than a dedicated translation model, which matters at catalogue scale.
So the choice is the usual one of 12.11.1: a dedicated translation model for high-volume, uniform text, a language model where instructions, context and tone matter, measured on each language separately.
6. Measuring translation quality
A translation can be right in many ways, so measuring it automatically is hard, and the three generations of metrics mirror the three generations of translation.
BLEU: counting shared word sequences
BLEU, proposed by Kishore Papineni and colleagues at IBM in 2002, compares a machine translation with a human reference translation by counting shared sequences of words, called n-grams: single words (1-grams), pairs (2-grams), and so on up to 4-grams. For each length, it computes what fraction of the candidate's n-grams also appear in the reference, takes the geometric mean of the four fractions, and multiplies by a brevity penalty that punishes translations shorter than the reference, which could otherwise score well by leaving words out.
Work through one. The reference is "the parcel was delivered to the wrong address" and the machine wrote "the parcel was sent to the wrong address". Of the candidate's 8 single words, 7 appear in the reference (not "sent"): 7/8. Of its 7 word pairs, 5 appear ("the parcel", "parcel was", "to the", "the wrong", "wrong address"): 5/7. Of its 6 three-word sequences, 3 appear: 3/6. Of its 5 four-word sequences, only "to the wrong address" appears: 1/5. Both sentences have 8 words, so the brevity penalty is 1. Then
\text{BLEU} = \left(\tfrac{7}{8} \times \tfrac{5}{7} \times \tfrac{3}{6} \times \tfrac{1}{5}\right)^{1/4} = (0.0625)^{1/4} = 0.5
BLEU's weakness shows at once. "The package went to an incorrect address", a perfectly good translation of the same meaning, shares almost no n-grams with the reference and would score near zero. BLEU rewards matching the reference's words, not its meaning. It is still widely reported, because it is cheap and comparable over decades, but it correlates only loosely with human judgement of quality.
Better metrics
chrF (Maja Popović, 2015) compares sequences of characters instead of words, which works better for languages like Hindi and Tamil, where one word appears in many inflected forms and word-level matching misses near-matches. COMET (Ricardo Rei and colleagues at Unbabel, 2020) is a neural network trained to predict human quality ratings from the source, the translation and the reference; it recognises paraphrases and agrees with people far more closely than BLEU. Related quality estimation models predict quality without a reference at all, which is what a production system needs, since new product descriptions have no human translation to compare against.
People, for what matters
Human evaluation remains the standard for important content. The usual method is MQM (Multidimensional Quality Metrics). Trained reviewers mark every error with two labels. The first is its category: accuracy (such as something left out or translated wrongly), fluency, terminology or style. The second is how serious it is: minor, major or critical. Then the scores are added up. A dropped "not" in a safety instruction is a critical accuracy error, however fluent the sentence.
7. Cartwise's translation pipeline
Putting it together, Cartwise translates its catalogue and policies through a pipeline designed around the failure modes above:
① Protect. Brand names, model numbers, sizes and placeholders such as {order_id} are replaced by tags the translator is told to leave alone, and restored afterwards. This prevents "Apple" becoming the fruit and {order_id} becoming a translated word that breaks the email template.
② Translate with the right engine. The catalogue, hundreds of thousands of short, uniform texts that change daily, goes to a dedicated translation model, hosted as in 12.11.4, at a fraction of a language model's cost. Policy pages and support replies, where tone, honorifics and consistent terms matter, go to a language model with the glossary, the instruction to address customers with आप, and the surrounding page as context.
③ Check the certain things in code. Every tag restored; every number, amount and unit identical to the source; the length within what the screen can show; and a check that negations survived. If the source says "do not" or "never", the translation must contain the equivalent word, from a list per language. These checks are simple and catch the most dangerous errors with certainty, the same principle as the output checks of 12.10.8.
④ Score the rest. A reference-free quality-estimation model scores each translation. Low scores go to human translators. Every safety instruction, legal text and returns policy goes to a person regardless of its score, because the cost of one wrong word there is far higher than the cost of review.
⑤ Publish and cache. Translations are stored keyed on the source text and the glossary version (12.12.2), so a product whose description has not changed is never translated twice, and a glossary change retranslates exactly the texts that used the changed term. Translation happens when content is written, not when a customer views it, so pages load at full speed in every language.
Two choices came from watching how customers actually read. Many Hindi speakers search and shop in romanised Hindi ("earphone ka case", "wapas kaise karein"), so search indexes include transliterated and romanised forms alongside the Devanagari, connecting translation to the hybrid search of 12.6.4. And quality is measured per language, by human MQM review of a monthly sample, because a pipeline that is excellent in Hindi can be poor in Odia, and an average across languages would hide it.
What the interviewer will push on
"How did machine translation work before neural networks?" Rules first (dictionaries and grammar rules, from the 1954 Georgetown-IBM demonstration to SYSTRAN), which broke on everything unanticipated. Then statistics: learn from parallel corpora using \hat e = \arg\max_e P(e)\,P(f \mid e), a language model for fluency times a translation model from word alignments (IBM, 1993), later phrase-based (Koehn, Och and Marcu, 2003), which powered Google Translate from 2006.
"What changed with neural translation?" Encoder-decoder models with attention (2014), then the Transformer (2017, a translation paper); Google's 2016 switch cut errors by about 60%; multilingual models translate zero-shot between unseen pairs. The failure mode flipped from broken structure to fluent but wrong: omissions and additions that read naturally.
"How do you evaluate translation?" BLEU (n-gram precision up to 4-grams, geometric mean, brevity penalty) is cheap but rewards matching words, not meaning; chrF for morphologically rich languages; COMET and reference-free quality estimation agree better with people; MQM human review for what matters. Being able to compute BLEU for a short example is a good signal.
"Would you use an LLM or a dedicated model to translate a product catalogue?" A dedicated model for high-volume, uniform text; an LLM where tone, glossary and context matter; measured per language. Protect untranslatable items, check numbers and negations in code, and send safety-critical text to people.
Something to volunteer: the most dangerous translation errors are fluent ones. A dropped "not" produces a perfectly natural sentence with the opposite meaning, which is why a simple code check that negations survived is worth more than a better model.
Recall
- Translation is hard: word order (English SVO, Hindi SOV), words without one-to-one equivalents (आप/तुम, gender), context, and words that should stay untranslated or be transliterated.
- Rules (1954 Georgetown-IBM; ALPAC report 1966; SYSTRAN) broke on the unanticipated.
- Statistics (IBM, 1993): \hat e = \arg\max_e P(e)\,P(f \mid e), language model times translation model learned from parallel corpora by word alignment; phrase-based (2003); Google Translate from 2006. Failure: broken structure.
- Neural (2014: encoder-decoder, attention; 2017: the Transformer, built for translation; Google 2016, about 60% fewer errors; zero-shot multilingual; NLLB, IndicTrans2). Failure: fluent but wrong (omissions, additions, reversed meaning).
- LLMs add instructions, glossaries, tone and context; weaker on low-resource languages, can hallucinate, cost more.
- Metrics: BLEU (n-gram overlap, brevity penalty; rewards words, not meaning), chrF (characters), COMET and reference-free quality estimation, MQM human review.
- Pipeline: protect untranslatables, choose the engine by content, check tags, numbers, length and negations in code, score and send low scores and all safety text to people, cache by source and glossary version, translate at write time, measure per language.
Self-test: Compute the 1-gram and 2-gram precisions for "the order was cancelled yesterday" against the reference "the order was cancelled today". Why does the statistical formula split into P(e) and P(f \mid e), and which half needs parallel text? Which pipeline step would catch "do not use with a fast charger" translated without its negation?
Next: 12.14.1 Prompt Injection — every system in this Part reads text written by someone else, and Chapter 12.14 starts with the security problem that creates: instructions hidden in that text, aimed at the model.