Our group's favourite DL architecture · XLM-RoBERTa
The Review
Room
How one insights team learned to hear customers in a hundred languages, and what we think is still wrong with the model that helped them.
The team and company are fictional. Model facts and numbers: Conneau et al., "Unsupervised Cross-lingual Representation Learning at Scale", ACL 2020 (arXiv:1911.02116)
Chapter 1 · The pile that never shrinks
Reviews arrive in every language. Insights only in one.
Huda's team sells across many markets. Every morning the review queue fills with Arabic, Hindi, Swahili, French, Urdu. Their classifier, trained on English labels, can only sort the English ones. Everything else waits.
Arriving 0
Labelled 0
Waiting · no model 0
Chapter 2 · Three options on the whiteboard
Which would you fund?
Ravi lays out three plans. All start from the same asset: labelled reviews in English only. Mariam asks which one actually works.
Translate every review into English
- Machine-translate each incoming review
- Run one English-only model (RoBERTa)
- Translation cost on every review, forever
One multilingual model, English labels only
- Fine-tune XLM-R on English examples
- Apply directly to every other language
- No translation at all
Translate the labels once, train one model
- Machine-translate the English training set
- Fine-tune one XLM-R on all languages together
- One-time translation cost
Table 1. XNLI is a sentence-understanding test with human-made test sets in 15 languages.
Chapter 3 · Meet the apprentice
Same engine as BERT. Different diet.
Huda expects something exotic. Ravi shows her the spec sheet: the engine is the standard Transformer encoder. The paper's own words are that it follows XLM as closely as possible, changing only what improves performance at scale.
WHAT CHANGED vs XLM
No translated sentence pairs. No language tags. A bigger vocabulary. Web data instead of Wikipedia. Much longer training.
§3, Table 7. Base version: 12 layers, 768 hidden, 12 heads, 270M parameters.
Where do the parameters go?
Computed with the approximation the paper itself gives in Appendix B. Click a model.
Chapter 4 · The library
Feed it the web, then ration the English.
Before XLM-R, multilingual models studied Wikipedia. For Swahili or Urdu that is a thin pamphlet. The authors built a new library, CC-100: 2.5 TB of cleaned CommonCrawl text in 100 languages, with twelve monthly web snapshots for every language except English.
How much text each language got
CC-100 size in GiB, log scale. Orange: low-resource.
Appendix A, Table 6
Trained on Wikipedia, Swahili and Urdu performed about the same as a randomly initialised model. Switching to CommonCrawl improved them by up to 10 points. The authors estimate a few hundred MiB of text is the minimum to learn a language.
§3, §5.4
The rationing dial: α
Grey: each language's share of the raw data. Orange: its share of training batches after smoothing. Shown for 8 languages, using their token counts.
§5.1, Figure 5: higher α helps high-resource languages and hurts low-resource ones. 0.3 was optimal overall.
Chapter 5 · Breaking reviews into bricks
One brick box for all 100 languages.
Ravi's old pipeline had a different tokenizer per language. XLM-R uses a single SentencePiece model trained on raw text, with a shared vocabulary of 250,000 sub-word pieces. A review becomes a row of bricks.
Illustrative split. Real pieces depend on the trained SentencePiece model.
NO LANGUAGE TAGS
Earlier XLM told the model which language each sentence was in. XLM-R does not, which the authors say lets it handle code-switching, like Gulf customers mixing English and Arabic.
§3
ONE TOKENIZER, NO LOSS
Replacing language-specific preprocessing with SentencePiece on raw text showed no loss in performance, and made the model far easier to use on messy real text.
§5.1, Figure 7
BIGGER BOX HELPS
At a fixed model size, growing the vocabulary from 32K to 256K added +2.8% XNLI accuracy. From 128K to 512K added more than 3%.
§5.1, Figure 6
Chapter 6 · The meeting room
Every word asks every other word: who matters to me?
Inside, each brick enters a meeting. Click a word to see who it listens to. This is self-attention, the core of the Transformer the paper builds on.
Illustrative attention weights for one review. The pattern is invented for teaching; the mechanism is real.
A word's query is matched against every key. Strong matches pass along their values. 16 heads run this in parallel, each learning a different pattern, then a feed-forward step (width 4,096) digests the result.
Table 7: XLM-R Large · L=24, H=1024, Hff=4096, A=16
16 heads × 24 floors
Floor 1 at the bottom → floor 24 at the top
Chapter 7 · How it learned
One game, played 1.5 million times.
No teacher. No translations. The librarian hides words in a sentence and the apprentice guesses them. This is masked language modelling, and it is the only training objective XLM-R uses.
Illustrative guesses and probabilities.
SCALE
1.5M updates on 500 32GB V100 GPUs, batch size 8,192.
§5.2
DROPPED TRANSLATIONS
A well-tuned fill-in-the-blank model went from 71.3% to over 75% on XNLI, on par with the version that needed translated pairs. So they dropped them.
§5.1
DON'T STOP EARLY
Earlier models stopped when validation perplexity plateaued. The authors found downstream accuracy kept improving after the plateau; XLM had been under-tuned.
§5.1
Chapter 8 · The curse of multilinguality
More languages help, until they don't.
Mariam asks the obvious business question: why not add every language? The paper's answer is its most important finding. With a fixed-size model, each new language gets a smaller share of the shelf.
§5.1, Figures 2 and 4. Wikipedia-trained models; ablations use a 150K vocabulary.
TWO FORCES
Positive transfer: similar languages teach each other, so low-resource languages improve as you add a few more.
Capacity dilution: the model's size is fixed, so every language gets less room. Past a point, all languages get worse.
The authors: the curse can be alleviated by simply increasing model capacity, but it remains an important limitation for anyone with a modest compute budget.
§1, §5.1
Chapter 9 · Back in the review room
From one language to fifteen, with English labels.
Huda's team maps their insight jobs to the paper's tests. Sorting reviews is classification (XNLI). Pulling out product and brand names is entity recognition (NER). Answering "what do people say about the battery?" is question answering (MLQA).
Grey: mBERT with English labels. Axis starts at 40.
Table 1, XNLI accuracy (%)
Answering questions about reviews
| MLQA F1 | en | es | de | ar | hi | vi | zh |
|---|---|---|---|---|---|---|---|
| mBERT | 77.7 | 64.3 | 57.9 | 45.7 | 43.8 | 57.1 | 57.5 |
| XLM-R | 80.6 | 74.1 | 68.5 | 63.1 | 69.2 | 71.3 | 68.0 |
Table 3. Trained on English SQuAD only.
ONE MODEL vs SEVEN
Seven separate monolingual BERT models averaged 77.5. One shared 7-language model trained on all their data scored 80.0. And on English GLUE, XLM-R is only 1.0 point behind RoBERTa (91.8 vs 92.8).
Tables 4 and 5
Chapter 10 · What I personally think is wrong with it
The paper's own evidence argues back.
Six months later, Huda's dashboard works. But the team keeps a list of what still bothers them. Click each to see the evidence.
1Half the brain is a dictionary+
Using the paper's own formula, the vocabulary table holds about 256M of ~558M parameters in XLM-R Large (46%), and about 69% in XLM-R Base. And the authors show a bigger vocabulary would still help (+3% from 128K to 512K), but stopped at 250K for simplicity and softmax compute. The bottleneck is known and left in place.
Appendix B formula · §5.1
2The cure for the curse is "buy a bigger brain"+
The fix offered is simply increasing model capacity. Even then, XLM-100 at hidden size 1152 still lagged XLM-7. Every language still competes for one shared budget. The authors themselves flag this as a limitation for modest compute budgets, which describes most businesses.
§1 · §5.1, Figure 4
3The poorest languages are still poor+
After all the effort: English 89.1 vs Swahili 73.9 and Urdu 73.8, a 15-point gap. Several CC-100 languages (Assamese, Oromo, Sundanese, Xhosa) have about 0.1 GiB, below the "few hundred MiB" the paper says a language needs.
Table 1 · Table 6 · §3
4"Zero-shot" isn't quite zero+
Model selection uses development sets in all target languages. The authors concede this is not fully zero-shot, arguing real projects usually have a little data per language. For our team, that means budgeting for a small labelled check set in every market.
§5.2
5Thin evidence where we need it+
Entity recognition, the task behind extracting product names, was tested on only four European languages (English, Dutch, Spanish, German). There is no direct NER evidence for Arabic, Hindi or Urdu reviews.
§4 · Table 2
6Nobody here can rebuild it+
Pretraining took 500 V100 GPUs and 1.5M updates. A business can fine-tune it, but cannot fix flaws 1–3 itself. Every adopter inherits the vocabulary and language balance chosen in 2019.
§5.2
Epilogue
One model, trained only by reading, learned to hear a hundred languages.
Its lesson: data and scale beat clever objectives. Its flaw: every language shares one fixed budget, and the smallest languages get the least.
NEW IDEAS
No translations, no language tags, web-scale data, rationed sampling, one big vocabulary.
WELL EXECUTED
Careful ablations of every trade-off: languages, α, vocabulary, capacity, data source.
BUSINESS VALUE
Label once in English, deploy in many languages, with one model to maintain.
"The apprentice learned to understand everyone. The next one must learn to treat everyone equally."
Questions & answers
Ask us anything.
Why not just translate every review into English?
The paper tested it. Translating test data and using RoBERTa averaged 77.8; XLM-R without any translation averaged 80.9. Translation also adds a cost and a delay to every single review. (Table 1)
Does going multilingual hurt English quality?
Barely. On GLUE, XLM-R scored 91.8 vs RoBERTa's 92.8, and beat BERT-Large (90.2). (Table 4)
Can it read reviews mixing Arabic and English?
That is a stated design goal. XLM-R drops language embeddings, which the authors say lets it better deal with code-switching. (§3)
Why drop translated sentence pairs (TLM)?
A properly tuned MLM-only model reached over 75% XNLI, on par with the TLM version, and parallel data is scarce for low-resource languages. (§5.1)
What does α = 0.3 actually do?
It flattens how often each language is sampled. Higher α favours high-resource languages, lower α favours low-resource ones; 0.3 was the best overall balance. (§5.1, Figure 5)
Why 250K vocabulary, if bigger helps?
Bigger did help (+2.8% from 32K to 256K at fixed size). They chose 250K for simplicity and because of softmax compute constraints. (§5.1)
Is it really zero-shot?
Not fully. Model selection uses a joint dev set across languages. One model selected jointly scored 0.6 lower than N separately selected models (71.3 → 70.7 for XLM-100). (§5.2)
How much text does a new language need?
The authors estimate a few hundred MiB is usually the minimum to learn a BERT-style model. Wikipedia alone was too small for Swahili and Urdu. (§3, §5.4)
Quick check for the room
Tap an answer.