As of May 2026, here is the multilingual classification pattern recurring across MemX user traffic. An Indian user sends a message to the MemX WhatsApp memory bot: `मेरी फोटो दिखाओ` ("show me my photos"). The MemX LLM intent classifier, running on Gemini 2.5 Flash with the single-prompt setup, classifies it as `save_memory`. The message gets saved as a new memory titled "show me my photos." The user never gets their photos. They open the app, scroll the library, give up, go back to texting. This is what an English-only developer never sees: the silent failure that pushes a multilingual user back into the manual workflow.
When the MemX team ran the benchmark in early 2026, Gemini 2.5 Flash single-prompt scored 58% on multilingual cases. GPT-4.1 nano failed Hindi `मेरी फोटो दिखाओ` on 2 of 3 runs. Arabic `اعرض لي صوري` ("show my photos") failed on 1 of 3. GPT-4o mini failed it on 1 of 3 too. The failure pattern was identical across the eval: when the model could not confidently understand the language, it defaulted to `save_memory`. It treated every confused input as a memory worth keeping.
Quick takeaways (as of May 2026): GPT-4.1 nano scores 62.9% on Hindi MMMLU and 65.9% on Arabic vs 66.9% overall, so the "nano tax" lands hardest on non-English users. Arabic users pay roughly 70% more tokens per word on the GPT-4o tokenizer than English users (our measurement: Arabic ~1.97, Hindi ~1.89, English ~1.16 tokens per word). Cross-lingual Collapse: chain-of-thought reverts to the model's dominant pretraining language even when the prompt is in another. Anthropic's Opus 4.5 and 4.6 system cards run automated language evals in English, Arabic, French, Korean, Mandarin, and Russian. Hindi is not on that list. Cohere Aya Expanse 32B is the most credible non-English-first model shipping right now.
The 58% problem
Specific examples from the MemX benchmark:
- Hindi `मेरी फोटो दिखाओ` got classified as `save_memory` on 2/3 runs (gpt-4.1-nano).
- Arabic `اعرض لي صوري` got `save_memory` on 1/3 runs (gpt-4.1-nano, gpt-4o-mini).
- Hinglish "yaar woh restaurant ka naam kya tha?" got `save_memory` on Gemini 2.5 Flash single-prompt (language bias category).
- Hindi `मेरी होटल बुकिंग कब है?` ("when is my hotel booking?") got `save_memory` on 1/3 runs across multiple models.
- The Gemini 2.5 Flash-Lite production model (two-stage pipeline) scored 100% on both multilingual and language-bias categories. The same model in single-prompt setup dropped to 75% on Hindi and Hinglish.
Why classifiers default to save when confused
Same mechanism documented in the `12345` post. RLHF rewards action over abstention. When the model has low confidence, it falls back to the highest-prior label. For the MemX app, that label is `save_memory` (the most common action). For your app, it will be whichever action your training distribution emphasised. The failure mode is universal. The specific failure depends on your prior.
The multilingual version of this is sharper because confidence drops cliff-edge on non-English inputs. The model is not 5% less confident on Hindi. It is 30 to 40% less confident, because Hindi tokens make up a tiny fraction of training. Low confidence plus action-bias equals silent misclassification.
The English pivot, explained
The June 2025 paper "Cross-lingual Collapse" (arXiv:2506.05850) named what others had hinted at. Under reinforcement learning with verifiable rewards, the chain-of-thought of a multilingual model systematically drifts toward its dominant pretraining language as reasoning depth rises. The model thinks in English, translates back at the end, loses nuance during the round trip. The paper attributes the effect to English-centric priors, long-CoT GRPO optimization, task difficulty, and high-entropy decoding.
For Hindi or Arabic users, this means their intent gets round-tripped through an English-dominant latent space before classification. Linguistic features that matter in the source language (formal versus informal address, kinship terms, code-switching markers) get flattened. Pure-English speakers do not see this because there is no round trip.
What the benchmarks actually say (and don't)
OpenAI's own MMMLU evals tell the story:
| Model | Overall MMMLU | Hindi | Arabic |
|---|---|---|---|
| GPT-4.1 nano | 66.9% | 62.9% | 65.9% |
| GPT-4o mini | 70.5% | 69.2% | 70.9% |
| GPT-4.1 | 83.7% | 84.2% | 84.4% |
| Gemini 2.5 Flash-Lite (Global MMLU Lite avg) | 84.5% | n/a | n/a |
GPT-4.1 nano is materially worse than 4o mini on Hindi (a 6.3 point gap) and Arabic (a 5.0 point gap), despite being OpenAI's newer model. The "nano tax" gets paid disproportionately by non-English users. Marketing copy says "40+ languages supported." The per-language breakdown says "benchmarked on 14, optimised for English."
Tokenization is a tax on non-English users
Petrov et al. (NeurIPS 2023) measured tokenizer disparities across languages and found up to 15x differences between language pairs. The 2025 "Token Tax" paper extended that analysis and quantified the cost: doubling a language's fertility quadruples training time and cost. We measured the GPT-4o tokenizer ourselves on production text: English averaged 1.16 tokens per word, Hindi 1.89 (+63%), Arabic 1.97 (+70%). Some Devanagari, Thai, and Burmese content still spikes 3x or more.
What this means in production: a Hindi user sending the same semantic message as an English user pays 63% more tokens, fits 63% less content into the context window, and gives the model 63% less prompt-tuning headroom. The bias sits in the tokenizer, before any model reasoning happens.
Code-switching: Hinglish is its own language
WhatsApp messages in India are rarely pure Hindi or pure English. They are Hinglish: "yaar woh restaurant ka naam kya tha?" ("hey what was that restaurant's name?"). Same in Singapore (Singlish), Egypt (Arabish/Franco-Arabic), and most multilingual markets. GLUECoS (Microsoft, ACL 2020) covers English-Hindi and English-Spanish code-switching across six tasks. CodeMixBench (arXiv:2505.05063, May 2025) extends the idea to code generation across Hinglish, Spanish-English, and Chinese Pinyin-English. Both show that code-switched inputs degrade LLM classification well beyond what either pure language would.
There is a darker version of this finding in safety research. Yoo et al.'s 2024 paper on code-switching red-teaming (arXiv:2406.15481) showed that code-mixed prompts land 46.7% more successful attacks than English-only adversarial prompts, with attack success rising as more low-resource languages get interleaved. Safety alignment, like accuracy, is English-centric. The same code-switching that breaks classification also breaks the safety guardrails.
What the frontier labs claim vs what they ship
Anthropic's Opus 4.5 and 4.6 system cards (November 2025 and February 2026) run single-turn automated Trust & Safety evals in six languages: English, Arabic, French, Korean, Mandarin, and Russian. Hindi is not on that list, even though earlier Claude 3.5 work referenced it. Google markets Gemini Live with "40+ languages" (the October 2024 expansion included Hindi); the published per-language MMMLU benchmark covers 14.
Cohere Aya Expanse 32B is the most credible non-English-first frontier release available as of May 2026. It serves 23 languages including Hindi and Arabic, was built with data arbitrage and multilingual preference training, and posted a 54.0% win-rate against Llama 3.1 70B on m-ArenaHard with roughly half the parameters. The credible counter-example exists. It just is not the default vendor anyone reaches for.
The bias goes beyond classification
Liang et al. (2023) evaluated seven widely-used GPT detectors on 91 TOEFL essays written by non-native English speakers. The detectors misclassified the essays as "AI-generated" with an average false-positive rate of 61.22%. All seven detectors unanimously flagged 19.78% of essays. At least one detector flagged 97.80%. The same essays from US 8th graders got classified correctly. The bias is not only in classifiers we build for ourselves. It is in classifiers that gatekeep university admissions.
Production fixes that worked (and what to do in your own stack)
Concrete actions, in order of cost.
- Build a language-bucketed eval set before you ship. Minimum 30 prompts per target language per intent. Do not extrapolate from English accuracy.
- Run a language-detection pre-classifier before the LLM (fastText, CLD3, GlotLID all run in microseconds on CPU). Route confidently-detected non-English through a different prompt or a different model.
- Prefer the model that actually benchmarks better, not the bigger one. The MemX data: Flash-Lite beat Flash on multilingual. OpenAI's data: GPT-4o mini beat GPT-4.1 nano on Hindi by 6.3 points.
- Budget for the tokenizer tax. Forecast Hindi and Arabic at roughly 1.6 to 1.7x English token cost; some scripts at 3x or more. Your unit economics for non-English users are 2 to 4x your English unit economics.
- Add an explicit "uncertain" classification path. Do not let the model fall back to the modal class. That is exactly the failure mode that turned `मेरी फोटो दिखाओ` into a saved memory called "show my photos."
- Test code-switching explicitly (Hinglish, Singlish, Arabish). Use GLUECoS or generate your own with CodeMixBench's methodology.
- Red-team in the target language. Code-switched prompts deliver 46.7% more successful jailbreaks than English-only attacks; safety RLHF is overwhelmingly English.
- Do not trust marketing language counts. "Supports 40 languages" rarely means "benchmarked on 40 languages." Demand per-language numbers from your vendor.
If your users speak Hindi, Arabic, or Hinglish, your eval set has to speak it first. Every English-only test you ship is a silent regression for the roughly 4 billion humans who do not have English as a first language. The bias sits in the tokenizer, the training distribution, the eval rotation, the safety alignment, and the support docs. The fix has to happen at every one of those layers.
Key takeaway: multilingual LLM failures are not "AI is racist" hot takes. They are engineering problems with engineering fixes. Build language-bucketed evals, route by detected language, budget for the tokenizer tax, and never let your classifier default to the modal class on low confidence. The next Hindi user is going to send `मेरी फोटो दिखाओ` to your bot. Make sure it works.
01Why do LLM classifiers misclassify Hindi and Arabic queries?
When the model has low confidence (which happens disproportionately on non-English input), it falls back to the highest-prior label from its training distribution. For most apps that label is the most common action class. The mechanism is the same RLHF action-bias that causes English edge cases like `12345` to be misclassified. Non-English inputs hit it constantly.
02How much more do non-English users pay in LLM token costs?
On the GPT-4o tokenizer, Hindi averages 1.89 tokens per word vs 1.16 for English (+63%). Arabic averages 1.97 (+70%). Some Devanagari, Thai, and Burmese content still spikes 3x or more. The bias sits in the tokenizer, before any model reasoning happens. Petrov et al. (NeurIPS 2023, arXiv:2305.15425) and the 2025 "Token Tax" paper (arXiv:2509.05486) document the methodology; the Token Tax paper shows doubling fertility quadruples training cost.
03Which LLM is best for multilingual classification in 2026?
For classification: Gemini 2.5 Flash-Lite (84.5% Global MMLU Lite average per Google's model card, 100% on our Hindi and Arabic tests in two-stage pipeline) is the cost-effective default. For non-English-first applications: Cohere Aya Expanse 32B (serves 23 languages including Hindi and Arabic, 54.0% win-rate vs Llama 3.1 70B on m-ArenaHard at roughly half the parameters). Avoid GPT-4.1 nano for Hindi (62.9% MMMLU) and Arabic (65.9%).
04What is Cross-lingual Collapse in LLMs?
An empirical phenomenon (arXiv:2506.05850, June 2025) whereby a multilingual model's chain-of-thought reverts to its dominant pretraining language (usually English) even when prompted in another. The effect amplifies under RLVR training and longer CoT. Practical impact: non-English inputs get round-tripped through an English-dominant latent space and lose linguistic nuance before classification.
05What is the easiest fix for multilingual LLM failures?
Run a language-detection pre-classifier (fastText, CLD3, GlotLID) before the LLM. Route confidently-detected non-English through a different prompt or model. Combine that with an explicit "uncertain" classification class so the model never falls back to the modal action on low-confidence non-English input.
06How do I test multilingual LLM behaviour in CI?
Build a language-bucketed eval set with minimum 30 prompts per target language per intent. Include code-switched inputs (Hinglish, Arabish, Singlish) via GLUECoS or CodeMixBench. Run every PR against the multilingual set, not just the English set. Report per-language accuracy in CI comments. An English regression and a Hindi regression are different bugs.
07Does the bias against non-English speakers appear outside classification?
Yes. Liang et al. (2023, arXiv:2304.02819) tested seven GPT-detection tools on 91 TOEFL essays by non-native English speakers. Average false-positive rate: 61.22%. At least one detector flagged 97.80% of essays as "AI-generated." The same tools classified US 8th-grade essays correctly. The English-centric training distribution biases every downstream classifier built on top, not just intent classifiers like the MemX one.
08Are code-switched prompts a safety risk, not just an accuracy problem?
Yes. Yoo et al. (2024, arXiv:2406.15481, accepted ACL 2025) found that code-switched adversarial prompts achieved 46.7% more successful attacks than English-only baselines across ten frontier LLMs. Attack success rose as more low-resource languages got interleaved. Safety alignment, like accuracy, is English-centric, so testing safety only in English under-counts your real exposure.
