Why Token Costs Run Higher in Non-English Deployments
LLM token costs run higher in non-English deployments because tokenizers are built and tuned primarily on English text, and non-English languages — especially those using non-Latin scripts — typically require more tokens to represent the same amount of meaning. The same sentence that costs roughly 15 tokens in English can cost 30–45 tokens in a language like Hindi or Arabic, and since providers charge per token, that difference shows up directly in the monthly bill, not just in model behaviour.
How Tokenization Actually Works
Models don't process whole words. They break text into sub-word units drawn from a fixed vocabulary learned during training, using a method called byte-pair encoding or a close variant of it. That vocabulary is shaped by the training data it was built from, and training data is disproportionately English. The practical effect: common English words and word-fragments get their own efficient, single tokens, while other languages — particularly those using non-Latin scripts — get split into more, smaller tokens to represent the same meaning, because the vocabulary was never optimized for them in the first place.
The Multiplier in Practice
Representative figures: Spanish runs around 1.3x the English token count for equivalent meaning, Bahasa and Arabic closer to 1.8–2.0x, and scripts like Bengali, Tamil, Telugu, and Gujarati can run as high as 2.5–3.0x. The full standard multiplier table, covering ten languages plus a custom option, is in Introducing the Location-Adjusted Edition.
Why the Effect Varies by Script, Not Just by Language
The size of the multiplier tracks how far a language's writing system sits from the Latin-alphabet, English-heavy data a tokenizer's vocabulary was built from, more than it tracks how many people speak the language. Spanish, written in the same alphabet as English and sharing a fair amount of vocabulary structure, sees a comparatively modest multiplier. Languages written in scripts with little representation in that training data — and especially languages where a single character can carry more meaning than a single Latin letter — see the vocabulary fall back to splitting text into smaller, less efficient fragments far more often, which is what drives the multiplier toward the higher end of the range rather than the lower end.
Why This Compounds With Conversation Length
The effect doesn't just raise the starting cost of a single call — it compounds across a conversation. Multi-turn workflows that carry accumulated history forward resend that history, in the same higher-token-count language, on every subsequent turn. A non-English conversation's input token count doesn't just start higher than its English equivalent; it grows faster, turn over turn, because every re-sent piece of history is paying the same multiplier again.
Why This Is Easy to Miss When Piloting in English
Most AI pilots are built and demonstrated in English first, even when the eventual deployment market speaks a different language, because English tooling, documentation, and example prompts are simply more available. That sequencing quietly hides the token-cost gap: the pilot's cost figures look reasonable because they were measured in the language the tokenizer handles most efficiently, and the multiplier only shows up once the system is actually localized and run against real conversations in the target language — often after the budget has already been set from the pilot's numbers.
What This Means for a Budget or Business Case
Don't apply a single flat per-call cost assumption across markets and languages — apply a language-specific multiplier to the token cost line specifically, not to the whole cost model uniformly. It's also worth knowing that the standard multipliers assume a general-purpose, frontier model trained predominantly on English and adapted to other languages afterward. A model built natively for a specific language — the kind some providers now train from the ground up for a particular linguistic context — can carry meaningfully better token efficiency for that language than a general-purpose model handling it as a secondary capability. That's worth validating case by case against the specific model and task, not assumed from a vendor's marketing.
A Hypothetical Illustration
Picture two otherwise-identical customer support deployments, one serving an English-speaking market and one serving a market where the primary language carries a 2.5x multiplier. If both deployments handle the same call volume and the same average conversation length, the second deployment's token bill runs well over double the first — not because the AI system is doing more work in any meaningful sense, but because the same conversation costs more tokens to represent in that language. A budget or business case built by applying the first market's per-call cost to the second market, without adjusting for the multiplier, understates the second deployment's true run-cost by exactly that margin.
The AI Investment ROI Model — Location-Adjusted Edition provides the same NPV/IRR/payback/BCR core as the base model, plus a Blended Token Cost Multiplier across ten languages and a Market Configuration tab that converts local deployment costs to USD.
→ Available at www.viksya.com