Artificial intelligence models require significantly more tokens to process some languages than English. This difference can increase costs, slow down responses and reduce the amount of information users can fit into a conversation.
A study published last month by the Coalition for Language Equity in AI Research found that AI models use far more tokens to process certain languages than others. Tokens are the units of text or data that models process. Researchers have called this disparity a hidden “tokenizer tax”, which can make AI more expensive to use in languages with a smaller online presence.
A tokenizer is a tool that breaks down raw text, images or audio into tokens. The study found that token usage varies significantly even among the 20 most widely used languages on the internet. The most expensive languages require between 1.5 and five times as many tokens as the least expensive ones. When researchers expanded their analysis to 200 languages, the gap in token costs for one AI model increased to 17 times.
The disparity affects more than just the price of using AI. Higher token counts increase response latency and reduce the effective capacity of a model’s context window. This means users working in languages other than English may be able to retain less information from documents or previous conversations within the same context.
Pratyush Choudhury, co-founder of Activate AI, India’s first venture capital fund dedicated exclusively to artificial intelligence, said the issue could also affect service quality. He compared it to an AI tutor that needs to retain a student’s previous mistakes and explanations. If that history fills the context window much faster in one language, the disadvantage can directly affect the quality of the lesson.
One way to address the problem is to improve the quality and availability of language data used to train models. Raoul Nanavati, co-founder and CEO of Mumbai-based AI voice infrastructure startup Navana.ai, highlighted the importance of collecting data in local languages and dialects.
He pointed to RESPIN, a collaborative initiative involving the Indian Institute of Science and funded by the Gates Foundation. The project collected text and recorded speech from native speakers of 38 dialects.
However, the study found that a language’s online presence is not the only factor determining how many tokens it requires. Its writing system also plays a significant role.
For example, Asturian has relatively few speakers and no representation in the Common Crawl dataset. Yet it benefits from using the Latin script and from its similarities to other Romance languages, such as Italian.
By contrast, languages written in scripts that tokenizers have encountered less frequently — including Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki — consistently face greater disadvantages across nearly all the models examined.
Dheemanth Reddy, CEO of Maya Research, said many tokenizers remain disproportionately optimised for English and other languages with abundant digital resources. He identified the lack of high-quality digital language data as a fundamental part of the problem. However, he warned that improving training data alone would not fix the tokenizer gap.
Reddy said that rapidly falling token prices could make the direct financial impact less significant over time, while better multilingual and synthetic data could improve model quality. However, addressing the tokenization gap will still require AI laboratories to deliberately design future tokenizers to process a wider range of languages and writing systems more efficiently.
Choudhury argued that language efficiency should be treated as a product requirement alongside accuracy, speed and reliability. He acknowledged that perfect equality may be unrealistic, but said persistent and significant disparities should be explained and accompanied by plans for improvement.
He suggested that major AI laboratories publish language-by-language scorecards showing the cost of successfully completing comparable tasks. He also called on them to commit to measurable improvements for the languages their models serve least effectively.
