Building AI products for Indian languages: what changes beyond English

Short answer: the hard parts of AI for Hindi and other Indian languages are rarely in the model. Normalise Unicode text, measure token cost on real user messages, decide how you handle mixed scripts and formality, and evaluate with native speakers. Voice is often the easiest way in for users, so test it early with real accents and noise.

Large language models now handle Hindi and many other Indian languages far better than they did a few years ago. But shipping a product that people trust in those languages takes more than translating the prompt. The problems are rarely in the model; they are in the text, the costs, the users' habits and the way quality is measured.

Devanagari and other Indic scripts combine base letters with vowel signs (matras), the nukta and other marks. The same visible word can be stored as different sequences of Unicode code points.

• Normalise before you compare. Convert text to a single Unicode normalisation form (opens in a new tab) (usually NFC) before searching, deduplicating, caching or checking answers, or identical-looking strings will fail to match.
• Expect legacy encodings. Some users still type in older non-Unicode fonts such as Kruti Dev, and pasted text from those documents looks like gibberish to software. Detect and convert it, or tell users clearly what you support.
• Watch invisible characters. Zero-width joiners and non-joiners change how text renders; strip or preserve them deliberately.

Many tokenisers split Devanagari text into more tokens per word than English: a NeurIPS 2023 study of tokenisers across languages (opens in a new tab) found the same text can take several times as many tokens in some languages as in English. The same message can cost noticeably more and fill the model's context window faster. Measure with your provider's token counter (Claude (opens in a new tab), Gemini (opens in a new tab)) on real user text, not translated samples, and set budgets and context limits accordingly.

Real users write Hinglish in Latin script, switch languages mid-sentence and borrow English technical terms. Decide up front:

• Which scripts the product accepts, and whether it replies in the user's script or a fixed one.
• Whether you transliterate (for example Latin-script Hindi to Devanagari) and where.
• How formal the product should be. The choice between आप and तुम sets the tone of every answer.

English evaluation sets do not tell you whether Hindi output is correct, natural or appropriately formal. Build a separate evaluation set with native speakers:

• Real requests in the registers your users actually write in.
• Domain vocabulary (legal, medical or educational terms) with agreed correct forms.
• Scores for grammar, spelling, naturalness and tone, tracked separately from English quality.

Re-run it whenever you change a prompt or a model; improvements in English do not guarantee improvements in Hindi.

For many users, speaking is easier than typing in their own script. Current speech models make this practical: Google's Gemini Live API supports real-time conversation in dozens of languages (opens in a new tab), Hindi, Marathi, Tamil and Telugu among them, and Gemini's text-to-speech (opens in a new tab) covers those four, with more Indian languages in preview. Test with real accents, background noise and mixed-language speech before you commit.

• Use fonts with full coverage, such as Noto Sans Devanagari (opens in a new tab), and give lines extra height so matras are not clipped.
• Support phonetic and native keyboards, and make search tolerant of common spelling variants.
• Check how numbers, dates and currency appear (Indian digit grouping, lakh and crore) wherever money is shown.

• Normalise text to one Unicode form everywhere.
• Measure token usage on real user text.
• Decide your script, transliteration and formality rules.
• Build a native-speaker evaluation set and track it per language.
• Test voice with real accents and noise.
• Choose fonts and layouts that render every mark cleanly.

Building for users beyond English is one of the most rewarding problems in AI product work. We built Vyakarni, a Hindi-first AI grammar and writing assistant, end to end for SNS Innovation Labs. If you are planning a product for Indian-language users, our AI product engineering practice can help, or book a call.

• Unicode Standard Annex #15: Unicode Normalization Forms (opens in a new tab)
• Petrov et al., "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023 (opens in a new tab)
• Claude: token counting (opens in a new tab) and Gemini: count tokens (opens in a new tab)
• Gemini Live API: supported languages (opens in a new tab)
• Gemini-TTS: available languages (opens in a new tab)
• Google Fonts: Noto Sans Devanagari (opens in a new tab)