Indian AI model’s local language viability faces content availability barrier

A key goal of Indian startups and the IndiaAI Mission has been to create a foundational large language model that is tuned to Indian languages. That has so far been a tall order, as the amount of Indian language content online — a key source of training data in English, which is what most foundational models like OpenAI and Google’s primarily work with — has been a fraction of other well-represented languages. Digitising content like news and books to extract local language content is also not a surefire solution, Mr. Pani said, as the sheer volume of public user posts on the internet dwarfs user generated content on the public web by volume. The creation of indic language datasets for a homegrown AI model that may be substantially useful, therefore, would depend on better availability of Indian language data, which in turn depends on more Indian language content being posted online. The IndiaAI Mission is also planning a repository of Indian language datasets, IT Minister Ashwini Vaishnaw said earlier this month, with details of the IndiaAI Datasets Platform to be announced later.