Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Hugging Face and partners released a 2.8 trillion-token Indic pretraining corpus with an open GPU pipeline, aiming to expand Indian-language AI resources and technical model development.
A new open corpus of approximately 2.8 trillion tokens across 22 Indian languages has been introduced, developed jointly by NVIDIA, IIT Madras, and AI4Bharat. The dataset includes native-script and romanized text, covering languages listed in India's Constitution and spoken by over 97% of the country's population. The release accompanies an open, GPU-accelerated pipeline for translation, quality assessment, and repair, enabling developers to adapt the process to their own data needs.
The corpus integrates curated English materials such as mathematics, code, and question-answer content translated into Indian languages, expanding training resources for models that explain technical concepts and solve problems in those languages. Each output is traceable to its source document and undergoes quality checks to ensure fidelity, addressing a core challenge identified by AI4Bharat: the shortage of high-quality Indic training data. The workflow preserves technical content like equations, tables, and code while supporting 34 language-script configurations.
The open generation recipe allows teams to reproduce the data, adapt prompts and quality checks, or use their own source documents to generate additional synthetic Indic training data. By sharing both the corpus and the pipeline, the initiative seeks to democratize large-scale Indic data generation and accelerate model development across Indian languages. The approach complements naturally authored Indic text, which remains critical for local knowledge and cultural context.
Continued pretraining for Nemotron 3.5 Super VL incorporated 10 billion Indic tokens from a twelve-language subset of this corpus and natural Indic text. The technical report details the model's performance and the full data generation process, including translation, quality assessment, and targeted repair stages. The pipeline, implemented using NVIDIA's NeMo Data Designer and Gemma-4-31B-it, supports document-level translation and script conversion while maintaining provenance for auditing and reuse.