Building Data Infrastructure for Low-Resource Languages (2025)

Abstract: The MLCommons Datasets Working Group presents a comprehensive initiative to advance the development and accessibility of artificial intelligence (AI) training and testing resources. This paper introduces three key projects aimed at addressing critical gaps in the AI data ecosystem: the Unsupervised People’s Speech Dataset, containing over 821,000 hours of speech across 89+ languages; a strategic collaboration with Common Crawl to enhance web crawling capabilities for low-resource languages; and a framework for knowledge graph extraction evaluation. By focusing on languages other than English (LOTE) and creating permissively licensed, high-quality datasets, these initiatives aim to democratize AI development and improve model performance across diverse linguistic contexts. This work represents a significant step toward more inclusive and capable AI systems that can serve global communities.

## introduction the mlcommons foot_0 consortium supports numerous activities in the machine learning innovation space. these activities include building performance benchmarks including those in ai risk and reliability and building benchmarks for evaluating ai systems requires rigorous, standardized test datasets. mlcommons builds open, large-scale, and diverse datasets and a rich ecosystem of techniques and tools for ai data that helps the broader community deliver more accurate and safer ai systems. our shared research infrastructure and diverse community aid the scientific research community to derive new insights for new breakthroughs in ai. the datasets working group foot_1 is one of 17 working groups under the mlcommons umbrella and currently has three main projects it is opensourcing: while these are our current projects, the working group is always open to new topics and projects. ## background datasets fuel machine learning: a model is only as good as the data upon which it was trained. ima-genet @xcite , created for less than half a million dollars, is arguably the one that gave rise to modern machine learning. unfortunately, most public datasets today are either small (relative to private commercial datasets), static, licensed for research use only, or some combination of those things. datasets must be large to train accurate models. to remain relevant, datasets should be constantly improved as gaps in their coverage are identified. lastly, datasets require a permissive public license to enable new businesses, products, and services globally. the datasets working group creates and hosts public datasets that are large, actively maintained, and permissively licensed -especially for commercial use. we aim to develop a center of expertise and supporting technologies that dramatically improves the quality and reduces the cost of new public datasets. we believe that a modest investment in public datasets can have an impressive return in terms of machine learning innovation and market growth. the datasets working group's first project was the people's speech dataset @xcite , an open speech recognition dataset that is approximately 100 times larger than existing open alternatives.