L 3 C ube- I ndic N ews: News-based Short Text and Long Document Classification Datasets in I ndic languages (2023)

Abstract: In this work, we introduce L3Cube-IndicNews, a multilingual text classification corpus aimed at curating a high-quality dataset for Indian regional languages, with a specific focus on news headlines and articles. We have centered our work on 10 prominent Indic languages, including Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada, Odia, Malayalam, and Punjabi. Each of these news datasets comprises 10 or more classes of news articles. L3CubeIndicNews offers 3 distinct datasets tailored to handle different document lengths that are classified as: Short Headlines Classification (SHC) dataset containing the news headline and news category, Long Document Classification (LDC) dataset containing the whole news article and the news category, and Long Paragraph Classification (LPC) containing sub-articles of the news and the news category. We maintain consistent labeling across all 3 datasets for in-depth length-based analysis. We evaluate each of these Indic language datasets using 4 different models including monolingual BERT, multilingual Indic Sentence BERT (IndicSBERT), and IndicBERT. This research contributes significantly to expanding the pool of available text classification datasets and also makes it possible to develop topic classification models for Indian regional languages. This also serves as an excellent resource for cross-lingual analysis owing to the high overlap of labels among languages. The datasets and models are shared publicly at https://github.com/ l3cube-pune/indic-nlp.

## introduction india boasts a rich linguistic diversity, with over 700 languages spoken, out of which 22 are officially recognized. some of the primarily spoken languages include hindi, bengali, marathi, telugu, tamil, gujarati, kannada, odia, malayalam, and punjabi. despite their widespread use, there's a notable scarcity of comprehensive indic language datasets, primarily due to their low-resource status and linguistic complexity @xcite . these indic languages are widely spoken all over india and have abundant data available on news websites, and social media but it's difficult to find a labeled dataset of news headlines, articles, and their categories for text classification. this disparity has hindered progress in machine learning and natural language processing (nlp) research for these languages. most indic languages share similarities, but they utilize different writing scripts, making it more challenging to accurately predict news categories. since english is used widely around the world, and many researchers have studied how to classify text in english, it has resulted in a surplus of classification datasets. the same is not the case for indic languages. although there are existing datasets for indian languages, they have some limitations, such as a lesser number of categories or inconsistent classification criteria for news articles, which narrows down the scope of research in this field. while the indicnlp news article dataset covers the major languages it is limited by the count of target labels and high accuracy @xcite . this calls for the creation of more intricate datasets to effectively assess the performance of models. in essence, there is a need for complex datasets that can thoroughly evaluate model effectiveness. text classification, a critical task in both machine learning and natural language processing (nlp), involves categorizing text documents into predefined classes based on their content. while there are a handful of publicly available datasets related to news, they often lack diversity in categories and sources, potentially leading to biased results. transformer-based models, like longformer foot_0 @xcite , require datasets with varying sequence lengths due to their sensitivity to text length. this highlights the need for specialized datasets to develop models for these lowresource languages. hence, we introduce l3cube-indicnews, a comprehensive indicnews classification dataset, sourced from diverse news websites specifically targeting low-resource languages. this dataset encompasses over 3 lakh records, distributed across 12 diverse news categories, offering an extensive resource for supervised text classification. each language dataset contains more than 26,000 rows, covering at least 10 significant news categories. l3cube-indicnlp foot_1 presents monolingual and multilingual models tailored to each indian regional language. the repository includes individual bert models for the languages focused on in this work. we conduct a comparative analysis of various monolingual and multilingual bert models, including l3cube monolingual bert @xcite , monolingual @xcite , indicsbert (multilingual) @xcite and indicbert @xcite . the key contributions of this work are as follows: ## curating the dataset we introduce l3cube-indicnews, a comprehensive dataset compilation designed to facilitate the classification of both short text and long documents. within indicnews, there are three meticulously crafted supervised datasets: short headlines clas-sification (shc), long document classification (ldc), and long paragraph classification (lpc). featuring a broad spectrum of information, the dataset comprises a minimum of 10 distinct categories corresponding to 10 prominent languages spoken in india. the careful organization of both language and newly defined categories enhances overall clarity. sourced from multiple websites, it ensures a diverse array of content, exclusively from reputable news sources. notably, there are no constraints on the length of articles and headlines, emphasizing a commitment to quality. in essence, it stands as a high-quality, versatile, and meticulously curated dataset. ## results for evaluating the models on these datasets, we use accuracy as our main evaluation metric to understand the performance of our models. key observations are outlined as follows: ## future work there is potential for expanding the dataset's labels to achieve broader category coverage or facilitate further category expansion. the dataset creation process has been streamlined through automation, eliminating the need for manual typing. looking forward, there is a prospect of implementing the curation of a manually typed or verified dataset, ensuring an even higher level of accuracy and reliability for future applications. currently, we have achieved proficiency in long document classification (ldc) for a single language, and the performance in long paragraph classification (lpc) is satisfactory. however, we acknowledge that the performance in short headlines classification (shc) falls short of optimization. to address this, we have implemented a unified model selection, specifically opting for the monolingual sbert based on its significant performance on individual datasets for each language. this chosen model exhibits competence in handling mixed datasets, comprising of shc, lpc, and ldc, and its performance has been thoroughly examined. it's important to note that our current focus remains on a single language. looking ahead, we recognize the potential for future developments, particularly in the realm of cross-dataset analysis. ## conclusion in this research paper, we introduce l3cube-indicnews, a comprehensive collection of three labeled datasets, encompassing over 3 lakh records in ten different indic languages. these datasets are designed for text classification tasks in the context of indian languages. within this paper, we provide an in-depth overview of the creation process, which involves the use of 10 to 12 distinct categorical labels to curate these supervised datasets. to assess the effectiveness of these datasets, we conducted fine-tuning on bert-based models, serving as a valuable benchmark for future research and development. our experiments involved four key models: l3cube monolingual bert, l3cube monolingual sbert, l3cube-indicsbert (multilingual), and ai4bharat indicbert. notably, our findings indicate that the bengalibert model achieved the highest accuracy when applied to the ldc dataset. we believe that the availability of our datasets will contribute significantly to the enhancement of nlp support for the indic languages, promoting its growth and development in this field.