COMI - LINGUA : Expert Annotated Large-Scale Dataset for Multitask NLP in H indi- E nglish Code-Mixing (2025)

Abstract: We introduce COMI-LINGUA, the largest manually annotated Hindi-English code-mixed dataset, comprising 125K+ high-quality instances across five core NLP tasks: Token-level Language Identification, Matrix Language Identification, Named Entity Recognition, Part-Of-Speech Tagging and Machine Translation. Each instance is annotated by three bilingual annotators, yielding over 376K expert annotations with strong inter-annotator agreement (Fleiss’ Kappa ≥ 0.81). The rigorously preprocessed and filtered dataset covers both Devanagari and Roman scripts and spans diverse domains, ensuring real-world linguistic coverage. Evaluation reveals that closed-weight LLMs significantly outperform traditional tools and open-weight models in zero-shot settings. Notably, one-shot prompting consistently boosts performance across tasks, especially in structure-sensitive predictions like POS and NER. Fine-tuning open-weight LLMs on COMI-LINGUA demonstrates substantial improvements, achieving up to 95.25 F1 in NER, 98.77 F1 in MLI, and competitive MT performance, setting new benchmarks for Hinglish code-mixed text. COMI-LINGUA is publicly available at this URL: https://huggingface.co/datasets/LingoIITGN/COMI-LINGUA.

## introduction code-mixing is the blending of multiple languages within a single utterance-a pervasive phenomenon in multilingual societies, especially on social media platforms @xcite . over half of the world's population is bilingual or multilingual and frequently uses mixed-language expressions in digital communication @xcite . in the indian context, hindi-english (hinglish) code-mixed text is particularly widespread and presents significant computational challenges due to orthographic complexity, frequent language switches, and script variation between devanagari and roman forms @xcite @xcite . a characteristic example is: kal mujhe ऑिफस जाना hai, but ट्रािफक will be an issue, where hindi and english tokens co-occur and certain english words like "office" and "traffic" may appear in devanagari script. (english translation: "tomorrow i have to go to the office, but traffic will be an issue."). despite growing interest, current hinglish datasets have critical limitations: (1) a predominant focus on roman script, ignoring natural script variation @xcite @xcite , (2) limited scale and coverage @xcite @xcite , (3) insufficient task diversity @xcite @xcite , and (4) reliance on synthetic data generation and labeling rather than human annotation @xcite @xcite . to address these limitations, we present a novel comprehensive dataset comi-lingua (code-mixing and linguistic insights on natural hinglish usage and annotation) that advances hinglish code-mixing research. the key contributions include: ## related work code-mixing-the blending of multiple languages in a single utterance-poses major challenges for nlp due to its structural variability @xcite . this is especially true for hinglish, given their distinct scripts and syntax @xcite . progress is hindered by the lack of large, annotated datasets, as collecting and labeling such data remains costly and labor-intensive @xcite . language identification is a foundational task in code-mixed nlp. multiple approaches have been developed to detect language boundaries within mixed-language sequences, including statistical models, crfs, and deep learning-based techniques @xcite @xcite . these efforts have paved the way for improved preprocessing and downstream modeling of code-mixed data. ## results and observations traditional tools vs. llms: the comparative analysis of traditional nlp tools and llms reveals clear distinctions in performance across codemixed tasks. as shown in beyond these limitations, our analysis of smaller-weight models (7-8b parameters) uncovered some failure patterns; for mt task, qwen2.5-7b-instruct inappropriately provided empty outputs with structured labels such as "unit 1: english, unit 2: romanized hindi, unit 3: devanagari hindi" instead of providing actual translations. in ner tasks, aya-expanse-8b misclassified punctuation marks, tagging "(" as opening parenthesis and ")" as closing parenthesis rather than using standard entity categories and labelling as 'x'. pos tagging revealed more hallucina-tion patterns, with llama-3.1-8b-instruct generating repetitive sequences like "verb noun propn noun noun noun noun" for multiple instances. more concerning was the tendency of these models to output code snippets instead of task responses -llama-3.1-8b-instruct and qwen2.5-7b-instruct output python import statements and function templates, rather than returning direct predictions as: similar code generation patterns emerged across tasks, with the model providing import re and langdetect modules for the lid task rather than providing the actual labels. entity hallucination was prevalent in qwen2.5-7b-instruct, which generated anomalous labels like (live india) as 'x x x x x x' and inappropriately tagged terms such as "आईपीएल" as hashtag entity. mt task suffered from incomplete generation, with outputs abruptly ending mid-sentence, as observed in romanized hindi translation: "madras hāikōrt ne dāk vibhāg ko." (see ## conclusion and future directions llms often struggle with tasks like pos tagging, ner, and mt in code-mixed hindi-english due to their lack of exposure to indian multilingual data. errors such as mislabeling entities or hallucinating content arise from limited training on structurally complex and script-variable inputs. the comi-lingua dataset addresses these issues by providing high-quality, task-diverse, and richly annotated code-mixed text. fine-tuning on this dataset enables models to better handle linguistic ambiguity, reduce overfitting, and improve reliability across tasks. its inclusion of contextual examples and diverse sources-like social media and news-enhances the models' ability to generalize across formal and informal registers, while iterative refinement through active learning ensures sustained performance gains.