Commentator: A Code-mixed Multilingual Text Annotation Framework (2024)

Abstract: As the NLP community increasingly addresses challenges associated with multilingualism, robust annotation tools are essential to handle multilingual datasets efficiently. In this paper, we introduce a code-mixed multilingual text annotation framework, COMMENTATOR, specifically designed for annotating code- mixed text. The tool demonstrates its effectiveness in token-level and sentence-level language annotation tasks for Hinglish text. We perform robust qualitative human-based evaluations to showcase COMMENTATOR led to 5x faster annotations than the best baseline.

## introduction code mixing is prevalent in informal conversations and in social media, where elements from different languages are interwoven within a single sentence. a representative example in hinglish such as "i am feeling very thand today, so i'll wear a sweater." (in this sentence, "thand" is a hindi word meaning "cold", while the rest of the sentence is in english), demonstrating seamless integration of hindi and english. a major challenge in nlp research is the scarcity of highquality datasets, which require extensive manual efforts, significant time, domain expertise, and linguistic understanding, as highlighted by @xcite . the rise of social media has further complicated annotation tasks due to non-standard grammar, platform-specific tokens, and neologisms @xcite . annotating these datasets presents unique challenges, including ensuring data consistency, efficiently managing large datasets, mitigating annotator biases, and reporting poor-quality instances. existing annotation tools often fail to address these diverse issues effectively. this paper introduces commentator, a robust annotation framework designed for multiple codemixed annotation tasks. the current version1 of commentator supports two token-level annotation tasks, language identification, pos tagging, and sentence-level matrix language identification. while commentator has already been used to generate a large number of annotations (more than 100k) in our ongoing project2, these are not part of the current demo paper. the focus of this paper is to present the capabilities and initial functionalities of the framework. figure 1 presents the framework commentator. we evaluate commentator by comparing its features and performance against five state-of-theart text annotation tools, (i) yedda @xcite , (ii) markup @xcite , (iii) in-ception @xcite , (iv) ubiai3 and (v) gate @xcite . the major perceived capabilities (see section 4.1) of commentator are (i) simplicity in navigation and performing basic actions, (ii) task-specific recommendations to improve user productivity and ease the 1as a continual development effort, it will be further extended to three more popular code-mixing tasks ner, spell correction and normalization, and machine translation. in addition, the codebase, the demo website with a detailed installation guide, and some hinglish sample instances are available on github4. currently, the functionality is tailored for hinglish, but it can be extended to support any language pair. ## existing text annotation frameworks text annotation tools are vital in nlp for creating annotated datasets for training and evaluating machine learning models. this summary reviews several key tools, each with unique features and limitations. ## experiments in this section, we perform two human studies to evaluate commentator against recent state-ofthe-art tools to ensure a comprehensive comparison with modern advancements and cutting-edge functionalities: (i) yedda ( yang et al., 2018) figure 6 : the admin interface of the commentator. ## conclusion and future work we introduce commentator, an annotation framework for code-mixed text, and compared it against five-six state-of-the-art annotation tools. commentator shows better user collaboration, operational ease, and efficiency, significantly reducing annotation time for tasks like language identification and part-of-speech tagging. future plans include expanding commentator to support tasks such as sentiment analysis, q&a, and language generation, making it an even more comprehensive tool for multilingual and code-mixed text annotation.