Event Annotation and Detection in K annada- E nglish Code-Mixed Social Media Data (2023)

Abstract: Code-mixing (CM) is a frequently observed phenomenon on social media platforms in multilingual societies such as India. While the increase in code-mixed content on these platforms provides good amount of data for studying various aspects of code-mixing, the lack of automated text analysis tools makes such studies difficult. To overcome the same, tools such as language identifiers, Parts-of-Speech (POS) taggers and Named Entity Recognition (NER) for analysing code-mixed data have been developed. One such important tool is Event Detection, an important information retrieval task which can be used to identify critical facts occurring in the vast streams of unstructured text data available. While event detection from text is a hard problem on its own, social media data adds to it with its informal nature, and code-mixed (Kannada-English) data further complicates the problem due to its word-level mixing, lack of structure and incomplete information. In this work, we have tried to address this problem. We have proposed guidelines for the annotation of events in Kannada-English CM data and provided some baselines for the same with careful feature selection.

## introduction with the rising popularity of social media platforms such as twitter, facebook and reddit, the volume of texts on these platforms has also grown significantly. twitter alone has over 500 million test posts (tweets) per day foot_0 . india, a country with over 300 million multilingual speakers, has over 23 million users on twitter as of january 2022 foot_1 , and codeswitching can be observed heavily on this social media platform @xcite . code-switching or code-mixing foot_2 occurs when "lexical items and/or grammatical features from two languages appear in one sentence" @xcite . multilingual society speakers often tend to switch back and forth between languages when speaking or writing, mostly in informal settings. it is of great interest to linguists because of its relationship with emotional expression @xcite and identity. however, research efforts are often hindered by the lack of automated nlp tools to analyse massive amounts of code-mixed data @xcite . below is an example of a code-mixed kannada-english tweet that has also been translated into english. named entities have been tagged along with the language tags (ka-kannada, en-english, ne-named entity, univ-universal). ## background and related work the study of events dates back quite a long time, and pre-linguistic definitions of events sought to describe and recognise events as "change in aspects of the perceived sense." before we start with event detection, we should be first clear on what constitutes an event in a sentence. the guidelines for annotation of events have been published in english in 2006 @xcite , while guidelines for event annotation in monolingual kannada data was recently published by @xcite . automated event mention detection in an open domain setting is a keystone for various information extraction tasks. this task was first brought to light during semeval-2007, where the shared task task 15: tempeval temporal relation identification @xcite was added as a new task with a focus on identification of temporal constructions. one of the six proposed tasks was concerned with the detection of events mention extent in the text. in early works, most of the methods @xcite proposed for event extraction have focused on news articles, which is the only best source of information for current events. more recently, @xcite proposed nlp techniques, handwritten rules and wordnet for event extraction from emails. they have used methods like event trigger identification and morphological analysis for event extraction from the email and achieved an accuracy of 72%. with the ability of social media tools to virally popularize news items and their acceptance across the masses, numerous media agencies have been relying on twitter, facebook feed pages to disseminate their news highlights. twitter feeds for hindi 45 and kannada 67 are few examples of social media forums continuously posting the news items. among the posts made by these feeds, only a small fraction of tweets contain events. @xcite developed the first open-domain event extraction tool (twical) for twitter data. there have been attempts at event detection from social media streams (hossny and mitchell, 2018), but we will not be working on those as part of this work. we have recently seen an interest related to kannada-english code mixed data. sowmya lakshmi and @xcite have proposed an automatic word-level language identification (lid) system for sentences from social media posts. @xcite reported a work on annotating cm kannada-english data collected from twitter and creating pos tags for this corpus. s and shrivastava (2022) presented an automatic ner of kannada-english cm data. we are using the dataset created in the above works related to ner and pos tags in our event extraction task. ## annotation methodology in this section, we shall discuss the method that we have used to annotate our corpus. we label each tokens with the inside-outside-beginning (iob) format @xcite , where b refers to the beginning of an event, i refers to the token that is part of the event but not the first token and o refers to all other tokens. we propose these principles, which are inspired by timeml, and are organised by the part-of-speech (pos) of the event nugget. nouns, finite verbs, non-finite verb constructions such as infinitives, and adjectival and adverbial participle constructions are examples of these components of speech. as most of the codemixed kannada-english sentences follow the structure of kannada grammar while swapping language 1009 for keys words such as common nouns, we will follow the guidelines that we have proposed for kannada monolingual event annotation in our paper -detection and annotation of events in kannada (2020) @xcite . for english grammar based sentences, we have the timeml annotation guidelines @xcite . for kannada grammar based sentences, we have the annotation guidelines from @xcite . some examples are given below. noun-nominal events refer to abstract nouns that relate to a temporal phenomenon and inherently convey a notion of finiteness, such as chunavane (election), pasavu (famine), etc. ka-en: karnataka chunavane sheeghra agutte antha namme home minister announce madidru transation: our home minister announced that karnataka election will be soon finite verb-categorized as events because they denote actions that bring about a change in the state of the world. they possess tense and aspect information, which inherently conveys a notion of temporality. ## supervised approaches for event detection supervised machine learning is a category of machine learning where a model is trained on labeled training data, where each data point has both input features and corresponding output labels. the goal of supervised learning is for the model to learn the mapping between input features and output labels so that it can make accurate predictions on unseen data. as we have a limited amount of data for our task of event detection (2,250 sentences), we will only explore probabilistic models. they have been described in the following sub-sections. ## feature selection in "a semantico-syntactic approach to event-mention detection and extraction in hindi", by goud et al., they have used specific features for hmms and some additional features for crf. we shall use those features along with an additional feature -language identifier (lid) which will either be kannada (ka), english (en), named entity (ne) or a universal (univ) token. the list of features picked for the hmm are: ## experimental setup in this section, we conducted a series of experiments to assess the impact of different features and model parameters. our goal was to understand the effect of individual features that we discussed in section 6 and explore the influence of various model settings that we discussed in the section 5. to achieve this, we performed experiments using different combinations of features and systems with rigorous hyper-paparemeter tuning, for both hmm and crf using gridsearch. we experimented with using a subset of features together and all features simultaneously. to evaluate the performance of our classification models, we employed 5-fold cross-validation. this approach helped us validate the models by partitioning the data into five subsets, training the models on four subsets, and evaluating their performance on the remaining subset. we repeated this process five times, each time using a different subset for evaluation. for the implementation of the algorithms, we utilized the data handling libraries in python for hmm and crf++ 9 tool for crf, which are efficient and user-friendly tools for our tasks. in terms of data splitting, we allocated 60% of the data for training, 10% for validation, and 30% for testing. this division allowed us to train our models on a substantial portion of the data, validate their performance on a separate subset, and finally assess their generalization ability on a dedicated testing set. by conducting these experiments, we aimed to gain insights into the impact of different features and model parameters, enabling us to make informed decisions about the best configuration for our classification models. we use precision, recall and f1-score as our evaluation metrics. ## results and analysis as crfs are trained to develop a local model at the sentence level for event identification, as we observe improvements in the accuracy of the crf's local model, it indicates that the engineered features effectively capture all the available information within the sentence's local structure. even then, the results are nowhere near perfect but the purpose of the models was just to provide an exploratory baseline for the dataset created with the annotation guidelines. we should also note that the grammatical structure of the sentences are not standard, making prediction harder. this gets more difficult with kannada-english code-mixed data as mixing happens at word-level, mostly for kannada language 9 https://taku910.github.io/crfpp/ model precision recall f-1 score hmm 0.38 0.42 0.39 crf 0.46 0.63 0.53 ## conclusion in conclusion, our work on event detection for code-mixed kannada-english social media data has yielded the following: