An Annotated Corpus for Realis Event Detection in Short Stories Written in E nglish and Low Resource A ssamese Language (2023)

Abstract: This paper presents an annotated corpora of Assamese and English short stories for event trigger detection. This marks a pioneering endeavor in short stories, contributing to developing resources for this genre, especially in the low-resource Assamese language. In the process, 200 short stories were manually annotated in both Assamese and English. The dataset was evaluated and several models were compared for predicting events that are actually happening, i.e., realis events. However, it is expensive to develop manually annotated language resources, especially when the text requires specialist knowledge to interpret. In this regard, TagIT, an automated event annotation tool, is introduced. TagIT is designed to facilitate our objective of expanding the dataset from 200 to 1,000. The best-performing model was employed in TagIT to automate the event annotation process. Extensive experiments were conducted to evaluate the quality of the expanded dataset. This study further illustrates how the combination of an automatic annotation tool and human-in-the-loop participation significantly reduces the time needed to generate a high-quality dataset.

## introduction event detection aims to find event instances, commonly called event triggers. an event is a particular instance of the occurrence of something at a specific time and place, thus indicating state change @xcite . event detection helps extract relevant information from vast amounts of text. it is considered a crucial part of information extraction due to its importance in several downstream tasks like question-answering @xcite , knowledge-base construction and completion @xcite , etc. significant efforts have been put into event detection in texts from domains like newswire @xcite and biomedical @xcite . however, event detection from short stories remains under-studied, specifically in low-resource languages @xcite . the overarching aim of this work is to explore event detection in assamese and english short stories written in the indian context. recognizing events within literary texts, such as short stories, poses substantial challenges. challenges arise due to several factors. literary narratives tend to feature more complex and intricate storytelling structures @xcite . additionally, the language used in news articles or biomedical texts often describes real-life occurrences and establishes clear cause-and-effect connections between events @xcite . in contrast, literature is fundamentally a creative pursuit, and within most literary narratives, events are not necessarily grounded in factual reality. the coexistence of both actual or realistic events and non-realistic events within these texts further complicates the task of event detection. exploring events in short stories would involve examining how events are conceptualized, encoded, and expressed. towards the primary objective, this work presents an annotated corpus for assamese and english short stories. two hundred short stories in both assamese and english languages are manually annotated. only actually happening events or realis events are considered for annotation. examples of sentences with events in assamese and english are shown in figure 1 . several baseline models are trained on a subset of manually annotated datasets and evaluated for the event detection task. however, creating a large gold standard dataset takes time and effort. a silver standard corpus is an alternative to a gold standard annotated corpus @xcite . the idea of employing automatic annotation systems and a method to synchronize the resulting annotations to produce the silver standard was first put forth @xcite . this work also opts for a silver standard dataset as it is still being used widely @xcite ). an automatic annotation tool, "tagit", is developed to create this dataset. using tagit, the event detection dataset is expanded from 200 to 1,000 stories in each language. the quality of the silver standard augmented data is significantly enhanced when the process is integrated with human-in-theloop participation. for an input document (short story), tagit recommends the possible events. this aids in further verification (with greater ease) by the human annotator. the major contributions of this work are as follows: ## related work event trigger detection in a text document is a necessary first step for identifying events @xcite . over time, there has been a steadily growing demand for corpora with extensive annotations @xcite . to study linguistic phenomena at different levels, a corpus in a machine-readable form is crucial. this corpus can be utilized in several domains. also, supervised machine learning techniques depend on annotated corpora to create and assess nlp techniques. ace 2005 marked the pioneering news corpus dedicated to addressing real events @xcite . the absence of biological corpus for event detection led to the development of the ge-nia corpus @xcite . annotators from amazon mechanical turk developed a set of sentences extracted from news articles, covering reports on various topics such as events, science journalism, and finance, classifying them as either general or specific @xcite . the gun violence database (gvdb) was unveiled as a fresh dataset containing articles on gun violence from local newspapers and television station reports, all of which have been annotated @xcite . @xcite thoroughly explored numerous datasets for the purpose of event detection. most prior research focused on handling factual information. in contrast, very few have analyzed events within literary texts. the first attempt to identify events in literary content by generating a marked collection of real events from novels was proposed in @xcite . the scarcity of annotated data poses a significant challenge in event detection tasks, especially when dealing with indian languages with limited linguistic resources @xcite . several efforts have been made to detect events in indian language documents written in hindi @xcite , tamil @xcite , bengali @xcite , and marathi @xcite . however, these works have primarily focused on news and social media text. only a single work focused on the detection of natural calamities by processing assamese language posts in social media @xcite . this work attempts to identify events within children's short stories. literary works like short stories are longer than fact-based articles and exhibit unique event structures. detecting real events within these imaginative narratives poses a significant challenge. the narrative structure caters to the psychology of children, featuring situations where inanimate objects or animals converse with each other. these distinctive scenarios set these stories apart from both factual and other literary texts. ## dataset this section describes the dataset collection method (subsection 3.1), annotation guidelines (subsection 3.2), and the annotation process (subsection 3.3). finally, the dataset statistics are summarized in subsection 3.4. ## results comparative performance analysis of different baseline neural models are shown in ## dataset expansion this section illustrates the process of expanding the dataset from 200 to 1000 using an automatic process. for this purpose, "tagit", an easy-to-use automatic annotation tool, was developed and employed. later, in this section, the process of data augmentation is also elaborated. ## conclusion datasets for event detection have predominantly emphasized on news and biomedical do-mains. this work focuses on developing resources (for event detection) in the domain of short stories. accordingly, this paper introduces a novel corpora containing short stories in both low-resource assamese and english language. initially, a gold standard corpus was created, which contains 200 manually annotated short stories in the two languages. further, this dataset is expanded to 1000 short stories using an automatic annotation tool, resulting in a silver standard dataset. for that purpose, this work proposes an automatic annotation tool called tagit. event detection models were separately trained using the gold and silver standard datasets. the experimental observations indicate that the model learned from the silver standard dataset yields comparable results. the silver standard dataset is further verified using human-in-theloop process, improving the performance while reducing the annotation time significantly. in future, we aim to classify realis events short stories written in other indian languages.