Negative documents are positive: Improving event extraction performance using overlooked negative data (2023)

Abstract: The scarcity of data poses a significant challenge in closed-domain event extraction, as is common in complex NLP tasks. This limitation primarily arises from the intricate nature of the annotation process. To address this issue, we present a multi-task model structure and training approach that leverages the additional data, which is found as not having any event information at document and sentence levels, generated during the event annotation process. By incorporating this supplementary data, our proposed framework demonstrates enhanced robustness and, in some scenarios, improved performance. A particularly noteworthy observation is that including only negative documents in addition to the original data contributes to performance enhancement. Our findings offer promising insights into leveraging extra data to mitigate data scarcity challenges in closed-domain event extraction.

## introduction closed-domain event extraction is a specialized task in natural language processing (nlp) that focuses on automatically identifying and extracting specific events or occurrences from text within a restricted domain, such as biomedical research, financial markets, political events, or sports @xcite . it plays a crucial role in capturing and categorizing relevant events, their attributes, and relationships, enabling applications such as information retrieval @xcite , trend analysis @xcite , and knowledge base construction @xcite @xcite . however, despite the advancements in nlp models, the scarcity of anno-tated data poses a persistent bottleneck in achieving accurate and reliable event extraction models. the limited availability of annotated data, crucial for training and evaluating such models, hinders their performance and generalizability @xcite . the annotation process plays a vital role in event extraction, requiring domain experts to meticulously label relevant events and their associated attributes. however, this process is often laborintensive, time-consuming, and expensive @xcite . the complexity and diversity of event types further complicate the task, as events can vary in structure, context, and representation. moreover, the need for inter-annotator agreement adds to the complexity, requiring multiple annotators to reach a consensus on the event labels. these challenges contribute to the limited availability of annotated data, restricting the performance and generalizability of event extraction models. to overcome the data scarcity challenge, we propose a model structure and training schema that harnesses the additional data generated as a natural by-product of the annotation process. specifically, we utilize coarse-grained data that classifies documents or sentences as containing an event or not, as shown in document no sentence no sentence sentence label document label 1 1 he said the union had already send a statutory letter to the uber office here in connection with the strike. negative positive 2 the leaders of the union also said the local taxi drivers had launched an attack against the online taxi drivers at the airport. positive 3 the online taxi drivers have been having a tough time for the last one year. negative 4 uber and ola are two prominent online taxi service providers in kochi. negative 5 earlier, some trade unions representing local taxi operators had come out in protest against the online taxi networks such as uber and ola. positive 2 1 tributes paid to field marshal cariappa, students sing prayers at his 'samadi' negative negative 2 madikeri: rich tributes were paid to the late field marshal k.m.cariappa at "roshanara" here, where his "samadhi" is located, to observe the birth anniversary of one of the great soldiers of the country. negative 3 prayers in different languages were rendered by students of the bharatiya vidya bhavan-kodagu vidyalaya (bvb-kv) and family members of the late field marshal. negative in our training schema, we incorporate the extra coarse data as two auxiliary tasks alongside the main event extraction task: document binary classification and sentence binary classification. by utilizing this supplementary data, our approach aims to augment the training set and enhance the performance and robustness of the event extraction model. the integration of this additional data has yielded promising results, effectively addressing the limitations caused by the lack of annotated data in closed-domain event extraction. this study contributes to the field by providing a practical solution to the data scarcity problem in closed-domain event extraction. by leveraging the extra data generated during the annotation process, we strive to advance the state-of-the-art in event extraction, paving the way for more accurate and efficient systems across various domains. the outcomes of our research have potential implications for numerous downstream applications, ultimately benefiting various sectors that rely on event extraction for knowledge extraction and decision-making processes @xcite . the following section provide a brief overview on studies related to our study. next we provide details of the multi-task model and the data we utilize for our experiments in sections 3 and 4 respectively. the experimental setting is described in section 5 in terms of a baseline and three experiment sets. we report results of our experiments in section 6 and summarize our findings in section 7. ## related work the performance of event extraction has been significantly depended on the amount of relevant data utilized for creating an event extraction system @xcite . the variety of the data contributes to the performance and generalizability of an event extraction system as well @xcite . figure 1 : the main structure of the multi-task model. after creating embeddings for each sentence in the document (see figure 2 ), these are sent through a bi-gru to get a context-aware representation for each sentence and a single representation for the whole document. the losses for the document and the sentences are obtained by passing these embeddings through their respective classification layers. the design of an event extraction system is another major determiner of the performance @xcite . the design should be able to utilize and encode as much as information available in the data for the target task. consequently, syntaxoriented rule creation @xcite , joint learning @xcite , multi-task learning (drissiya el-allaly et al., 2021), and pre-trained architectures @xcite have been developed and successfully applied in many event extraction scenarios. we follow these approaches both for data by increasing the size and variety of the data and benefiting from multi-task learning that is based on pre-trained architectures. although in different domains, both @xcite and @xcite are highly similar to our approach. they both adopt a multi-task structure within a joint learning framework, leveraging data at multiple levels of granularity. however, our approach diverges from theirs primarily in terms of incorporating document-level information, alongside sentence-level. an innovative aspect of our study is the revelation that integrating negative documents substantially augments performance and robustness, particularly in scenarios with limited data availability. ## model structure to solve event extraction tasks using deep learning techniques, they are commonly approached as token classification problems. in token classification, each word or token in the input text is assigned a label indicating its role in the event extraction process. one popular labeling scheme is the bio format @xcite , which stands for "beginning, inside, and outside." in this format, each token is labeled as either b-event, i-event, or o. the b-event label denotes the beginning of an event mention, the i-event label indicates that the token is inside the event mention, and the o label signifies that the token is outside any event mention. by converting the event annotations into the bio format, deep learning models can be trained to recognize and classify tokens based on their involvement in events, facilitating the automated extraction of important information from text. the model structure is designed to effectively leverage document and sentence-level information, alongside the main task of token classification, in a coherent manner. to achieve this, our model foot_0 predicts labels and trains on all three levels simultaneously, enabling comprehensive learning. inspired by scopeit @xcite , our multi-task architecture, illustrated in figure 1 , enables the creation of representations for tokens, sentences, and documents to then put these through the respective classification layers for each task. we build on their model structure by adding the facilities for the token classification task. so, our model trains on the two auxiliary tasks, document and sentence classification tasks, in addition to the primary token classification task. the model processes each sentence, with its split tokens, using a transformers-based encoder 2 to ob-127 tain representations for individual tokens. to address the limited input problem of the encoder, each sentence is processed independently. within each sentence, a bidirectional gated recurrent unit (bi-gru) @xcite , dubbed intra-sentence bi-gru, is employed to further enhance token representations and generate a representation for the entire sentence by concatenating the last hidden states from both directions of the bi-gru. these sentence embeddings are further enriched with contextual information by passing them through a second bi-gru, named inter-sentence bi-gru. additionally, a single representation for the entire document is obtained by concatenating the last hidden states from the inter-sentence bi-gru in both directions. each representation, whether for tokens, sentences, or documents, is then passed through their respective classification layers to calculate the corresponding losses. the document and sentence tasks employ binary cross-entropy loss, while the token task utilizes categorical cross-entropy loss. the losses from each task are combined, yielding a final loss value for backpropagation @xcite . it is important to note that we maintain a consistent model structure across all our experiments, even if document or sentence loss is not calculated in certain scenarios. this ensures a standardized approach and facilitates fair comparisons across different variations of the model. ## data we leverage the data provided by the acl case 2021 shared task @xcite , which focuses on detecting protest events in the languages english, portuguese, spanish, and hindi. the shared task encompasses four sequential steps, representing different stages of a real-world event extraction pipeline @xcite . for our experiments, we specifically utilize a subset of the english training data from subtask four, along with the corresponding english test data. the training data consists of 717 tokenannotated documents. these annotations were distributed in bio format, meaning there are no overlapping labels for any individual token, effectively turning this task into token classification. the test set, which remains the same across all experiments since token classification serves as the primary task, includes 179 token-annotated documents. the distribution of token labels for the training and test sets is outlined in ## experimental setup aside from the baseline, we conducted three main sets of experiments to address three key research questions, with each subsequent set incorporating additional data. in the first set, we utilized the inherent coarse-grained data available in tokenannotated documents. in the second set, we introduced negative documents to balance the positive ones and further explored the effects of the document classification task. in the third set, we removed some of the 717 documents to be used as extra coarse-grained data without token annotations. for each experiment set, we conducted three experiments based on different combinations of losses in addition to the token classification loss: only sentence classification loss (variation 1), only document classification loss (variation 2), and both sentence and document classification losses (variation 3). this approach allowed us to assess the individual effects of each auxiliary task introduced. although some weights of the model may not update in certain cases due to the architecture, we maintained the same model for all experiments to ensure fair comparisons. each experiment was run three times to calculate average performance and standard deviation scores. additionally, we gradually decreased the amount of data in each experiment to measure the influence of data size on model performance. listed below are the parameters employed for our model. it's important to note that no parameterspecific experiments were conducted to fine-tune these values. they remain consistent throughout all experiments, thereby minimizing the potential impact of parameter variations. the selection of these parameters was driven by pragmatic considerations, encompassing factors such as data size, gpu capacity, and practical feasibility. the parameter settings are as follows: as for the baseline, our model was trained using the 717 span-annotated documents. it's important to note that for the baseline model, the inter-sentence bi-gru and mlps for document and sentence classification did not train, as we solely utilized the loss for the primary task. however, the same model structure was retained to facilitate a fair comparison. to evaluate our experiments, we use a python implementation foot_2 of the original foot_3 conlleval evaluation script, which we simply refer to as the f1 score. ## conclusion and future work in this study, we addressed the challenge of data scarcity in closed-domain event extraction, a common hurdle in complex nlp tasks. through our proposed multi-task model structure and training approach, we successfully leveraged additional data generated during the token annotation process. the inclusion of this supplementary data, particularly negative documents without event information, proved to be crucial in enhancing the performance and robustness of our event extraction model. our experiments demonstrated that introducing extra coarse-grained data, which identifies documents and sentences without events, significantly contributed to performance improvements. the integration of document and sentence classification tasks alongside token classification did not yield noticeable benefits on their own, reaffirming that the internal representations of our model already encompassed essential information for coarser tasks. remarkably, even in scenarios where only a portion of the data was token-annotated, the model's performance remained comparable to situations with complete token annotations. we observed a clear trend of increasing performance gains as the token-annotated data size decreased and the extra coarse-grained data size increased. this trend was further reinforced when examining few-shot learning settings, where leveraging coarse-grained data notably enhanced the model's robustness even with minimal data sizes. in conclusion, our findings offer promising insights into mitigating data scarcity challenges in closed-domain event extraction by effectively utilizing extra data obtained during the annotation process. this practical solution opens the door to more robust and efficient event extraction systems across various domains, with implications for knowledge extraction and decision-making processes. we utilized gold-standard data throughout all our experiments. we will be investigating the possible usage of silver coarse-grained data, which does not even require the considerable ease of labeling documents or sentences. we also plan to include more event information extraction data sets to test our hypothesis further.