Incorporating Sub-Word Level Information in Language Invariant Neural Event Detection (2019)

Abstract: Detection of TimeML events in text have traditionally been done on corpora such as TimeBanks. However, deep learning methods have not been applied to these corpora, because these datasets seldom contain more than 10,000 event mentions. Traditional architectures revolve around highly feature engineered, language specific statistical models. In this paper, we present a Language Invariant Neural Event Detection (ALINED) architecture. ALINED uses an aggregation of both sub-word level features as well as lexical and structural information. This is achieved by combining convolution over character embeddings, with recurrent layers over contextual word embeddings. We find that our model extracts relevant features for event span identification without relying on language specific features. We compare the performance of our language invariant model to the current state-of-the-art in English, Spanish, Italian and French. We outperform the F1-score of the state of the art in English by 1.65 points. We achieve F1-scores of 84.96, 80.87 and 74.81 on Spanish, Italian and French respectively which is comparable to the current states of the art for these languages. We also introduce the automatic annotation of events in Hindi, a low resource language, with an F1-Score of 77.13.

## introduction automatic extraction of events has gained sizable attention in subfields of nlp and information retrieval such as automatic summarization, question answering and knowledge graph embeddings @xcite , as events are a representation of temporal information and sequences in text. various developments in guidelines and datasets for event detection have been met with equally fast paced evolution of automatic event annotation and detection methodologies in the last few years @xcite @xcite . on a larger scale, event extraction has extended to many languages beyond english, including french @xcite , spanish @xcite , italian @xcite ) and very recently, hindi @xcite . event detection architectures have their origins in statistical models such as k-means and hierarchical clustering methods @xcite , which have more recently given way to neural models. deep neural architectures on event annotation vary based on the approach taken to identifying and handling the data. however, event detection as a problem shifts when we move away from the annotation paradigm of datasets such as ace @xcite and tac kbp @xcite to timeml datasets such as timebank @xcite , which are used in this paper. there has been limited use of deep learning methods on timebanks due to fewer event mentions and a need for data augmentation and bootstrapping. however, in this paper, we show that using subword level information, a language invariant deep learning model can provide similar event detection accuracies as heavily feature engineered language specific statistical methods without using any augmented data. this paper has two main contributions. first, we introduce our model, the architecture for language invariant neural event detection (alined), which is a deep learning model for event extraction from timeml event annotated datasets from five languages. we show that for four of these languages, using no augmented data, we achieve comparable f1 score on these datasets to heavily feature engineered language specific statistical models, with less than 12,000 event mentions in each. secondly, to the best of our knowledge, we present the first ever baseline for neural event detection in hindi using this model. our architecture uses both word and character embeddings and captures information from them distinctly, before combining them into a coherent representation of both. this is then used to determine the label for each input word. the proposed architecture is language invariant as well, such that no part of the system undergoes a change when training on different languages. in presenting this architecture, we highlight the importance of using subword level information in order to incorporate morphological as well as syntactic features in event extraction. this can also be extended to other semantically oriented sequence labeling tasks 2 related work neural approaches to sequence tagging are common due to extensive developments in named entity recognition. @xcite introduced and cultivated the use of bidirectional lstms to incorporate features that could be used for sequence tagging using a crf. ma and hovy (2016)'s architecture and the neuroner program @xcite provided a basic architecture and influenced multiple developments to most sequence labeling tasks, including event @xcite . the task of event extraction in any language involves the identification of the event nugget @xcite . prominent work has been done to analyze the lexical and semantic features of event representation @xcite , which served as a basis for neural event nugget detection @xcite . the task of neural event detection has been attempted using a combination of networks, but mostly revolving around the use of convolutional neural architectures. work in this approach focused on various aspects such as max-pooling to retrieve the structure of event nugget information @xcite , modeling the skipgram architecture to learn lexical feature representations @xcite as well as using dynamic cnns in order to extract lexical and syntactic features in parallel @xcite . recurrent neural architectures have also been employed for this task, which predict the location of the trigger based on combining the for-ward and backward features of sentences in which events occur @xcite . note that in both cases architectures focused on dealing with structural, lexical and contextual features. in the domain of multi-lingual and cross lingual event detection, @xcite uses a combination of both lstms and cnns for creating a language independent architecture for capturing events, while @xcite used stacked rnns for sequence labeling and a language discriminator to learn language features. the latter architecture implements the use of the character embeddings, but does not identify the relevant features independent of the word embeddings. ## model description in this section, we describe the alined model for the event detection. primarily, we focus on how to capture event representation at both a character and a word level. in this model, we had to focus on the following major considerations: fundamentally, our architecture generates character embeddings through convolution and aggregates this information using bidirectional lstms @xcite . the same is done over pretrained word embeddings in parallel, creating distinct intermediate representations. these representations are combined using a highway architecture for a final representation, which is used for the sequence tagging task. ## results and analysis in this section, we analyze the results of the alined model, and compare them to the current state of the art systems for the various languages we train on. we also provide a rigorous error analysis of our system and methodology. since no single system has compared work in event detection across the five languages that we have chosen for the experiments here, we draw comparisons to the various systems that trained on the individual or group of languages that have been used. italian tipsemit basic @xcite 90.00 77.00 83.00 tipsemit fpc5 @xcite 89.00 81.00 85.00 tipsemit fpc5sem @xcite 91.00 83.00 87.00 the system is a modification of the tipsem system. we compares our models to their reported scores. however, the corpus used in @xcite is the ita-timebank which has been augmented with further annotations and resources, while our system uses just the ita-timebank for event extraction. in most comparisons, our models perform equally well or better than the current systems for each of the above languages. we do not annotate or augment any of our data sources for using this model, so the reference corpora are being trained and tested upon, which are mentioned in section 4.1. the calculation of the metrics of comparison, precision, recall and accuracy are calculated as follows: we note a lower precision score in case of english and spanish, as the number of false positives are slightly higher. we attribute this difference to the fact that due to the combination of sub-word level features, the model seems to sometimes "spill over" the boundary of single word or nominal. however, higher recall implies that there are fewer false negatives, meaning the model more accurately identifies those words which are in the event span. more labeled data would be very useful in learning the span boundaries, especially for nominal events, as the network would have more samples to learn the variations in the methods of event representation. for english, surprisingly, we see that an increase in the f1-scores. we attribute this to a combination of factors, including well defined verbal affixes which are attributed to events, and effective weighted combination of character and word embeddings. for italian, we train and test solely on the ita-timebank, whereas the current state of the art system trained on an augmented ita-timebank @xcite , which was enriched with more labeled data. similarly, in french, we use the established french timebank, while experiments in french so far have been on self-annotated @xcite or timeml corpora @xcite . since these repositories of augmented data were not available to us at the time of writing this paper, the values reflect the same. however, it is to be noted that our system does provide an accuracy that is close to the currently reported stateof-the-art even in the absence of language specific features, explaining the fact that sub-word information is necessary for event detection in italian and french as well. for hindi, our architecture provides a good baseline. however, the training data consists of far too many words that are out of vocabulary, which is a major issue in working with word embeddings. while the concatenation of sub-word information mitigates this, a system focused on a better representation of out of vocabulary words would significantly help the network. however, this required a larger labeled corpus as well, which makes this a challenge as hindi is a low-resource language in terms of corpora for event detection and extraction. ## conclusion in this paper, we show the development of alined, a language invariant neural sequence tagging architecture for event detection in five different languages, namely, english, spanish, italian, french and hindi. we develop insight into the use of sub-word level information and combining it effectively. with the lexical and syntactic infor-mation. for our training and testing, we use only established corpora, which have not been augmented or changed in any way. we perform almost at par or better then the current state of the art in all the languages we train in. we establish a new best f-score for event extraction in english. we also establish the baseline for training and testing on the french timebank and for event extraction as a task in hindi. our model has been thoroughly error-analyzed, which we have explained based on the comparison of system output and expected tags. given the nature of our results, we aim to establish the importance of sub-word level information in event detection. further work in this task could be done by providing augmented reference corpora, so that problems based on lack of labeled data do not limit further research in this topic. this could also be tackled by effectively introducing transfer learning to neural event detection, where the model learns the representation of events irrespective of language, while accounting for sub-word, lexical and structural information.