Deep Learning in Event Detection in P olish (2019)

Abstract: Event detection is an important NLP task that has been only recently tackled in the context of Polish, mostly due to lack of language resources. The available annotated corpora are still relatively small and supervised learning approaches are limited by the size of training datasets. Event detection tools are very much needed, as they can be used to annotate more language resources automatically and to improve the accuracy of other NLP tasks, which rely on the detection of events, such as question answering or machine translation. In this paper we present a deep learning based approach to this task, which proved to capture the knowledge contained in the training data most effectively and outperform previously proposed methods. We show a direct comparison to previously published results, using the same data and experimental setup.

## introduction the task of identifying events in natural language has a direct impact on the effectiveness of many other tasks in the area of natural language processing. an obvious example is the task of question answering, where the knowledge base has the form of a collection of texts in natural language @xcite . the answer to the question when was the current president elected? requires recognition of the current system time, determining who the current president (of poland, by implicit assumption) is and identifying the event of election. other nlp tasks directly influenced by the results of event detection include summarization @xcite , @xcite , @xcite and machine translation @xcite . in the first case, the events identified in the text allow organizing the content of the summarized document by topics and ordering them chronologically. in the case of machine translation, event detection may be used to create the intermediate knowledge representation layer that is independent of any natural language, which is then used to form the final translation. in the case of the polish language, there are only a few published papers on the identification of temporal expressions in natural language text. this is largely due to the current lack of resources, enabling this type of study. for example the authors of @xcite use parallel corpora and annotation projection to polish to gather the necessary evaluation material. they use the national corpus of polish @xcite , which contains the basic annotation of simple temporal expressions. specifically, the manually annotated subcorpus of the ncp includes such tags as: date (calendar dates, such as 24 october, 1945) and time (hours, minutes and seconds, e.g. five after twelve). the recently published subcorpus of the kpwr corpus @xcite has been specifically annotated with temporal expressions and events, using an adaptation of the timeml specification @xcite . this collection of annotated texts along with additional dictionaries has been used in @xcite to train a crf-based classifier for the task of identifying events. ## event detection task we define the task of detecting events in text as a problem of identifying tokens or token sequences, which should be annotated as an event mention according to the timeml specification, adapted to polish by @xcite . as in the original timeml specification, we understand events as situations that happen or occur, an "event is anything that takes place in time (date, time and/or duration) and space (has a location), may involve agents (executor or participants), may contain or be part of other events and may produce some outcome (object)." @xcite . we aim to classify identified events into one of the following categories, defined by the specification: the goal of the task is thus to create an annotation layer, which associates event category labels with corresponding tokens. below is an example annotation, taken from the training corpus (other annotation layers not shown here for readability): ## deep learning approach to event detection preprocessing in the first stage of the proposed method we preprocess the available data and generate feature vectors for the neural network. we scan through the text using a fixed-length processing window: for each token in a sentence a sequence composed of this token (in the center of the window) and its w nearest neighbors within the sentence is generated. thus, the sequence has a length of 2w + 1, where w is called the window size. the neural network takes as an input a sequence of feature vectors of individual tokens and classifies the central token into one of previously described categories, with an additional not event class for not relevant tokens. features we use two kinds of embeddings for the real-valued feature vector generation: in the case of word sense ambiguity during generation of plwordnet-based features (several matching synonyms or hypernyms), the first base form common to all synonyms from all matching synsets is chosen (in alphabetically sorted order). if there is no common base form, or there is no match, the base form of the original token is selected. word2vec embedding models were trained in two main steps: word2vec feature vectors are assigned to individual tokens by computing given feature value (lemma, hypernym etc.), which then is directly mapped to corresponding feature vector. eg. ludzie -> człowiek -> [feature vector]. the input vector of an individual token is a concatenation of all component feature vectors. the size of the input vector of the individual token in a sequence with all described features included was 1138 elements. network architecture based on preliminary experiments (described in the experimental results section), we have chosen a network consisting of two distinct subnetworks as the most promising for further experiments. the network is split into two branches, bi-lstm and bi-gru subnetworks. each of these subnetworks takes the same input, but with a different random dropout applied to it. bi-lstm and bi-gru can simultaneously model word representation with its preceding and following information. they are composed of two lstm/gru neural networks with a hyperbolic (tanh) and hard sigmoid activation functions. the forward lstm/gru allows to model the preceding contexts, and a backward lstm/gru to model the following contexts respectively. we flatten and concatenate the bidirectional sequence features learned from the subnetworks and apply random dropout to the result. then, we use a dense softmax approach to perform final classification. the architecture of the network has been presented on figure 1 . we train our model with categorical cross-entropy loss function and adam optimizer (kingma and ba, 2014) with small learning rate decay. for gru and lstm we use glorot @xcite and orthogonal @xcite initializers. ## conclusions and future work in this paper we have applied a deep-learning approach to the problem of detecting events in text. as in many other nlp tasks, modern neural networks proved to perform very well in this domain and outperformed the previously proposed method, which was based on conditional random fields. @xcite . it is interesting to note that features based on words (orth, lemma in figure 2 ) influenced the resulting accuracy the most, proving that such embeddings carry essential information for this task. on the other hand, features based on partof-speech tags (ctag, class) were among the least informative. a characteristic feature of processing inflected languages is the importance of lemmatization and including lemmas in the feature set. the large number of inflected word forms in languages such as polish (and other slavic languages), makes it more difficult for word-form embeddings to capture information that is properly generalized. generating embeddings from lem-mas helps to solve the problem, as long as the lemmatization is accurate and does not introduce additional disambiguation difficulties. in future work we would like to tackle a more general task of event recognition in text, including the identification of textual arguments of events. this may include such entities as place names (where the event takes place), time and date specifications (when it takes place), or person names (agents or beneficiaries of an event). we would also like to analyze the relationships occurring between several events recognized in a text fragment (e.g. event identity).