Multi-linguality helps: Event-Argument Extraction for Disaster Domain in Cross-lingual and Multi-lingual setting (2019)

Abstract: Automatic extraction of disaster-related events and their arguments from natural language text is vital for building a decision support system for crisis management. Event extraction from various news sources is a well-explored area for this objective. However, extracting events alone, without any context, provides only partial help for this purpose. Extracting related arguments like Time, Place, Casualties, etc., provides a complete picture of the disaster event. In this paper, we create a disaster domain dataset in Hindi by annotating disaster-related event and arguments. We also obtain equivalent datasets for Bengali and English from a collaboration. We build a multi-lingual deep learning model for argument extraction in all the three languages. We also compare our multi-lingual system with a similar baseline mono-lingual system trained for each language separately. It is observed that a single multi-lingual system is able to compensate for lack of training data, by using joint training of dataset from different languages in shared space, thus giving a better overall result.

## introduction the ability to extract real time news of disaster events automatically, can potentially help in better decision-making for planning and coordination of disaster relief efforts. event extraction from text entails the extraction of particular types of events along with their arguments. information obtained from extracted event mentions provides a more structured and clear picture when augmented with related arguments like time, place, participant, casualty etc. in a language rich world where each event is documented in multiple languages, argument extraction in multi-lingual setting stands as a crucial task. extraction of events from news is a well explored area in natural language processing. com-petitions such as ace2005 @xcite and tac-kbp2015 @xcite have investigated the area and provided a large body of literature on event extraction from news articles. event extraction was done on ace2005 dataset by @xcite by combining global evidence from related documents with local decisions. @xcite introduced a method of event argument extraction based on crfs model for ace 2005 chinese event corpus. event and its arguments were extracted by @xcite , for the purpose of extracting breaking news. although extraction of events is quite well examined, there is a scarcity of work in extraction of detailed arguments for disaster domain like casualties, reason, after-effects etc. in this paper we create and publish a dataset annotated for events in disaster domain, for three different languages, i). hindi, ii). bengali and iii). english. this dataset is annotated for the task of argument extraction by expert annotators. we build a 'mono-lingual' deep learning system, based on cnn (convolutional neural network) and bi-lstm (bi-directional long short term memory) for the task of argument extraction. in order to leverage the information from all the languages while training, and improve the performance of the system, we build a 'multi-lingual' argument extraction system. this is done by adding separate language layers for each language to our 'mono-lingual' system. to bring the datasets of all the languages to the same vector space, we make use of 'multi-lingual' word embeddings. we show that by training our model in this way we are able to utilize the dataset of all the three languages and improve the performance of our system for most arguments in the three languages. we also investigate how the syntactic difference of the languages is handled by our system. through analysis, we show that 'multi-lingual' training is espe-cially helpful in improving the performance when some argument is under-represented in the 'monolingual' training data. ## methodology in this paper, we propose that joint training of ie system on different language datasets, using 'multi-lingual' word embeddings and language layers helps in better extraction of arguments. this is particularly true when the dataset is limited in size. to corroborate our claim, we device two different systems, i). monolingual baseline system, and ii). multi-lingual system. the 'monolingual baseline' system only takes input data (sentence wise) from one language and extracts the arguments. for word representation, it uses monolingual word embeddings. the 'multi-lingual' argument extraction system uses separate language layers and multi-lingual word embeddings for joint training on all the three languages. ## dataset and experiments in this section, we describe the dataset used and the experiments conducted. ## results and analysis in this section, we discuss the results obtained for the two experiments described in section 4.2. we also provide analysis of the results. f1-score is used as an evaluation metric, and all the results reported are 5-fold cross-validated. the results for both, 'monolingual' and 'cross-lingual' experiments are reported in we also test the statistical significance of each increment in f1-score for argument extraction. the 'p-values' obtained after 't-test' are shown in it is observed that multi-word time arguments are better captured by 'multi-lingual' model than by the 'monolingual baseline' model. an example of this can be seen in the following sentence: in the aforementioned sentence the actual phrase denoting time is 'र ववार रात' (sunday night). however the 'monolingual' model only detects 'र ववार' another interesting observation is that, for casualty argument of english dataset, the 'monolingual' system often confuses people as casualties, even when they are not. an example of such observation is as follows: in the above example the phrase '200000 people' does not denote casualty, however the 'monolingual' model confuses it as casualty. this is due to the lack of training data in english to learn the difference between some count of people and actual casualty. however, after 'multi-lingual' train-ing the model is able to make this distinction correctly. the f1-score for place arguments for all the datasets, is better for the 'monolingual baseline' model. this is because place argument is present in good numbers for all the datasets, therefore there are enough instances for proper training of deep learning model, even in monolingual setting. using 'multi-lingual model' for such cases is of little help. furthermore, the syntactic difference between languages confuses the system, thus degrading the performance of the 'multi-lingual' system. a good example of this phenomenon is show below: it can be observed that the 'monolingual baseline' model predicts the entire phrase describing the place argument correctly. however the prediction by 'multi-lingual model' misses the preposition 'in', which is present between 'naraingarh' and 'ambala'. the same sentence can be written in bengali as follows: the phrase 'in ambala' is represented by a single word 'ambālāra', in bengali. this difference in syntax between languages, makes the 'multilingual' system miss the word 'in' thus degrading the performance of the system. the best improvement in f1-score is observed for the arguments reason and after-effects for the english language. this is because these two arguments have least support in the dataset, and thus multi-lingual training helps by mitigating the scarcity in training examples. the same phenomenon can also be observed for reason argument which has a low support in hindi dataset. thus through our analysis we can conclude that, 'multi-lingual' training can help in improving the performance of the system for low support classes. however, it can also cause confusion and deteriorate the performance for high support classes. ## conclusion in this paper we create a dataset for argument extraction for disaster domain, for three languages hindi, bengali and english. we then build a deep learning model for extraction of these argument in each language separately. since the data is limited in size, we build another model that leverages data from all the languages. to make use of different language datasets, we first bring the word embeddings of all the three languages to the same vector space. we also use separate language layers to accommodate divergence in syntax of the languages. through our experiments we show that training in shared vector space by using 'multi-lingual' system helps in improving the performance of low support arguments. we also show that the for high support arguments, the syntactic difference in language can sometimes overcome the benefit of 'multi-lingual' training and cost in performance of our proposed 'multi-lingual' system. in future we would like to explore how to handle these syntactic differences so that the performance can be further improved. it would also be interesting to explore the range of languages that can be trained successfully in a multi-lingual setting.