Data Selection for Bilingual Lexicon Induction from Specialized Comparable Corpora (2020)

Abstract: Narrow specialized comparable corpora are often small in size. This particularity makes it difficult to build efficient models to acquire translation equivalents, especially for less frequent and rare words. One way to overcome this issue is to enrich the specialized corpora with out-of-domain resources. Although some recent studies have shown improvements using data augmentation, the enrichment method was roughly conducted by adding out-of-domain data with no particular attention given to how to enrich words and how to do it optimally. In this paper, we contrast several data selection techniques to improve bilingual lexicon induction from specialized comparable corpora. We first apply two well-established data selection techniques often used in machine translation that is: Tf-Idf and cross entropy. Then, we propose to exploit BERT for data selection. Overall, all the proposed techniques improve the quality of the extracted bilingual lexicons by a large margin. The best performing model is the cross entropy, obtaining a gain of about 4 points in MAP while decreasing computation time by a factor of 10.

## introduction mining translation equivalents to automate the process of generating and extending dictionaries from bilingual corpora is a well known natural language processing (nlp) application. initially conducted on parallel data, this modus operandi has rapidly moved to using comparable corpora, mainly due to their availability and the fact that they are easier to acquire @xcite . comparable corpora gather texts of the same domain, often over the same period without being in a translation relation. journalistic or encyclopedic websites as well as web crawl data are usually popular for this purpose @xcite @xcite . in technical and scientific domains, comparable corpora suffer from their modest and limited size compared to general language corpora. this phenomenon is amplified by the difficulty to obtain many specialized documents in a language other than english @xcite . this aspect, which is gradually changing with the deployment of open access data, is nevertheless a major difficulty. one way to overcome the limited size of specialized (in-domain) comparable corpora for bilingual lexicon induction (bli) is to associate them with out-of-domain resources as lexical databases @xcite or large general domain corpora @xcite . for instance, by combining a specialized comparable corpus with a general domain corpus, methods based on distributional analysis are boosted @xcite . general domain corpora greatly enhance the representation of specialized vocabulary by adding new contexts. the main drawbacks of this data augmentation approach is the introduction of polysemous information as well as the tremendous increase of computation time. consider for instance, a french/english comparable corpus in the medical domain and the french terms os (bone) and sein (breast). when looking for additional contexts in general corpora, os is very likely to recover contexts dedicated to an operating system since os is the abbreviation used in french. similarly, when studying the french term sein, many contexts related to the french preposition au sein de (within) will be added. in the same way, the english medical term breast can be associated with food just as in chicken breast to mention only the most obvious ones. associating out-of-domain data with a specialized corpus is not the mainstream research direction in bilingual lexicon induction from specialized comparable corpora. in computational terminology it is generally accepted that the specialized corpus conveys knowledge of the domain. in the same way, polysemy is often considered as a marginal phenomenon in technical and scientific domains. in this sense, associating out-of-domain data with a specialized comparable corpus can be seen as a "heresy" or, as in the previous examples, an unfortunate way to introduce polysemy. in the context of statistical machine translation (smt), @xcite demonstrated that adding out-of-domain data to the training material of a system was detrimental when translating scientific and technical domains. instead of using a data augmentation strategy, data selection is proposed to improve the quality of smt systems @xcite @xcite . the basic idea is that in-domain training data can be enriched with suitable sub-parts of out-of-domain data. within this context, we apply two well-established data selection techniques often used in machine translation that is: tf-idf and cross entropy. we also propose for the first time to exploit bert for data selection. we show that a subtle selection of the contexts of out-of-domain data allows to improve the representation of specialized domains, while preventing the introduction of polysemy induced by data augmentation. overall, all the proposed techniques improve the quality of the extracted bilingual lexicons by a large margin. the best performing model is the cross entropy, obtaining a gain of about 4 points in map while decreasing computation time by a factor of 10. ## related work the historical distributional approach @xcite , known as the standard bag of words approach, builds a context vector for each word of the source and the target languages, translates the source context vectors into the target language using a bilingual seed lexicon, and compares the translated context vectors to each target context vector using a similarity measure. while this approach gives interesting results for large general comparable corpora @xcite @xcite , it is rather unsuitable for small specialized comparable corpora due to the context vectors sparsity. @xcite @xcite . more recent distributed approaches, based on deep neural network models @xcite , have come to renew traditional ones. using these approaches, words are embedded into a low-dimensional vector space. for instance, @xcite proposed an approach to learn a linear transformation from the source language into the target language. @xcite used the canonical correlation analysis to project the embeddings in both languages into a shared vector space. @xcite proposed an orthogonal transformation based on normalized word vectors to improve the inconsistency among the objective function used to learn the word vectors and to learn the linear transformation matrix. more recently, @xcite proposed and then improved an approach that generalizes previous works in order to preserve monolingual invariance using several meaningful and intuitive constraints (i.e. orthogonality, vector length normalization, mean centering, whitening, etc.). finally, @xcite and @xcite proposed unsupervised mapping methods getting results close to supervised methods. a thorough comparison of cross-lingual word embeddings has been proposed by @xcite . a comparison of the approaches of @xcite and @xcite with the standard bag of words approach was carried out by @xcite . the authors showed that embedding approaches perform well when the terms to be translated occur very frequently while the standard approach is slightly better when the terms are less frequent. on specialized domains, @xcite compared different approaches and observed that the one described by @xcite (an enhanced variant of the skip-gram and c-bow models) outperformed the others. recently, @xcite proposed a deep contextualized word representation that improves the state-of-the-art across different challenging nlp tasks. this representation is useful to model different types of syntactic and semantic information about words-in-context. it is thus possible from a general language corpus to have different embeddings for a given word, with each of these embeddings reflecting one of the word's meanings. in specialized domains, it is widely accepted that words can have only one meaning even if their contexts may convey different meanings in a general language corpus. el @xcite showed that using a combination of out-of-domain contextualized word representation (elmo) with small in-domain uncontextualized word representation (word2vec) does not improve the results while compared to models trained on large in-domain corpora. for this reason, we focused this study on uncontextualized word embeddings (fasttext). ## data selection techniques while abruptly adding out-of-domain data to a specialized corpus, with no further thoughts on how to do it improves the results, the computational time is greatly increased and polysemous information is introduced. to leverage the aforementioned issues, we present in this section different data selection techniques that we use in our experiments. tf-idf is the term frequency-inverse document frequency score and is performed at the document level. it is computed for both in-domain and out-of-domain vocabularies, then the out-of-domain documents are ranked by measuring their cosine similarity with the in-domain corpus. given an indomain document d i and an out-of-domain document d o , the similarity is computed as follows: cross entropy is used as defined in @xcite to select out-of-domain data that is close to the in-domain corpus. conversely to the tf-idf technique, the data selection is performed at the sentence level. given the in-domain corpus i, a language model is first computed for the source side (lm i,s ) and the target side (lm i,t ) of the bilingual corpus. similarly, we compute a language model for the out-of-domain corpus o (lm o,s and lm o,t ) which is drawn from a random sample of the same size of the specialized corpus. the cross entropy is computed as follows: bert is a supervised learning model that has proven to be efficient in many downstream nlp tasks @xcite . it has been trained on the masked language model (mlm) and next sentence prediction (nsp) objectives @xcite . hence, it can be used whether for word representation or for sentence classification. in order to perform data selection, we use bert as a binary classifier to predict if a given input sentence is in-domain or out-of-domain. the intuition behind this strategy is that bert can learn shared features between in-domain sentences. for each positive (in-domain) training sentence, we randomly select a negative sentence from a general domain data set to keep a balanced training set. class balancing is often a vital setup for classifiers' efficiency. even if some existing techniques do reduce the impact of unbalanced classes, we only deal with balanced training in this work. random data selection we also apply a random data selection at the sentence level, where we shuffle randomly the sentences of the out-of-domain corpora. ## experimental protocol we present in this section the used data that is: the comparable corpora, the seed lexicon, the reference lists and the methodology applied for our experiments. ## results in this section we present the results obtained for bli on our corpora as well as the differences in computation time for each method. ## conclusion in order to improve bli from specialized comparable corpora, we have studied in this paper the impact of common data selection techniques such as crossentropy, t f -idf and also applied bert for the first time as a data selection technique. the experiments on two specialized comparable corpora and two general corpora revealed that selecting a small and adapted amount of out-of-domain data is sufficient to obtain better results (map of 87.6% for bc and 80.9% for we) than a high amount of data (resp. 83.9% and 75.9%) while reducing computation time by a factor 10. the data selection strategy also allowed to reduce the impact of polysemy compared to data augmentation. this strategy shows that bringing new occurrences for both the word to translate and the general vocabulary improves their mutual discrimination. an interesting perspective could be to select for each word to translate the adequate amount of out-of-domain data instead of a common selection for all the terms to translate.