Metric Learning in Multilingual Sentence Similarity Measurement for Document Alignment (2021)

Abstract: Document alignment techniques based on multilingual sentence representations have recently shown state of the art results. However, these techniques rely on unsupervised distance measurement techniques, which cannot be fined-tuned to the task at hand. In this paper, instead of these unsupervised distance measurement techniques, we employ Metric Learning to derive task-specific distance measurements. These measurements are supervised, meaning that the distance measurement metric is trained using a parallel dataset. Using a dataset belonging to English, Sinhala, and Tamil, which belong to three different language families, we show that these task-specific supervised distance learning metrics outperform their unsupervised counterparts, for document alignment.

## introduction document alignment is an important precursor to build parallel corpora from noisy data sources. document alignment is also useful in multilingual information retrieval, as well as for tasks such as false news detection across different languages. traditionally, document alignment has been achieved by metadata-based methods @xcite and translation based methods @xcite . metadata-based methods rely on exploiting the meta information of the selected data sources, which may be inconsistent across different sources. on the other hand, translation based methods assume the availability of a machine translation (mt) system. to overcome these issues, recent research has exploited the use of multilingual sentence representations (multilingual sentence embeddings) such as laser @xcite foot_0 . here, vector representations are derived for documents in both source and target languages. then, for a given document in the source side, the most similar counterpart is identified from the target side. euclidean distance and cosine distance are used in existing document alignment systems @xcite . however, these similarity metrics cannot be fine-tuned for the selected task or data at hand. the alternative is to use metric learning, which focuses on constructing a problem-specific distance metric automatically from data @xcite . metric learning-based distance measurement techniques have been successfully employed in image classification and image identification tasks @xcite . in this paper, we apply two such metric learning algorithms on multilingual sentence representations to identify similar document pairs in a bilingual setting. we experimented with sparse high-dimensional metric learning (sdml) @xcite , and information theoretic metric learning (itml) @xcite algorithms. to the best of our knowledge, this is the first work to exploit the use of metric learning with respect to multilingual sentence representations for the purpose of document alignment. el-kishky and guzmán (2020)'s system was used as the baseline. this research can be considered as the best in this line of research at the moment. they used sentence embeddings using the laser toolkit. they used euclidean similarity as the sentence similarity measurement. weighted sentence similarity scores were aggregated to derive the document similarity. in this work, we replaced euclidean distance measurement with metric learning based distance measurement. we used a dataset that contains documents crawled from news websites belonging to three languages sinhala (si), tamil (ta), and english (en), which belong to three different language families (indo-aryan, dravidian, and indo-european, respectively) @xcite . note that sinhala is a low-resource language. in the recent language categorization by @xcite , sinhala belongs to class 0, meaning that it has exceptionally limited resources. tamil is considered as a medium-resourced language. it is categorised as 3, meaning that it has a strong web presence, and a cultural community that backs it. we separately report results for the three language pairs, en-si, en-ta and si-ta. for all the document alignment tasks except one news dataset in en-si, metric learning based distance measurement performed better than the unsupervised distance measurement techniques. to train the metric learning models, a very small parallel corpus of 5000 words were used. further experiments showed that the content (i.e. the domain) in the parallel corpus has minimal impact on the performance of metric learning models. the above results were obtained using laser embeddings, to be a fair comparison with @xcite . we experimented with xlm-r multilingual embeddings @xcite foot_1 as well, using the itml metric learning model. since xml-r showed superior performance over laser, we conducted extensive experiments where the itml model was built using different amounts of parallel data with respect to xlm-r. our results show that the itml model trained with xlm-r embeddings works equally well even with 1000 parallel sentences. our code is publicly released foot_2 . ## metric learning unsupervised metrics such as euclidean or cosine are commonly used to calculate the distance between two embeddings or vectors. this is the same for sentence embeddings. however, these unsupervised metrics cannot be optimised for the particular data set or the task. in contrast, metric learning uses a supervised algorithm to learn the best distance measurement metrics that are task specific, using the given data. metric learning has been successfully used for tasks such as pattern recognition and face identification in the field of image classification @xcite @xcite . all the metric learning algorithms use mahalanobis distance as the distance metric. mahalanobis distance between two points x, x is defined as in equation 1 , ## data set we used the dataset presented by @xcite . this dataset contains news articles belonging to four news websites published in english, tamil and sinhala languages. from each of the websites, around 2000 documents (per language) were selected based on the published date of the articles. ## experiments for our experiments we used the python implementation of the two metric learning algorithms @xcite . the first experiment was to determine whether metric learning based distance measurement could outperform the unsupervised counterpart. to be a fair comparison, we used the laser embedding model as used by @xcite . euclidean and cosine unsupervised distance metrics, as well as itml and sdml metric learning based distance metrics were calculated for both sl and idf weighting schemes. 5000 parallel sentences from @xcite 's parallel corpora were used to train the metric learning models for en-ta and si-ta. for en-si, sdml and itml were trained with sl weighting for fernando et al. (2020)'s parallel corpus and the open subtitle corpus. idf was trained with the latter corpus only. this is to measure the impact of the type of parallel corpus on the metric learning models. we conducted another experiment to compare the performance of laser and xlm-r multilingual embedding models. first, the same 5000 sentence pairs from @xcite ## results we followed the method used for document alignment evaluation in wmt16 document alignment shared task @xcite . the ground truth documents only contain a small fraction of parallel documents. there can be many more valid cross lingual document pairs in the dataset. the results confirm that metric learning based distance measurement has been very effective in aligning documents for all three language pairs. even for the newsfirst data source for the en-si pair, metric learning (sdml) lags behind cosine similarity only by a very small margin. it is evident that xlm-r significantly outperforms laser. xlm-r performing very well even for 1000 parallel sentence is very promising -this means that metric learning could be employed with languages included in the xlm-r model as long they have a very small parallel corpus. another factor we wanted to investigate is the impact of the level of language representation in the multilingual embedding model on document alignment. although laser and xlm-r include data from multiple languages, they do not include data from all the languages in equal amounts. if we hypothesize that the amount of language data is proportional to the language categorization proposed by @xcite , results related to sinhala has to be the lowest. however, the results do not convey this message. thus we believe that factors such as language relatedness and language complexity might be playing a role here. ## conclusion this paper presented the use of supervised distance measurement techniques on multilingual sentence embedding based document similarity calculation. results show that this supervised approach is being able to consistently outperform the unsupervised counterparts for three document alignment tasks. only a small parallel corpus is required to train the metric learning distance measurements. thus, these techniques can be employed with respect to low-resource languages as well. the main drawback of metric learning algorithms is that they are very much computationally expensive. therefore, in the future, we plan to optimize these algorithms to achieve better efficiency.