Cross-Lingual Phrase Retrieval (2022)

Abstract: Cross-lingual retrieval aims to retrieve relevant text across languages. Current methods typically achieve cross-lingual retrieval by learning language-agnostic text representations in word or sentence level. However, how to learn phrase representations for cross-lingual phrase retrieval is still an open problem. In this paper, we propose , a cross-lingual phrase retriever that extracts phrase representations from unlabeled example sentences. Moreover, we create a large-scale cross-lingual phrase retrieval dataset, which contains 65K bilingual phrase pairs and 4.2M example sentences in 8 English-centric language pairs. Experimental results show that outperforms state-of-the-art baselines which utilize word-level or sentence-level representations. also shows impressive zero-shot transferability that enables the model to perform retrieval in an unseen language pair during training. Our dataset, code, and trained models are publicly available at github.com/cwszz/XPR/.

## introduction phrase retrieval aims to retrieve relevant phrases from a large phrase set, which is a critical part of information retrieval. recent studies on phrase retrieval learn dense representations of phrases, and achieve promising results in entity linking, slot filling, and open-domain question answering tasks @xcite . nonetheless, most of the studies focus on monolingual scenarios, leaving the cross-lingual phrase retrieval unexplored. various methods have been proposed to perform cross-lingual text retrieval, which learns crosslingual word or sentence representations shared across languages. cross-lingual word representation methods typically train word embeddings on each language separately, and then learn an embedding mapping between the embedding spaces of different languages @xcite . then, the bilingual word pairs can be retrieved between vocabularies using nearest neighbor search, which is also known as bilingual lexicon induction @xcite . cross-lingual sentence retrieval is typically achieved by learning a sentence encoder on multilingual text corpora with self-supervised pretraining tasks @xcite , or large-scale parallel corpora @xcite , or both @xcite . the trained sentence encoders produce language-agnostic sentence representations, which enables sentences to be retrieved across languages. despite the effectiveness of word-level and sentence-level methods, how to learn phrase representations for cross-lingual phrase retrieval is still an open problem. learning cross-lingual phrase representations is challenging in two aspects. first, a phrase is a conceptual unit containing multiple words, so it is necessary to model the interaction between words, which is not considered in word-level methods. second, a phrase contains fewer words with less information compared to sentences, which prevents sentence encoders from taking the advantage of the ability of understanding full-length sentences. thus, in this paper, we propose a novel crosslingual phrase retriever named as xpr. unlike previous cross-lingual retrieval methods that directly encode the input text, xpr produces phrase representations using example sentences, which can be collected from unlabeled text corpora. initialized with a pretrained cross-lingual language model, xpr can either directly serve as an unsupervised retriever, or be further trained to produce better-aligned phrase representations. besides, we propose the cross-lingual phrase contrast (xpco) loss for training xpr, where the model is trained to distinguish bilingual phrase pairs from negative examples. furthermore, we create a cross-lingual phrase retrieval dataset, namely wikixpr. wik-ixpr contains 65k bilingual phrase pairs of eight language pairs, and provides example sentences for each phrase. we conduct a comprehensive evaluation of xpr on wikixpr under four evaluation settings, i.e., unsupervised, supervised, zero-shot transfer, and multilingual supervised. our xpr model substantially outperforms the retrieval baselines based on cross-lingual word embeddings and cross-lingual sentence encoders. xpr also shows impressive zero-shot transferability that enables the model to be trained in a language pair and directly perform phrase retrieval for other language pairs. moreover, we present an in-depth analysis on xpr, showing that using example sentences improves both the learned xpr model and the phrase representations. our contributions are summarized as follows: phrase retrieval recent research on phrase retrieval typically learns phrase representations. @xcite propose to treat phrases as the smallest retrieval unit for open-domain question answering, where the phrases are encoded as indexable query-agnostic representations. the retrieval methods can be further improved with selfsupervised pretraining, leading to better performance on open-domain question answering @xcite . additionally, deer @xcite formulates the entity linking task as an entity phrase retrieval problem. however, these works study phrase retrieval in a monolingual scenario while we focus on cross-lingual phrase retrieval. contrastive learning contrastive learning learns representations by a contrastive loss that encourages the positive data pairs to be more similar than other data pairs. it has shown to be effective for learning representations of a wide range of modalities including visual representations @xcite @xcite , sentence representations @xcite @xcite , audio representations @xcite , etc. different from previous work that performs contrastive learning at sentence level, we introduce contrastive learning to learn phrase representations. he climbed the apple tree. it takes five years to grow an apple tree. what is an apple tree? xpr … apple tree query phrase 𝒒 𝒑 𝟏 𝒑 𝟐 𝒑 𝟑 𝒑 𝟒 𝒑 𝑵-𝟏 example sentences pre-computed phrase representations (a) cross-lingual phrase retrieval with xpr xpr encoder momentum encoder momentum update what is a lemon tree? 我种下一颗柠檬树。 bilingual phrase pair example sentences example sentences (b) training xpr with cross-lingual phrase contrast ## methods figure 1 shows the overview of xpr. in this section, we first introduce the model architecture of xpr, and then present the cross-lingual phrase contrast loss. finally, we show the training procedure of xpr. ## wikixpr: cross-lingual phrase retrieval dataset to evaluate our model, we create wikixpr, a cross-lingual phrase retrieval dataset extracted from wikipedia. wikixpr consists of bilingual phrase pairs in eight english-centric language pairs, and contains large-scale example sentences for the phrases, which enable models to leverage contextual information to better understand phrases. in what follows, we describe how we construct the wikixpr dataset. ## experiments in this section, we first present four evaluation settings for cross-lingual phrase retrieval, and describe the models to be compared. then, we present the experimental results. ## conclusion in this work, we propose a cross-lingual phrase retriever xpr, which outperforms the baseline retrievers on a range of diverse languages. moreover, we create a cross-lingual phrase retrieval dataset that contains diverse languages with largescale example sentences. for future work, we would like to improve xpr by: 1) extending xpr to asymmetric retrieval scenarios such as opendomain question answering, 2) exploring how to utilize parallel corpora for training xpr.