Cross-lingual Contextualized Phrase Retrieval (2024)

Abstract: Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks by leveraging the fine-grained information that phrases offer. In our work, we propose a new task formulation of dense retrieval, cross-lingual contextualized phrase retrieval, which aims to augment cross-lingual applications by addressing polysemy using context information. However, the lack of specific training data and models are the primary challenges to achieve our goal. As a result, we extract pairs of cross-lingual phrases using word alignment information automatically induced from parallel sentences. Subsequently, we train our Cross-lingual Contextualized Phrase Retriever (CCPR) using contrastive learning, which encourages the hidden representations of phrases with similar contexts and semantics to align closely. Comprehensive experiments on both the cross-lingual phrase retrieval task and a downstream task, i.e, machine translation, demonstrate the effectiveness of CCPR. On the phrase retrieval task, CCPR surpasses baselines by a significant margin, achieving a top-1 accuracy that is at least 13 points higher. When utilizing CCPR to augment the large-language-model-based translator, it achieves average gains of 0.7 and 1.5 in BERTScore for translations from X=>En and vice versa, respectively, on WMT16 dataset. We will release our code and data.

## introduction compared with the dense retrieval at sentence (or passage) level @xcite @xcite , learning dense retrieval at the phrase level has shown more appealing characteristics in extensive nlp tasks, such as entity linking @xcite , open-domain question answering @xcite ), text generation @xcite , etc. an important reason is that phrases can provide more fine-grained information than sentences. in cross-lingual research, the phrase-level dense retrieval also shows the promise to solve a range of nlp problems @xcite . the cross-lingual phrase retrieval in @xcite focuses on the mapping of crosslingual wiki entities, where each of them is represented by averaged hidden representations from 32 different contexts. however, the constrained phrase type, i.e., the wiki entity, makes the cross-lingual phrase retriever difficult to augment general nlp tasks. moreover, unlike wiki entities, which may have fewer ambiguities, general-type phrases that are lexically identical can have different meanings depending on their contexts. thus, accounting for polysemy @xcite using the context information becomes critical. even when the lexically identical phrases share similar semantics, a more appropriate context would provide richer information for utilizing the phrases in downstream tasks @xcite . therefore, we advocate for a new task formulation, i.e., cross-lingual contextualized phrase retrieval, which aims to find the cross-lingual phrase that is mostly relevant to the provided (generaltype) source phrases, considering their meanings and surrounding contexts. however, achieving this goal is non-trivial, due to the scarcity of specific training data and models. since annotating cross-lingual contextualized phrase pairs of general type is very difficult and expensive, we first introduce a data collection method, which leverages the automatically induced word alignment information from parallel sentences to extract suitable cross-lingual phrase pairs for training. this ensures that the phrases are of general type and both cross-lingual phrases and contexts are well aligned. thereafter, we propose a cross-lingual contextualized phrase retriever (ccpr) which is trained on the constructed dataset. the ccpr mainly employs phrase-level contrastive learning to draw cross-lingual phrases with similar contexts and meanings closer in the hidden space. at inference time, we can leverage the well-trained ccpr to build a phrase-level index and use the query phases for search. unlike previous works @xcite , which focus solely on the task of cross-lingual phrase retrieval for wiki entities, our work also explores the potential of leveraging ccpr to augment downstream cross-lingual tasks, e.g., machine translation (mt). one critical question in front of us is how to select meaningful phrases for indexing and searching at inference time. to address this problem, we propose to learn a phrase segmentation module, which can be used to predict meaningful phrases from sentence-or passage-level retrieval data at inference time. the developed phrase segmentation module is important for ensuring the train-test consistency of ccpr. experiments on both of the cross-lingual contextualized phrase retrieval and a downstream task, i.e., mt, show the effectiveness of ccpr. we first evaluate ccpr on the cross-lingual contextualized phrase retrieval task, since a higher accuracy is more beneficial for downstream tasks, such as mt, cross-lingual dictionary induction, etc @xcite . for this task, we use the human annotated cross-lingual phrase pairs as the golden truth and evaluate whether ccpr and other baselines can retrieve those golden phrases from a large-scale index. experiments show that ccpr outperforms the baselines for at least 13 points in terms of the top-1 accuracy. we also conduct evaluation on the mt task using a large language model (llm), e.g., llama-2 @xcite . this task aims to evaluate whether the information retrieved by ccpr can enhance llm's ability of cross-lingual generation. following the fashion of retrieval-augmented generation (rag) @xcite , we simply integrate the retrieved phrase information to the input of the llm, and compare our method with other baselines. for both x⇒en and en⇒x translation directions on the wmt16 dataset, where x is from six languages, our ccpr achieves averaged gians of 0.7 and 1.5 bertscore points @xcite , respectively. results on other evaluation metrics, e.g., comet @xcite , are consistent with bertscore. in summary, our contributions are three-fold: • we propose a new formulation of dense retrieval, i.e., the cross-lingual contextualized phrase retrieval, which has substantial potential in augmenting cross-lingual tasks. dense retrieval in text generation, dense retrieval has been widely studied at both sentence and phrase levels. particularly after the emergence of large language models (llms), how to retrieve related sentences (or passages) @xcite for input to llms has became a popular research topic, namely, retrieval-augmented generation (rag) @xcite @xcite @xcite . however, compared to the sentence-level retrieval, phrase-level retrieval has been shown to not only enhance retrieval accuracy but also markedly boost performance in downstream tasks @xcite . by indexing and integrating retrieved phrases into the output of language models, @xcite and @xcite make the text generation process more attributable and accurate. these studies underline the effectiveness of phrase-level dense retrieval. our work notably diverges from those works, because we focus more on cross-lingual tasks. retrieval in cross-lingual tasks in crosslingual tasks, the majority of research has concentrated on sentence-level retrieval. many research works in retrieval-augmented mt @xcite @xcite have explored retrieving bilingual sentences based on similarity of source sentences, guiding mt models to utilize retrieved target information. a notable limitation is their reliance on bilingual data for constructing the index. this requirement substantially limits the scale and diversity of data available for retrieval. in contrast, @xcite propose to directly retrieve relevant target sen-tences using source sentences to assist mt, facilitating cross-lingual retrieval using a monolingual index. unlike the task-specific model for mt in @xcite , many studies in dense retrieval aim to enhance the cross-lingual sentence retrieval on more languages and at a larger scale @xcite @xcite @xcite . however, cross-lingual phrase retrieval has not been extensively investigated. @xcite utilize the source side of the bilingual data for building a retrieval index and employ the similarity between source phrases for retrieval. therefore, their indexing also suffers from the limitation of reliance on bilingual data. in addition, it uses n-grams as phrases by default, leading to plenty of meaningless and noisy spans in the index. zheng et al. (2022) limit their cross-lingual phrase retrieval task to a specific phrase type, i.e., the wiki entity. the setting in their work overlooks the nuanced meanings that lexically identical phrases may convey in different contexts, making the retriever difficult to augment general nlp tasks. in contrast, our research aims to develop a cross-lingual contextualized phrase retriever for general-type phrases. additionally, there is research focused on employing nearest neighbor retrieval to support mt @xcite @xcite . these efforts differ from the aforementioned studies as they aim to estimate prediction distributions through nearest neighbor retrieval at each translation step. ## task formulation this work aims a cross-lingual phrase retriever that can augment downstream nlp tasks. to ensure its effectiveness, the retriever must adhere to two essential criteria. first, it should ensure that phrases in the index are of general type. second, it must resolve the polysemy of general-type phrases by considering their contextual differences. thus, we introduce a new task formulation, cross-lingual contextualized phrase retrieval. we are given a large collection of n general-type phrases p @xmath0, . . . , p n } in language l y , and a general-type query phrase q in language l x . each phrase, by default, is associated with the context information (c, s, e), where c represents the original sentence containing this phrase, and s and e are its start and end positions in c. the primary objective is to identify a cross-lingual phrase @xmath1, considering their contexts and meanings. our formulation notably diverges from the one proposed by @xcite , which focuses on a restricted type of phrases, i.e., wiki entities, ignoring the nuanced meanings that general-type phrases may convey in different contexts. this new task formulation also introduces several challenges, with the primary issue being the lack of training data. specifically, there is a scarcity of cross-lingual phrase pairs of general type and accompanied by contextual information. such data are crucial for training the model to recognize crosslingual phrases that have similar contexts and meanings, allowing it to align these phrases closely in a hidden space. in addition, how to extract meaningful general-type phrases from sentences is also an open question, which is critical for employing the retriever to downstream tasks. ## training data collection annotating cross-lingual contextualized phrase pairs of general type is very difficult and expensive, posing a significant obstacle to training effective cross-lingual contextualized phrase retriever. however, the sentence-level parallel data in general domains are more readily available @xcite . additionally, as the lower unit of phrase alignment, the word alignment has been extensive studied @xcite @xcite @xcite @xcite . therefore, to overcome the data scarcity, we propose to use a word alignment model, such as giza++ @xcite and neural aligner @xcite , in order to automatically induce word alignments from parallel sentences and subsequently extract cross-lingual phrase pairs. this method allows us to produce phrase pairs that are not only of general type but also accompanied with contexts, aligning with our task requirements. formally, given a pair of parallel sentences @xmath2, x 2 , ..., x |x| }, and @xmath3, y 2 , ..., y |y| }, where |x| and |y| are the number of words in the sequences, we use a word alignment model to obtain the word alignment information of x and y @xcite . if every word in a consecutive span x i:j can be aligned to a consecutive span y u:v , we will use them as a pair of aligned phrases p @xmath4:j and p @xmath5:v , where e das parlament erhebt sich zu einer schweigeminute the house rose and observed a minute' s silence e einer schweigeminute a silence předložení dokumentů: viz zápis documents received: see minutes … e documents received see minutes : linear … … … … phrase non-phrase encoder e pos neg 1 ≤ @xmath6. notably, phrases that meet the constraints may range from a single word to the whole sequence, and extracted phrases are allowed to have overlaps. in addition, the context information of p x and p y are (x, i, j) and (y, u, v), respectively. unlike the phrase table extracted in smt @xcite , which is static and context-independent, here the phrase mapping between p x and p y is unique considering their context. as shown in fig. 1 (a ), the germany phrase ("einer", "schweigeminute") is aligned with the the english phrase ("a", "minute", "'s", "silence") within similar contexts, i.e., parallel sentences. more details about the extraction of cross-lingual contextualized phrase pairs are in app. a.1. ## experiments we propose to evaluate our model on cross-lingual contextualized phrase retrieval and mt. ## conclusion & future works this paper introduces a novel approach to dense retrieval, i.e., cross-lingual contextualized phrase retrieval, focusing on resolving phrase polysemy by utilizing contextual information. the main challenge identified is the scarcity of training data, specifically cross-lingual phrase pairs with context. to overcome this, we use a word alignment model to derive such phrase pairs from parallel sentences. we then present the cross-lingual contex-tualized phrase retriever (ccpr), which employs contrastive learning to effectively capture similar meanings and contexts of cross-lingual phrases. our extensive testing across retrieval and machine translation tasks shows that ccpr significantly outperforms existing baselines.