Cross-language Sentence Selection via Data Augmentation and Rationale Training (2021)

Abstract: This paper proposes an approach to cross-language sentence selection in a low-resource setting. It uses data augmentation and negative sampling techniques on noisy parallel sentence data to directly learn a cross-lingual embedding-based query relevance model. Results show that this approach performs as well as or better than multiple state-of-the-art machine translation + monolingual retrieval systems trained on the same parallel data. Moreover, when a rationale training secondary objective is applied to encourage the model to match word alignment hints from a phrase-based statistical machine translation model, consistent improvements are seen across three language pairs (English-Somali, English-Swahili and English-Tagalog) over a variety of state-of-the-art baselines.

## introduction sentence-level query relevance prediction is important for downstream tasks such as query-focused summarization and open-domain question answering; accurately pinpointing sentences containing information that is relevant to the query is critical to generating a responsive summary/answer (e.g., @xcite ). in this work, we focus on sentence-level query relevance prediction in a cross-lingual setting, where the query and sentence collection are in different languages and the sentence collection is drawn from a low-resource language. our approach enables english speakers (e.g., journalists) to find relevant information expressed in local sources (e.g., local reaction to the pandemic and vaccines in somalia). while we can use machine translation (mt) to translate either the query or each sentence into a common language, and then use a monolingual information retrieval (ir) system to find relevant sentences, work on probabilistic structured queries (psq) @xcite has shown that the performance of such mt+ir pipelines is hindered by errors in mt. as is well known, complete translation of the sentence collection is not necessary. inspired by previous work @xcite , we go a step further and propose a simple cross-lingual embedding-based model that avoids translation entirely and directly predicts the relevance of a query-sentence pair (where the query and sentence are in different languages). for training, we treat a sentence as relevant to a query if there exists a translation equivalent of the query in the sentence. our definition of relevance is most similar to the lexical-based relevance used in @xcite and @xcite but our query and sentence are from different languages. we frame the task as a problem of finding sentences that are relevant to an input query, and thus, we need relevance judgments for query-sentence pairs. our focus, however, is on low-resource languages where we have no sentence-level relevance judgments with which to train our query-focused relevance model. we thus leverage noisy parallel sentence collections previously collected from the web. we use a simple data augmentation and negative sampling scheme to generate a labeled dataset of relevant and irrelevant pairs of queries and sentences from these noisy parallel corpora. with this synthetic training set in hand, we can learn a supervised cross-lingual embedding space. while our approach is competitive with pipelines of mt-ir, it is still sensitive to noise in the parallel sentence data. we can mitigate the negative effects of this noise if we first train a phrase-based statistical mt (smt) model on the same parallel sentence corpus and use the extracted word alignments as additional supervision. with these alignment hints, we demonstrate consistent and significant improvements over neural and statistical mt+ir @xcite @xcite , three strong cross-lingual embedding-based models (bivec @xcite , sid-sgns @xcite , muse @xcite ), a probabilistic occurrence model @xcite , and a multilingual pretrained model xlm-roberta @xcite . we refer to this secondary training objective as rationale training, inspired by previous work in text classification that supervises attention over rationales for classification decisions @xcite . to summarize, our contributions are as follows. we (i) propose a data augmentation and negative sampling scheme to create a synthetic training set of cross-lingual query-sentence pairs with binary relevance judgements, and (ii) demonstrate the effectiveness of a supervised embedding-based cross-lingual relevance (seclr) model trained on this data for low-resource sentence selection tasks on text and speech. additionally, (iii) we propose a rationale training secondary objective to further improve seclr performance, which we call seclr-rt. finally, (iv) we conduct training data ablation and hubness studies that show our method's applicability to even lower-resource settings and mitigation of hubness issues @xcite . these findings are validated by empirical results of experiments in a low-resource sentence selection task, with english queries over sentence collections of text and speech in somali, swahili, and tagalog. ## related work query-focused sentence selection sentencelevel query relevance prediction is important for various downstream nlp tasks such as queryfocused summarization @xcite @xcite and open-domain question answering @xcite @xcite . such applications often depend on a sentence selection system to provide attention signals on which sentences to focus upon to generate a query-focused summary or answer a question. cross-language sentence selection a common approach to cross-language sentence selection is to use mt to first translate either the query or the sentence to the same language and then perform standard monolingual ir @xcite . the risk of this approach is that errors in translation cascade to the ir system. as an alternative to generating full translations, psq @xcite uses word-alignments from smt to obtain weighted query term counts in the passage collection. in other work, @xcite use a 2-state hidden markov model (hmm) to estimate the probability that a passage is relevant given the query. ## methods we first describe our synthetic training set generation process, which converts a parallel sentence corpus for mt into cross-lingual query-sentence pairs with binary relevance judgements for training our seclr model. following that, we detail our seclr model and finish with our method for rationale training with word alignments from smt. ## results and discussion we report mean average precision (map) of our main experiment in bivec and muse under-perform both of our model variants across all test conditions, suggesting that for the sentence selection task the relevance classification objective is more important than learning monolingual distributional signals. curiously, sid-sgns is quite competitive with se-clr, beating it on so and sw eval (both modalities) and tl dev speech (five out of 16 test conditions) and is competitive with the other baselines. again, the rationale training proves more effective as sid-sgns never surpasses seclr-rt. while mt+ir is a competitive baseline, it is consistently outperformed by psq across all test conditions, suggesting that in low-resource settings it is not necessary to perform full translation to achieve good sentence selection performance. smt, psq, and seclr-rt all make use of the same word-alignment information but only smt generates translations, adding additional evidence to this claim. psq and seclr are close in performance on analysis and dev sets with seclr eking out a slight advantage on seven of 12 anaylsis/dev set conditions. on the larger eval partitions, it becomes clearer that psq is superior to seclr, suggesting that the relevance classification objective is not as informative as word alignment information. the relevance classification and trained rationale objectives capture slightly different information it seems; seclr-rt, which uses both, out-performs psq across all 16 test conditions. ## training data ablation study in section 5, we have shown that seclr-rt consistently out-performs all baselines across all languages. since this work targets cross-language sentence selection in a low-resource setting, we perform a training data ablation study to understand how training data size affects effectiveness. we performed the ablation study for our two models seclr and seclr-rt, and the two strongest baseline methods psq and sid-sgns. to simulate further the scenario of data scarcity, we sub-sampled our parallel corpus uniformly at random for 5%, 10%, 25%, 50% of the sentence pairs of the original corpus. each sentence pair in the parallel corpus is sampled with equal probability regardless of sentence length. for consistency, for each sample size, the same sampled parallel corpus is used across all models. the word alignment probability matrix used by psq and seclr-rt is generated from the same sampled corpus. since we tune the vocabulary size on the dev set, for fair comparison we only report map scores on the analysis and eval sets. we plot map scores of the four models as a function of percentage of data sampled in figure 1 . overall, we see that seclr-rt consistently outperforms other baselines across all sample sizes in 9 out of 10 settings, and in the one case where it does not yield consistent improvement (tagalog analysis speech), seclr-rt achieves comparable performance to psq. in the low-resource setting when the sample size is 5% or 10%, seclr consistently underperforms other models, confirming our observation that seclr is sensitive to noise and vulnerable to learning co-occurrences of word pairs that are in fact irrelevant. when the sample size is 5% or 10%, psq consistently achieves better performance than sid-sgns and seclr (although still under-performing seclr-rt), indicating that alignment-based methods are more robust to noise and especially useful when data is extremely scarce. the fact that seclr-rt consistently out-performs seclr by a wide margin for small sample sizes indicates the necessity and effectiveness of incorporating alignment-based information into seclr to improve the robustness of the model and learn more precise alignments. ## alleviating the hubness problem in this section, we show that by incorporating alignment information through rationale training, seclr-rt significantly alleviates the hubness problem present in the trained cross-lingual embedding space produced by seclr. previous research on cross-lingual word embeddings has observed that a high-dimensional representation space with a similarity-based metric often induces a hub structure @xcite . specifically, in a high-dimensional space (e.g., a cross-lingual word embedding space) defined with a pairwise similarity metric (e.g., cosine similarity), there exist a few vectors that are the nearest neighbors of many other vectors. such vectors are referred to as "hubs." the hub structure is problematic in ir since the hub vectors are often wrongly predicted as relevant and similar in meaning to queries that are in fact irrelevant @xcite . let v q and v s be the embedding spaces for the query and sentence collection languages respectively. we define the size of the neighborhood of embeddings around @xmath0 following @xcite , we use s n 10 = e @xmath1, where µ and σ refer to the mean and standard deviation of n 10 (y) respectively. since cosine similarity is more frequently used as the similarity metric in hubness analysis, we re-train seclr and seclr-rt by replacing the dot product similarity metric with cosine similarity and still get performance comparable to we report s n 10 scores for seclr and seclr-rt respectively in ## conclusion in this work, we presented a supervised crosslingual embedding-based query relevance model, seclr, for cross-language sentence selection and also applied a rationale training objective to further increase model performance. the resulting seclr-rt model outperforms a range of baseline methods on a cross-language sentence selection task. study of data ablation and hubness further indicate our model's efficacy in handling lowresource settings and reducing hub structures. in future work, we hope to apply our sentence-level query relevance approach to downstream nlp tasks such as query-focused summarization and opendomain question answering.