Parallel Sentence Mining by Constrained Decoding (2020)

Abstract: We present a novel method to extract parallel sentences from two monolingual corpora, using neural machine translation. Our method relies on translating sentences in one corpus, but constraining the decoding by a prefix tree built on the other corpus. We argue that a neural machine translation system by itself can be a sentence similarity scorer and it efficiently approximates pairwise comparison with a modified beam search. When benchmarked on the BUCC shared task, our method achieves results comparable to other submissions.

## introduction having large and high-quality parallel corpora is critical for neural machine translation (nmt). one way to create such a resource is to mine the web @xcite . once texts are crawled from the web, they form large collections of data in different languages. to find parallel sentences, a natural way is to score sentence similarity between all possible sentence pairs and extract the topscoring ones. this poses two major challenges: scoring each source sentence against each target sentence results in unaffordable quadratic time complexity. a typical workflow reduces the search complexity in a coarse-to-fine manner by aligning documents then aligning sentences within documents @xcite . however, translated websites may not have matching document structures. more recent methods focus on direct sentence alignment. the results from building and using comparable corpora (bucc) shared task show that direct sentence alignment can be done by sentence-level lexical comparison, neural comparison or a combination of the two @xcite . a state-of-the-art method maps all sentences to multilingual sentence embeddings and compares them using vector similarity @xcite . such sentence embeddings are produced by neural encoders, but the rise of the attention mechanism demonstrates that sentence embeddings alone are insufficient to obtain full translation quality @xcite . to exploit quality gains from the attention mechanism, we propose to use a full nmt system with attention to score potentially parallel sentences. the way we avoid pairwise scoring is inspired by constrained decoding in nmt, where the choice of output tokens is constrained to a predefined list @xcite . our method works as follows: we designate one language as source and one language as target, and build a trie over all target sentences. then we translate each source sentence to the target language, but constrain left-to-right beam search to follow the trie. in other words, every translation hypothesis is a prefix of some sentence in the target language. rather than freely choosing which token to extend by, a hypothesis is limited to extensions that exist in the target language corpus. in effect, we are using beam search to limit target language candidates for each source sentence. our work makes two contributions to parallel sentence mining. first, instead of comparing translated text or neural similarity, we use an nmt model to directly score and retrieve sentences onthe-fly during decoding. second, we approximate pairwise comparison with beam search, so only the top-scoring hypotheses need to be considered at each decoding step. ## methodology nmt systems can assign a conditional translation probability to an arbitrary sentence pair. filtering based on this (junczys-dowmunt, 2018) won the wmt 2018 shared task on parallel corpus filtering @xcite . intuitively, we could score every pair of source and target sentences using a translation system in quadratic time, then return pairs that score highly for further filtering. we approximate this with beam search. ## results and analysis experiments on the sample data in regarding filtering methods, we notice that bicleaner achieves a more balanced precision and recall, while filtering by per-word cross-entropy leads to very high precision but lower recall. generally, the latter does better in terms of f1. taking a union of the output from the two filtering methods results in a even more balanced precision and recall, without damaging f1. this implies that the two filtering techniques keep different sentence pairs. overall our method does not outperform stateof-the-art which leverages neural embeddings. we identify several weaknesses: beam search can only find local optima, and a genuine parallel sentence cannot be recovered once it is pruned. thus the method is vulnerable when parallel sentences have different word ordering. for example, "por el momento, estoy bebiendo un café" (english: "at the moment, i am drinking a coffee") can hardly match "i am drinking a coffee at the moment", because an nmt system will have very low probability of generating a reordered translation, unless using an undesirably large beam size. moreover, compared to methods that consider textual overlap, nmt is sensitive to domain mismatch and rare words @xcite . when a system is confused by rare words in the source, we observe that the overly zealous language model in the decoder generates a fluent sentence in the trie rather than a translation. this problem is alleviated when our systems are fine-tuned on indomain data, as shown in finally we discuss the limitation of evaluating our method on the bucc task. first, our method based on nmt can be liable to favour machine-translated texts, whereas the bucc data is unlikely to contain those. next, we notice that some parallel sentences in bucc data are not included in the gold alignments. for instance, in de-en train set, "de-000081259" and "de-000081260" are the same german sentence, and so are "en-000036940" and "en-000036941" on the english side. gold alignments only include (de-000081259, en-000036940) and (de-000081260, en-000036941), but not the other two. lastly, it still remains unknown if a system optimised for f1 will produce the sentences that can truly improve nmt performance. ## related work a typical parallel corpus mining workflow first aligns parallel documents to limit the search space for sentence alignment. early methods rely on webpage structure @xcite . later, @xcite translate all documents into a single language, and shortlist candidate document pairs based on tf-idf-weighted n-grams. recently, @xcite suggest a neural method to compare document embeddings obtained from sentence embeddings . with the assumption that matched documents are parallel (no cross-alignment), sentence alignment can be done by comparing sentence length in words @xcite or characters @xcite , which is then improved by adding lexical features @xcite . after translating texts into the same language, bleu can also be used to determine parallel texts, by anchoring the most reliable alignments first @xcite . most recently, @xcite propose to compare bilingual sentence embeddings with dynamic programming in linear runtime. there are also research efforts on parallel sentence extraction without the reliance on document alignment. @xcite acquire parallel phrases from comparable corpora using bilingual tries and seed dictionaries. @xcite computes jaccard similarity of lexical translation overlap. @xcite use an autoencoder and a maximum entropy classifier. @xcite consider cosine similarity between averaged multilingual word embeddings. @xcite design a dual encoder model to learn multilingual sentence em-beddings directly with added negative examples. wieting et al. (2019) in our work, using nmt as a similarity scorer relies on constrained decoding @xcite , which has been applied on image captioning @xcite and keyword generation @xcite . ## conclusion and future work we bring a new insight into using nmt as a similarity scorer for sentences in different languages. by constraining on a target side trie during decoding, beam search can approximate pairwise comparison between source and target sentences. thus, overall we present an interesting way of finding parallel sentences through trie-constrained decoding. our method achieves a comparable f1 score to existing systems with a vanilla architecture and data. maximising machine translation scores is biased towards finding machine translated text produced by a similar model. more research is needed on this problem given the prevalent usage of nmt. we hypothesise that part of the success of dual conditional cross-entropy filtering (junczys-dowmunt, 2018) is checking that scores in both directions are approximately equal, whereas a machine translation would be characterised by a high score in one direction. finally, scalability is a key issue in large-scale mining of parallel corpora, where both quantity and quality are of concern. the scalability of direct sentence alignment without a document aligner has not been thoroughly investigated in our work, as well as other related work.