Paper:
Neural machine translation, which achieves near human-level performance in some languages, strongly relies on the availability of large amounts of parallel sentences, which hinders its applicability to low-resource language pairs. Recent works explore the possibility of unsupervised machine translation with monolingual data only, leading to much lower accuracy compared with the supervised one. Observing that weakly paired bilingual documents are much easier to collect than bilingual sentences, e.g., from Wikipedia, news websites or books, in this paper, we investigate the training of translation models with weakly paired bilingual documents. Our approach contains two components/steps. First, we provide a simple approach to mine implicitly bilingual sentence pairs from document pairs which can then be used as supervised signals for training. Second, we leverage the topic consistency of two weakly paired documents and learn the sentence-to-sentence translation by constraining the word distribution-level alignments. We evaluate our proposed method on weakly paired documents from Wikipedia on four tasks, the widely used WMT16 German↔English and WMT13 Spanish↔English tasks, and obtain 24.1/30.3 and 28.1/27.6 BLEU points separately, outperforming stateof-the-art unsupervised results by more than 5 BLEU points and reducing the gap between unsupervised translation and supervised translation up to 50%.

Review:
This paper proposes a method to train a machine translation system using weakly paired bilingual documents from Wikipedia. A pair of sentences from a weak document pair are used as training data if their cosine similarity exceeds c1, and the similarity between this sentence pair is c2 greater than any other pair in the documents, under sentence representations formed from word embeddings trained with MUSE. The neural translation model learns to translate from language X to Y, and from Y to X using the same encoder and decoder parameters, but the decoder is aware of the intended target language given an embedding of the intended language. The model is also trained to minimise the KL divergence between the distribution of terms in the target language document and the distribution of terms in the current model output. The model also uses the denoising autoencoding and reconstruction objectives of Lample et al. (2017). The results show improvements over the Lample et al. (2017) and that performance is heavily dependent on the number of sentences extracted from the weakly aligned documents.
Positives
- Large improvement over previous attempts at unsupervised MT for the En-De language pair.
- Informative ablation study in Section 4.4 of the relative contribution of each part of the overall objective function (Eq 9).
Negatives
- The introduction gave the impression that this method would be applied to low-resource language pairs but it was applied to two high-resource language pairs. Because you have not evaluated on a low-resource language pair, it's not clear how your proposed method would generalise to a low-resource setting.
Questions
- Can you give some intuition for why you remove the first principal component from the word embeddings in Equations 1 - 3?
- Are the Supervised results in Table 2 actually a fair reflection of a reasonable NMT model trained with sub-word representations and back translated data?
- What is the total number of sentences in the weakly paired documents in Table 1? It would be useful to know the proportion of sentences you managed to extract to train your models.
Comments
- Koehn et al. (2003) is not an example of any kind of neural network architecture.