Paper:
An Autoencoder Approach to Learning Bilingual Word Representations
Cross-language learning allows one to use training data from one language to build models for a different language. Many approaches to bilingual learning require that we have word-level alignment of sentences from parallel corpora. In this work we explore the use of autoencoder-based methods for cross-language learning of vectorial word representations that are coherent between two languages, while not relying on word-level alignments. We show that by simply learning to reconstruct the bag-of-words representations of aligned sentences, within and between languages, we can in fact learn high-quality representations and do without word alignments. We empirically investigate the success of our approach on the problem of cross-language text classification, where a classifier trained on a given language (e.g., English) must learn to generalize to a different language (e.g., German). In experiments on 3 language pairs, we show that our approach achieves state-of-the-art performance, outperforming a method exploiting word alignments and a strong machine translation baseline.

Review:
This paper proposes to learn bilingual word vector representations through an autoencoder.The novelty of this approach is to not rely on word-level alignments. It only requires aligned sentences.An autoencoder model is used to reconstruct the bag-of-words representation of aligned sentences, within and between languages.Two different variations are proposed. The first one is based on binary bag-of-words (i.e. words present or not) and the second considers the frequency of each word. The authors demonstrate the quality of these representations for the task of cross-language document classification, where they have been able to reach state-of-the-art performance.I like the idea to learn cross-lingual word representations while not relying on word-level alignment.This offers a clear advantage compared to other approaches.Furthermore, the method using binary bag-of-words reconstruction is quite simple and give the best overall results. And I like simple methods.However, the authors introduce a training with merged bags-of-words that it is not very clear to me.They said that they merged adjacent sentences to perform mini-batch training and speed up the process. But they don't say how they define sentences as adjacent.This model is also vocabulary-dependent. The vocabulary size in the experiment is relatively small (between 35,000 and 50,000). Would this method be scalable with a larger vocabulary?I am also surprised to see that the tree-based model performs significantly worse (less than 10-14%) for EN/DE, DE/EN, EN/FR and FR/EN pairs, while it performs much better for EN/ES (more than 10%).How do the authors explain such variation?That makes me think that the tree-based model is not properly trained.In my opinion, it remains too many open questions in the experiment section.The authors compared their results to Hermann and Blunsom's paper. Hermann and Blusom also proposed a method to learn multilingual distributed representations without word alignment.They showed in their paper the usefulness of adding training data to increase the overall performance.However the training size is never mentioned in this paper. The authors only said that the corpus contains roughly 2 million parallel sentences. Did they train the word representations with all of them? In this case, it would make the training data twice larger than the Hermann and Blunsom's training dataset and explain why they achieve better performance.Or it could be due to the fact that the authors fine-tuned the monolingual embeddings on RCV1/RCV2.To make a fair comparison, it would be nice to run experiments with the same number of training sentence pairs and without fine-tuning.The authors also said that they consider only documents which were assigned exactly one of the 4 top level categories in the topic hierarchy.This makes a 4-class classification problem. How is it possible to have a majority class accucary smaller than 25% for EN/FR, EN/ES and ES/EN pairs? Results for the other models are also not described in enough detail.For Klementiev's model with EN/DE language pairs, the authors report the results from the paper. But is it the same test set?What classifier is used with the MT's model?My intuition would be to use the translated words to construct a document representation with the model described in section 3.2.But I am not sure this is the case here, because the authors again report results from Klementiev et al. for EN/DE and DE/EN pairs. Authors also said that their embeddings are meaningful because they can generalize well even at low data sizes. It is however only true for EN/DE pairs, as we can see that the accuracy is still increasing with 10,000 training documents for DE/EN pairs.Finally, I think that the authors should mention the paper "Multilingual Distributed Representations without Word Alignment" from Hermann and Blunsom in the related work section.As their method looks very similar to the one proposed in this paper, it seems to me important to mention what are the differences and the benefits of the method compared to theirs. -- After reading authors' comments: we still believe there are too many imprecisions in the experimental setup for a NIPS paper.