We present a novel technique for learning semantic representations , which extends the distributional hypothesis to multilingual data and joint-space embeddings . Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences , while maintaining sufficient distance between those of dissimilar sentences . The models do not rely on word alignments or any syntactic information and are successfully applied to a number of diverse languages . We extend our approach to learn semantic representations at the document level , too . We evaluate these models on two cross-lingual document classification tasks , outperforming the prior state of the art . Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic relationships across languages without parallel data . Distributed representations of words provide the basis for many state-of-the-art approaches to various problems in natural language processing today . Such word embeddings are naturally richer representations than those of symbolic or discrete models , and have been shown to be able to capture both syntactic and semantic information . Successful applications of such models include language modelling ( Bengio et al . , 2003 ) , paraphrase detection ( Erk and Padó , 2008 ) , and dialogue analysis ( Kalchbrenner and Blunsom , 2013 ) . Within a monolingual context , the distributional hypothesis ( Firth , 1957 ) forms the basis of most approaches for learning word representations . In this work , we extend this hypothesis to multilingual data and joint-space embeddings . We present a novel unsupervised technique for learning semantic representations that leverages parallel corpora and employs semantic transfer through compositional representations . Unlike most methods for learning word representations , which are restricted to a single language , our approach learns to represent meaning across languages in a shared multilingual semantic space . We present experiments on two corpora . First , we show that for cross-lingual document classification on the Reuters RCV1 / RCV2 corpora ( Lewis et al . , 2004 ) , we outperform the prior state of the art ( Klementiev et al . , 2012 ) . Second , we also present classification results on a massively multilingual corpus which we derive from the TED corpus ( Cettolo et al . , 2012 ) . The results on this task , in comparison with a number of strong baselines , further demonstrate the relevance of our approach and the success of our method in learning multilingual semantic representations over a wide range of languages . To summarize , we have presented a novel method for learning multilingual word embeddings using parallel data in conjunction with a multilingual objective function for compositional vector models . This approach extends the distributional hypothesis to multilingual joint-space representations . Coupled with very simple composition functions , vectors learned with this method outperform the state of the art on the task of cross-lingual document classification . Further experiments and analysis support our hypothesis that bilingual signals are a useful tool for learning distributed representations by enabling models to abstract away from mono-lingual surface realisations into a deeper semantic space .

Challenge: Existing methods for learning word representations are restricted to a single language.
Approach: They extend the distributional hypothesis to multilingual data and joint-space embeddings and use parallel data to obtain multilingual word representations.
Outcome: Multilingual word vectors learned using the proposed model outperform existing methods on cross-lingual document classification tasks without word-level alignment or syntactic information in diverse languages.