Multilingual Universal Sentence Encoder for Semantic Retrieval (2020)

Abstract: We present easy-to-use retrieval focused multilingual sentence embedding models, made available on TensorFlow Hub. The models embed text from 16 languages into a shared semantic space using a multi-task trained dual-encoder that learns tied cross-lingual representations via translation bridge tasks (Chidambaram et al., 2018). The models achieve a new state-of-the-art in performance on monolingual and cross-lingual semantic retrieval (SR). Competitive performance is obtained on the related tasks of translation pair bitext retrieval (BR) and retrieval question answering (ReQA). On transfer learning tasks, our multilingual embeddings approach, and in some cases exceed, the performance of English only sentence embeddings.

## introduction we introduce three new multilingual members in the universal sentence encoder (use) (cer et al., 2018) family of sentence embedding models. the models target performance on tasks that involve multilingual semantic similarity and achieve a new state-of-the-art in performance on monolingual and cross-lingual semantic retrieval (sr). one model targets efficient resource usage with a cnn model architecture (kim, 2014). another targets accuracy using the transformer architecture (vaswani et al., 2017). the third model provides an alternative interface to our multilingual transformer model for use in retrieval question answering (reqa). the 16 languages supported by our multilingual models are given in languages family arabic (ar) semitic chinese (prc) (zh) sino-tibetan chinese (taiwan) (zh-tw) dutch(nl) english(en) germanic german (de) french (fr) italian (it) latin portuguese (pt) spanish (es) japanese (ja) japonic korean (ko) koreanic russian (ru) polish (pl) slavic thai (th) kra-dai turkish (tr) turkic ## model toolkit our multilingual models are implemented in ten-sorflow (abadi et al., 2016) and made publicly available on tensorflow hub. listing 2: python code embedding a question and answer for retrieval question-answering (reqa). multi-task dual-encoder training framework, with a single shared encoder supporting multiple downstream tasks. the training tasks include: a multifeature question-answer prediction task, 4 a translation ranking task, and a natural language inference (nli) task. additional task specific hidden layers for the question-answering and nli tasks are added after the shared encoder to provide representational specialization for each type of task. the model training architecture is illustrated at figure 1. ## experiments on transfer tasks for comparison with prior use models, english task transfer performance is evaluated on senteval (conneau and kiela, 2018). for sentence classification transfer tasks, the output of the sentence encoders are provided to a task specific dnn. for the pairwise semantic similarity task, the similarity of sentence embeddings 𝑢 and 𝑣 is assessed using -arccos ## resource usage figure 2 provides compute and memory usage benchmarks for our models. @xcite inference times on gpu are 2 to 3 times faster than cpu. our cnn models have the smallest memory footprint and are the fastest on both cpu and gpu. the memory requirements increase with sentence length, with the transformer model increasing more than twice as fast as the cnn model. @xcite while this makes cnns an attractive choice for efficiently encoding longer texts, this comes with a corresponding drop in accuracy on many retrieval and transfer tasks. ## conclusion easy-to-use retrieval focused multilingual models for embedding sentence-length text are made avail-21 cpu benchmarks are run on intel(r) xeon(r) platinum 8173m cpu @ 2.00ghz. gpu benchmarks were run on an nvidia v100. memory footprint was measured on cpu.