Paraphrastic Representations at Scale (2022)

Abstract: We present a system that allows users to train their own state-of-the-art paraphrastic sentence representations in a variety of languages. We release trained models for English, Arabic, German, Spanish, French, Russian, Turkish, and Chinese. We train these models on large amounts of data, achieving significantly improved performance from our original papers on a suite of monolingual semantic similarity, cross-lingual semantic similarity, and bitext mining tasks. Moreover, the resulting models surpass all prior work on efficient unsupervised semantic textual similarity, even significantly outperforming supervised BERT-based models like Sentence-BERT (Reimers and Gurevych, 2019). Most importantly, our models are orders of magnitude faster than other strong similarity models and can be used on CPU with little difference in inference speed (even improved speed over GPU when using more CPU cores), making these models an attractive choice for users without access to GPUs or for use on embedded devices. Finally, we add significantly increased functionality to the code bases for training paraphrastic sentence models, easing their use for both inference and for training them for any desired language with parallel data. We also include code to automatically download and preprocess training data.

## introduction measuring sentence similarity @xcite is an important task in natural language processing, and has found many uses including paraphrase detection @xcite , bitext mining @xcite , language modelling @xcite , questionanswering @xcite , and as reward functions or evaluation metrics for language generation tasks @xcite . within this context, fast and light-weight methods are particularly useful as they make it easy to compute similarity over the ever-increasing volumes of web text available. for instance, we may want to mine a hundred million parallel sentences @xcite or use a semantic similarity reward when fine-tuning language generation models on tens of millions of training examples. these tasks are much more feasible when using approaches that are fast, can be run on cpu, and use little ram, allowing for increased batch size. this need for fast inference is one motivation for using sentence embeddings. sentence embeddings allow the search for similar sentences to be linear in the number of sentences, or even sub-linear when using highly optimized tools like faiss @xcite that allow for efficient nearest neighbor search. this is contrast to models, like crossattention models, which are quadratic during inference as they require both of the texts being compared as inputs. as we show in this paper, our simple and interpretable word-averaging sentence embedding models @xcite @xcite , are orders of magnitude faster to compute than prior embedding approaches while simultaneously possessing significantly stronger performance on monolingual and cross-lingual semantic similarity tasks. since we are simply averaging embeddings and have no neural architecture, any models based on neural architectures, especially large pretrained neural architectures which are increasingly used, will not be as fast as the models described in this paper. lastly, we also show that this approach is competitive with laser @xcite , a state-ofthe-art multilingual model, on mining bitext and has stronger performance on cross-lingual semantic similarity, while having inference speeds that are twice as fast on gpu and orders of magnitude faster on cpu. we make several contributions in this paper that go beyond our prior work. firstly, we reformat the code to support training models on tens of millions of sentence pairs efficiently and with low ram usage. secondly, we train an english model on 25.85 million paraphrase pairs from paranmt @xcite , a paraphrase corpus we previously constructed automatically from bitext. we then train models directly on x-english bitext for arabic, german, spanish, french, russian, turkish, and chinese, producing models that are able to distinguish both paraphrases in english and their respective languages as well as cross-lingual x-english paraphrases. even though all models are able to model semantic similarity in english, we find that training on paranmt specifically leads to stronger models as it is easier to filter the data to remove noise and sentence pairs with little to no diversity. we refer to our models as paragram-sp, abbreviated as p-sp, foot_1 referring to how the models are based on averaging subword units generated by sentencepiece @xcite . we make all of these models available to the community for use on downstream tasks. we also add functionality to our implementation. besides the support for efficient, low-memory training on tens of million of sentence pairs described above, we add code to support (1) reading in a list of sentences and producing a saved numpy array of the sentence embeddings; (2) reading in a list of sentence pairs and producing cosine similarity scores; and (3) downloading and preprocessing evaluation data, bitext, and paraphrase data. for bitext and paraphrase data, we provide support for training using either text files or hdf5 files. lastly, this paper contains new experiments showcasing the limits of these scaled-up models and detailed comparisons with prior work on a suite of semantic similarity tasks in a variety of languages. we release our code and models to the community in the hope that they will be found useful for research and applications, as well as using them as a base to build stronger, faster models covering more of the languages of the world. ## methods we first describe our objective function and then describe our encoder. training. the training data consists of a sequence of parallel sentence pairs (s i , t i ) in source and target languages respectively. note that for training our english model, the source and target languages are both english as we are able to make use of an existing paraphrase corpus. for each sentence pair, we randomly choose a negative target sentence t i during training that is not a translation or paraphrase of s i . our objective is to have source and target sentences be more similar than source and negative target examples by a margin δ: ## code and usage we added a number of features to the code base to improve performance and make it easier to use. first, we added code to support easier inference. examples of using the code programmatically to embed sentences and score sentence pairs (using cosine similarity) are shown in figure 1 . our code base also supports functionality that allows one to read in a list of sentences and produce a saved numpy array of the sentence embeddings. we also included functionality that allows one to read in a list of sentence pairs and produce the sentence pairs along with their cosine similarity scores in an output file. these scripts allow our models to be used without any programming for the two most common use cases: embedding sentences and scoring sentence pairs. examples of their usage with a trained model are shown in figure 2 . from models import load_model @xmath0. ' @xmath1. ' # load english paraphrase model @xmath2. para . lc .100. pt ' @xmath3. model ' model , _ = load_model ( @xmath4, @xmath5. embed_raw_text ([ text1 , text2 ]) # 2 d numpy array of sentence embeddings @xmath6. score_raw_text ([( text1 , text2 ) ]) # list of cosine scores figure 1: usage example of programmatically loading one of our pretrained models and obtaining sentence embeddings and scores for two sentences. python -u embed_sentences . py --sentencefile paraphrase -at -scale / examplesentences . txt --load -file paraphrase -at -scale / model . para . lc .100. pt -output -file sentence_embeds . np python score_sentence_pairs . py -sentence -pair -file paraphrase -atscale / example -sentences -pairs . txt -load -file paraphrase -at -scale / model . para . lc .100. pt figure 2 : usage examples to embed sentences and score sentence pairs. the first command is a usage example of scoring a list of sentence pairs. the file example-sentences-pairs.txt contains a list of sentences, one per line. the output of the script is a saved numpy array of sentence embeddings in the same order of the input sentences. the second command is a usage example of scoring a list of sentence pairs. the file example-sentences-pairs.txt contains pairs of tab-separated sentences, one per line. the output of the script is a text file containing the tab separated list of sentences along with their cosine scores in the same order of the input sentences. secondly, we added a training mode using hdf5 4 format, allowing training data to remain on disk during training. this leads to a significant reduction in ram usage during training, which is especially true when using more than 10 million training examples. efficient training can now be done on cpu only using only a few gigabytes of ram. lastly, we also added code for preprocessing 4 https://docs.h5py.org/en/stable/ 1 cd preprocess / bilingual && bash do_all . sh fr -es -de 2 cd ../.. 3 cd preprocess / paranmt && bash do_all . sh 0.4 1.0 0.7 figure 3 : usage examples to download and preprocess bilingual and paranmt data. the first command downloads and preprocesses (filters, trains sentencepiece models, tokenizes if language is zh, converts files to hdf5 format) en-x bilingual data. the third command downloads and preprocesses paranmt data. the arguments are used to filter the data (semantic similarity scores between 0.4 and 1.0 and trigram overlap below 0.7, which have been used in prior papers when generating training data for paraphrase generation @xcite ). ## speed analysis from the results, we see that our model is easily the fastest on gpu, sometimes by an order of magnitude. interestingly, using a single core of cpu, we achieve similar speeds to inference on gpu, which is not the case for any other model. moreover, we repeated the experiment, this time using 32 cores and achieved a speed of 15,316 sentences/second. this is even faster than when using a gpu and indicates that our model can effectively be used at scale when gpus are not available. it also suggests our model would be appropriate for use on embedded devices. ## conclusion in this paper, we present a system for the learning and inference of paraphrastic sentence embeddings in any language for which there is paraphrase or bilingual parallel data. additionally, we release our trained sentence embedding models in english, as well as arabic, german, spanish, french, russian, turkish, and chinese. these models are trained on tens of million of sentence pairs resulting in models that achieve state-of-the-art performance on unsupervised english semantic similarity and are state-of-the-art or competitive on non-english semantic similarity, cross-lingual semantic similarity, and bitext mining. moreover, our models are significantly faster than prior work owing to their simple architecture. they can also be run on cpu with little to no loss in speed from running them on gpu--something that no strong models from prior work are able to do. lastly, we release our code that has been modified to make training and inference easier, with support for training on large corpora, preprocessing paraphrase and bilingual corpora and evaluation data, as well as scripts for easy inference that can generate embeddings or semantic similarity scores for sentences supplied in a text file.