Exploiting T witter as Source of Large Corpora of Weakly Similar Pairs for Semantic Sentence Embeddings (2021)

Abstract: Semantic sentence embeddings are usually supervisedly built minimizing distances between pairs of embeddings of sentences labelled as semantically similar by annotators. Since big labelled datasets are rare, in particular for non-English languages, and expensive, recent studies focus on unsupervised approaches that require not-paired input sentences. We instead propose a language-independent approach to build large datasets of pairs of informal texts weakly similar, without manual human effort, exploiting Twitter’s intrinsic powerful signals of relatedness: replies and quotes of tweets. We use the collected pairs to train a Transformer model with triplet-like structures, and we test the generated embeddings on Twitter NLP similarity tasks (PIT and TURL) and STSb. We also introduce four new sentence ranking evaluation benchmarks of informal texts, carefully extracted from the initial collections of tweets, proving not only that our best model learns classical Semantic Textual Similarity, but also excels on tasks where pairs of sentences are not exact paraphrases. Ablation studies reveal how increasing the corpus size influences positively the results, even at 2M samples, suggesting that bigger collections of Tweets still do not contain redundant information about semantic similarities. Code available at https://github.com/marco-digio/Twitter4SSE

## introduction and related work word-level embeddings techniques compute fixedsize vectors encoding semantics of words @xcite , usually unsupervisedly trained from large textual corpora. it has always been more challenging to build high-quality sentences-level embeddings. currently, best sentence-embeddings approaches are supervisedly trained using large labeled datasets @xcite @xcite @xcite @xcite , such as nli datasets @xcite or paraphrase corpora @xcite . round-trip translation has been also exploited, where semantically similar pairs of sentences are generated translating the non-english side of nmt pairs, as in paranmt @xcite and opusparcus @xcite . however, large labeled datasets are rare and hard to collect, especially for non-english languages, due to the cost of manual labels, and there exist no convincing argument for why datasets from these tasks are preferred over other datasets @xcite , even if their effectiveness on sts tasks is largely empirically tested. therefore, recent works focus on unsupervised approaches @xcite @xcite @xcite , where unlabeled datasets are exploited to increase the performance of models. these works use classical formal corpora such as openwebtext @xcite , english wikipedia, obtained through wikiextractor @xcite , or target datasets without labels, such as the previously mentioned nli corpora. instead, we propose a twitter-based approach to collect large amounts of weak parallel data: the obtained couples are not exact paraphrases like previously listed datasets, yet they encode an intrinsic powerful signal of relatedness. we test pairs of quote and quoted tweets, pairs of tweet and reply, pairs of co-quotes and pairs of co-replies. we hypothesize that quote and reply relationships are weak but useful links that can be exploited to supervisedly train a model generating high-quality sentence embeddings. this approach does not require manual annotation of texts and it can be expanded to other languages spoken on twitter. we train models using triplet-like structures on the collected datasets and we evaluate the results on the standard sts benchmark @xcite , two twitter nlp datasets @xcite and four novel benchmarks. our contributions are four-fold: we design an language-independent approach to collect big corpora of weak parallel data from twitter; we finetune transformer based models with triplet-like structures; we test the models on semantic similarity tasks, including four novel benchmarks; we perform ablation on training dataset, loss function, pretrained initialization, corpus size and batch size. ## datasets we download the general twitter stream collected by the archive team twitter foot_1 . we select english foot_2 tweets posted in november and december 2020, the two most recent complete months up to now. they amount to about 27g of compressed data (∼ 75m tweets). 4 this temporal selection could introduce biases in the trained models since conversations on twitter are highly related to daily events. we leave as future work the quantification and investigation of possible biases connected to the width of the temporal window, but we expect that a bigger window corresponds to a lower bias, thus a better overall performance. we collect four training datasets: the quote dataset, the reply dataset, the co-quote dataset and the co-reply dataset. the quote dataset (qt) is the collection of all pairs of quotes and quoted tweets. a user can quote a tweet by sharing it with a new comment (without the new comment, it is called retweet). a user can also retweet a quote, but it cannot quote a retweet, thus a quote refers to an original tweet, a quote, or a reply. we generate positive pairs of texts coupling the quoted texts with their quotes. the reply dataset (rp) is the collection of all couples of replies and replied tweets. a user can reply to a tweet by posting a public comment under the tweet. a user can reply to tweets, quotes and other replies. it can retweet a reply, but it cannot reply to a retweet, as this will be automatically considered a reply to the original retweeted tweet. we generate positive pairs of texts coupling tweets with their replies. the co-quote dataset (coqt) and co-reply dataset (corp) are generated respectively from the qt dataset and the rp dataset, selecting as positive pairs two quotes/replies of the same tweet. to avoid popularity-bias we collect only one positive pair for each quoted/replied tweet in every dataset, otherwise viral tweets would have been over-represented in the corpora. we clean tweets by lowercasing the text, removing urls and mentions, standardizing spaces and removing tweets shorter than 20 characters to minimize generic texts (e.g., variations of "congrats" are common replies, thus they can be usually associated to multiple original tweets). we randomly sample 250k positive pairs to train the models for each experiment, unless specified differently, to fairly compare the performances (in § 5 we investigate how the corpus size influences the results). we also train a model on the combination of all datasets (all), thus 1m text pairs. we show examples of pairs of texts from the four datasets in the appendix. ## approach we select triplet-like approaches to train a tranformer model on our datasets. we extensively implement our models and experiments using sentence-transformers python library foot_4 and huggingface @xcite . although the approach is model-independent, we select four transfomer models @xcite as pre-trained initializations, currently being the most promising technique (∼ 110m parameters): roberta base @xcite is an improved pre-training of bert-base architecture @xcite , to which we add a pooling operation: mean of tokens of last layer. preliminary experiments of pooling operations, such as max and [cls] token, obtained worse results; bertweet base (nguyen et al., 2020) is a bert-base model pre-trained using the same approach as roberta on 850m english tweets, outperforming previous sota on tweet nlp tasks, to which we add a pooling operation: mean of tokens of last layer; sentence bert @xcite are bert-base models trained with siamese or triplet approaches on nli and sts data. we select two suggested base models from the full list of trained models: bert-base-nli-stsb-mean-tokens (s-bert) and stsb-roberta-base (s-roberta). we test the two following loss functions: triplet loss (tloss): given three texts (an anchor a i , a positive text p i and a negative text n i ), we compute the text embeddings (s a , s p , s n ) with the same model and we minimize the following loss function: for each pair of anchor and positive, we select a negative text randomly picking a positive text of a different anchor (e.g., about the quote dataset, anchors are quoted tweets, positive texts are quotes and the negative texts are quotes of different quoted tweets); multiple negative loss (mnloss) @xcite : given a batch of positive pairs (a 1 , p 1 ), ..., (a n , p n ), we assume that (a i , p j ) is a negative pair for @xmath0.g., quote dataset: we assume that quotes cannot refer to any different quoted tweet). we minimize the negative loglikelihood for softmax normalized scores. we expect the performance to increase with increasing batch sizes, thus we set @xmath1, being the highest that fits in memory (see § 5 for more details). we train the models for 1 epoch foot_5 with adamw optimizer, learning rate 2 × 10 -5 , linear scheduler with 10% warmup steps on a single nvidia tesla p100. training on 250k pairs of texts requires about 1 hour, on 1m about 5 hours. ## evaluation we evaluate the trained models on seven heterogeneous semantic textual similarity (sts) tasks: four novel benchmarks from twitter, two well-known twitter benchmarks and one classical sts task. we planned to test the models also on twitter-based classification tasks, e.g., tweeteval @xcite . however, the embeddings obtained from our approach are not designed to transfer learning to other tasks, but they should mainly succeed on similarity tasks. a complete and detailed evaluation of our models on classification tasks is also not straightforward, since a classifier must be selected and trained on the top of our models, introducing further complexity to the study. we leave this analysis for future works. ## results and ablation study in as expected, we conclude that baseline models perform poorly in the new benchmarks, being trained for different objectives on different data, while our-bertweet (all) obtains the best performances. on established datasets, our training procedure improves the corresponding pre-trained versions. the only exception is when our model is initialized from s-bert and s-roberta and tested on turl, where we notice a small deterioration of performances (0.5 and 0.1 points respectively) and on stsb-test, since baselines where trained on stsb-train. this result proves that our corpora of weakly similar texts are valuable training sets and specific nli corpora are not necessary to train accurate sentence embeddings. we remark that for many non-english languages, models such as s-bert and s-roberta cannot be trained since datasets such as stsb-train do not exist yet foot_7 . the best initialization for novel benchmarks and pit is bertweet, being previously unsupervisedly trained on big amaunts of similar data, while for turl and stsb the best initializations are s-bert and s-roberta respectively. mnloss always produces better results than a simple triplet-loss, since the former compares multiple negative samples for each positive pair, instead of just one as in the latter. the training dataset does not largely influence the performance of the model on novel benchmarks, while, on enstablished benchmarks, qt and rp are usually better than coqt and corp training datasets. however, the concatenation of all datasets (all) used as training set almost always produces better results than when a single dataset is used. figure 1 (left) shows that performances improve by increasing the corpus size of qt dataset. since they do not reach a plateau yet, we expect better performances when a wider magnitude of tweets is collected. figure 1 (right) shows the performance of the same model when varying batch size in mnloss, i.e., the number of negative samples for each query. the performance plateaus at about 10, setting a sufficient number of negative samples. however, we set it to a higher value because it implies a faster training step. ## conclusions we propose a simple approach to exploit twitter in building datasets of weak semantically similar texts. our results prove that exact paraphrases, such as in nli datasets, are not necessary to train accurate models generating high-quality sentenceembeddings, since models trained on our datasets model dq cq dr cr avg pit turl stsb roberta-base 42.9 39.1 55.0 41.0 44.5 39.5 49.7 52.5 bertweet 46.9 42.5 56.7 44.1 47.5 38.5 48.2 48.2 s-bert 53.7 43.9 60.5 45.4 50.9 43.8 69.9 84.2 s-roberta 52.4 42.8 59. of weak pairs perform well on both enstablished and novel benchmakrs of informal texts. the intrinsic relatedness of quotes with quoted texts and replies with the replied texts is particularly useful when building large datasets without human manual effort. thus, we plan to expand the study to other languages spoken in twitter. two months of english data are more than enough to build large datasets, but the time window can be easily extended for rarer languages, as today more than 9 years of data are available to download. finally, we also hypothesize that this approach can be adapted to build high-quality embeddings for text classification tasks. we will extensively explore this on twitter-related tasks.