Evaluation Datasets for Cross-lingual Semantic Textual Similarity (2021)

Abstract: Semantic textual similarity (STS) systems estimate the degree of the meaning similarity between two sentences. Cross-lingual STS systems estimate the degree of the meaning similarity between two sentences, each in a different language. State-of-the-art algorithms usually employ a strongly supervised, resource-rich approach difficult to use for poorly-resourced languages. However, any approach needs to have evaluation data to confirm the results. In order to simplify the evaluation process for poorly-resourced languages (in terms of STS evaluation datasets), we present new datasets for cross-lingual and monolingual STS for languages without this evaluation data. We also present the results of several state-of-the-art methods on these data which can be used as a baseline for further research. We believe that this article will not only extend the current STS research to other languages, but will also encourage competition on this new evaluation data.

## introduction recently, research in natural language understanding is moving beyond monolingual solutions @xcite @xcite . however, any solution needs some kind of evaluation data. the common source of evaluation data for semantic meaning comparison are the sts tasks from semeval workshop. the sts task has a long history at semeval workshops. since 2012 till 2017 the sts task has been held annually creating a considerable amount of evaluation datasets. however, most of these datasets are in english and spanish. this has led us to the creation of new crosslingual evaluation data for sts because all algorithms need verification e.g. on evaluation data. the evaluation data consist of pairs of sentences and the degree of their semantic similarity. this paper presents new sts datasets and thorough experiments using linear transformations for cross-lingual sts. we see four main contributions of our work: ## related work this section presents related datasets for sts with main focus on cross-lingual datasets. even though most of the datasets is in english and spanish, there are also available monolingual datasets in arabic and czech. the cross-lingual datasets are also quite focused on english and usually one side of the sentence pair is in english. in summary, the cross-lingual datasets are available in english paired with arabic, croatian, czech, italian, spanish, and turkish. more detailed overview of these datasets can be found further in this section. in section 3, we present datasets for both monolingual and cross-lingual sts in czech, english, french, and german. they provided five datasets (paraphrase sentences msrpar,video descriptions msrvid, automatically translated sentences mtnews and mteuroparl, and gloss pairs onwn) consisting of 2234 training sentence pairs, 3108 testing sentence pairs, and 6 trial pairs. glavaš et al. (2017) translated one side (750 sentences each) of two english monolingual datasets (msrvid and onwn) from semeval 2012 task 6 to spanish, italian, and croatian. • semeval 2013: agirre et al. (2013) continued with english sts core task. the whole dataset contains 2250 test and 20 trial new sentence pairs from four datasets (news headlines, mapping of lexical resources onwn and fnwn, machine translation evaluations smt). they also introduced typed similarity task for predefined types (e.g. author, location, subject and description). agirre et al. (2014) divided the task into two subtasks one for english (3750 sentence pairs) and the other for spanish. the english data contains image descriptions (images), news headlines (headlines), gloss pairs (onwn), news title and tweet comments (tweet-news), discussion forum and news (deft-forum and deft-news). the newly introduced spanish dataset contained 480 sentence pairs from wikipedia, 324 sentence pairs from google news and 65 trial sentence pairs. • semeval 2015: agirre et al. (2015) continued with both english (3000 test pairs, 70 trial pairs) and spanish (751 pairs) subtasks and newly introduced an interpretable sts subtask. the interpretation is evaluated on the alignments of sentence pairs. the english dataset contains image descriptions (image), news headlines (headlines), student and reference answers (answers-students), answers from exchange forums (answers-forum), and discussion forum comments (belief ) • semeval 2016: agirre et al. (2016) proposed two spanish-english datasets in se-meval 2016 task 1. one consists of news headlines news (301 sentence pairs) and the other contains sentences from multiple sources including news headlines, question-answering, plagiarism detection, etc. multi-source (294 sentence pairs). glavaš et al. (2017) translated spanish sentences from both datasets to italian and croatian. we translated (using google translate) and then manually checked both sides of 250 pairs from the english sts dataset from semeval 2017 (track5) into czech, french, and german. the translations were manually checked by two upper intermediate (b2) level speakers of the given language in case of french and german and native speakers of czech. we also asked them to preserve the meaning of the translation as much as possible in relation to the semantic similarity score. we assume that the translated pairs preserve the same semantic similarity score. the resulting dataset thus contains monolingual and cross-lingual datasets for all language pairs. the dataset consisting of 2000 sentences is available for research purposes at https://gitlab. com/tigi.cz/cross-lingual-sts . each file contains one sentence per line. the sts evaluation pair consists of the first and second part (sentence) each stored in a separate file. the semantic similarity score is located in the gold score file. the lines in all files correspond with each other. to get the evaluation data for e.g. en-cs load the file sts.2017.input.track5.en.first.txt and sts.2017.input.track5.cs.second.txt and the gold standard in file sts.2017.gs.track5.first-second.txt. this new resource consists of four monolingual datasets and twelve cross-lingual datasets. ## experiments we follow brychcín (2020) who used bilingual dictionaries and a new transformation for word level semantic representations which reduces hubness in semantic spaces. he also evaluated his methods on several sts datasets including cross-lingual. the transformations of semantic spaces and combinations of word representations are too complex and beyond the scope of this short paper, for a thorough description of these methods please refer to the original publication. in the replicated experiments on new datasets we use monolingual semantic spaces transformed into a unified space using bilingual dictionaries. sts performance is measured by the pearson correlation between automatically estimated scores and human judgments. our experiments start with building monolingual semantic spaces for each of tested languages, namely, czech (cs), german (de), english (en), and french (fr). for all languages we use character-n-gram-based skip-gram model @xcite pre-trained on wikipedia foot_0 . for each language, we construct the vocabulary from 300k most frequent words. we estimate idf weights on the wikipedia corpus for every language. each wikipedia article represents a document. the bilingual dictionaries between each pair of languages are created from the 20k most frequent words in the corpus using google translate. the global post-processing techniques for semantic spaces used by brychcín (2020) consist of two steps column-wise mean centering and word vector normalization to unit vectors. this guarantees that all word pairs in the dictionary contribute equally to the optimization criteria of the linear transformation. we always apply this postprocessing for both semantic spaces before the linear mapping. ## conclusion and future work in this paper we presented new sts datasets for both cross-lingual and monolingual sts and provided them to the research community. we extended experiments of previous work on sts using linear transformations to create cross-lingual semantic spaces, by conducting initial experiments on the newly created datasets. we confirmed the findings of brychcín (2020) by replicating three (previously published) approaches to combine information from word representations. the used sts system does not require sentence similarity supervision and the only cross-lingual information is a bilingual dictionary. in the future, we intend to investigate the use of unsupervised methods to create the bilingual dictionary.