G i CCS : A G erman in-Context Conversational Similarity Benchmark (2022)

Abstract: The Semantic textual similarity (STS) task is commonly used to evaluate the semantic representations that language models (LMs) learn from texts, under the assumption that good-quality representations will yield accurate similarity estimates. When it comes to estimating the similarity of two utterances in a dialogue, however, the conversational context plays a particularly important role. We argue for the need of benchmarks specifically created using conversational data in order to evaluate conversational LMs in the STS task. We introduce GiCCS, a first conversational STS evaluation benchmark for German. We collected the similarity annotations for GiCCS using best-worst scaling and presenting the target items in context, in order to obtain highly-reliable context-dependent similarity scores. We present benchmarking experiments for evaluating LMs on capturing the similarity of utterances. Results suggest that pretraining LMs on conversational data and providing conversational context can be useful for capturing similarity of utterances in dialogues. GiCCS will be publicly available to encourage benchmarking of conversational LMs.

## introduction the semantic textual similarity (sts) framework is typically used for the extrinsic evaluation of nlp models @xcite , and in particular for language models (lms): if a model can successfully estimate the similarity between sentences, it's a good sign that it has learned good-quality semantically meaningful representations. in natural language generation, sts can provide a useful alternative to word overlap measures to analyse system output similarity @xcite . sts has been used both for training * these authors contributed equally to this work. @xcite and for evaluating lms @xcite . to the best of our knowledge, the majority of sts benchmarks have been created from written language resources using non-conversational data, and are mainly in english. conversational data, however, has some peculiarities which make it potentially challenging for the sts task: (1) questions and requests are frequent, (2) whether two sentences are semantically similar may depend on pragmatic factors triggered by the conversational context. for example, it may be challenging for human annotators to assess that could you turn it up a bit? and i'd like the ac to be colder advance the conversation in a similar way, without (1) sufficient conversational context and (2) an operative definition of what it means for a question and a declarative sentence to be semantically similar. moreover, sts datasets are typically annotated on a rating scale, which may lead to inconsistencies in annotation, scale region bias, and fixed granularity issues @xcite . in this paper, we introduce giccs, the first german in-context conversational benchmark for evaluating lms on the sts task. giccs is a multidomain dataset containing 300 items, each consisting of a domain name, a multi-turn german dialogue context (i.e., dialogue history) where the last utterance is paired with a target utterance, and a semantic similarity score between the paired utterances. giccs contains data from german dialogue resources as opposed to machine-translated data (e.g. the glue benchmark for sts; @xcite . the similarity scores were crowdsourced using the best-worst scaling (bws) annotation technique @xcite , as it was shown to address the limitations of the rating scales technique @xcite . the dialogue history (the previous 3 or 5 turns) was presented to the crowd-workers during crowdsourcing for better similarity judgements based on the conversational context. adopting bws in order to overcome the limitations of rating scales led to a high inter-annotator agreement with an overall krippendorff's α of 0.74. furthermore, we present a reliability study of the similarity scores obtained from the bws annotations and based on the conversational context. lastly, we present benchmarking experiments to evaluate different lms on the sts task using giccs. experiments show that pre-training lms on conversational data is beneficial for capturing conversational representations in downstream tasks and conversational applications, such as dialogue systems. giccs has a wide range of further applications, such as the evaluation of dialogue generation models, answer selection and ranking systems, and question answering based on dialogue history. to the best of our knowledge, the majority of the benchmarks have been mainly created from written language resources. this results in a sub-optimal benchmark for the evaluation of conversational language models in dialogue systems. ## score reliability study we now provide an analysis of the similarity scores obtained with the bws and of their reliability. ## conclusion we introduce giccs, a first german conversational sts benchmark for evaluating language models on semantic similarity. each item in the benchmark consists of a domain name, a multi-turn dialogue history, a target utterance paired with the last utterance in the dialogue, and a similarity score between the paired utterances. we leveraged two german dialogue resources, bas smartkom and crowdss, to collect our data as opposed to machine-translated data. annotations were crowdsourced using the best-worst scaling technique, which shows a high inter-annotator agreement with an overall krippendorff's α of 0.74 and reliable annotations with an average split-half reliability score of 0.973 for bas smartkom and 0.949 for crowdss. moreover, the dialogue history was presented to the crowdworkers for better similarity judgements based on the conversational context. final similarity scores cover a wide range of similarities between 0 and 1 introducing a challenge for language models to identify the nuances in semantics of different utterances. moreover, as the last utterance of each dialogue is paired with five different utterances each with its similarity score, giccs can be used for evaluating ranking systems. results of evaluating lms on the sts task using giccs shows that pre-training lms on conversational data brings benefits for the lms on meaning-ful representations of conversations. overall, giccs is a useful resource for evaluating conversational models on capturing similarity in conversational data. moreover, due to the lack of enough resources for evaluating conversational models in non-english languages, we hope that the annotation procedure described in this work will foster an interest in creating more reliable and high-quality resources similar to giccs.