Text Encoders Lack Knowledge: Leveraging Generative LLM s for Domain-Specific Semantic Textual Similarity (2023)

Abstract: Amidst the sharp rise in the evaluation of large language models (LLMs) on various tasks, we find that semantic textual similarity (STS) has been under-explored. In this study, we show that STS can be cast as a text generation problem while maintaining strong performance on multiple STS benchmarks. Additionally, we show generative LLMs significantly outperform existing encoder-based STS models when characterizing the semantic similarity between two texts with complex semantic relationships dependent on world knowledge. We validate this claim by evaluating both generative LLMs and existing encoder-based STS models on three newly-collected STS challenge sets which require world knowledge in the domains of Health, Politics, and Sports. All newly-collected data is sourced from social media content posted after May 2023 to ensure the performance of closed-source models like ChatGPT cannot be credited to memorization. Our results show that, on average, generative LLMs outperform the best encoder-only baselines by an average of 22.3% on STS tasks requiring world knowledge. Our results suggest generative language models with STS-specific prompting strategies achieve state-of-the-art performance in complex, domain-specific STS tasks.

## introduction the nlp community has seen a rapid advancement in many areas since the onset of large language models (llms) trained using reinforcement learning with human feedback, including text summarization, machine translation, and problem solving, amongst others @xcite . one area that has not been well explored is the applicability of generative llms to semantic textual similarity (sts) tasks. in recent works, it has been explicitly suggested that llms are not well-suited for the sts-b task. in @xcite they support this claim by showing chatgpt is inferior to pre-trained roberta models on a small (@xmath0. in @xcite , they suggest that sts-b, and more generally regression tasks, have "no use case" in the context of llms -citing the extreme misalignment between llm training and the prediction of a continuous value. in this study, we aim to show that there are two intuitive reasons as to why llms are highly applicable to semantic textual similarity. 1) world knowledge: llms do not rely on human-labeled data, allowing them to be exposed to a broad range of world knowledge. very little human-annotated domain-specific data exists for direct sts training or contrastive learning of sentence embeddings @xcite , making applications of text encoders to niche domains challenging. thus, if we can apply llms to sts, we may greatly expand the set of problem domains where sts is impactful. 2) sts regression may align with language modeling: the sts task can be formulated such that the output space is constrained to prediction of a continuous value between @xcite . such a formulation reduces the task to outputting similarity as a percentage (e.g. text a and text b are 60% similar). during pre-training, llms are very likely to see many texts that use percentages in various contexts, as humans frequently cite percentages in natural language. thus, when we combine llms strong pairwise textual reasoning capabilities with their predisposition to percentages in natural language -llms appear well-suited to the sts task. a limitation of using llms for sts is they can be highly expensive and inefficient. for example, sts models are often used in information retrieval, where the goal may be to compare a query text to a large number of documents and then rank the documents based on their similarity to the query @xcite . it may not be viable to leverage generative llms for such a task in production, figure 1 : comparing the performance of chatgpt vs a roberta-based sts cross encoder on a sample from our sts-sports challenge set. this sample requires significant world knowledge as proper inference requires knowing 1) that the cowboys nfl team are often referred to as "america's team" and 2) that "recovering" an onside kick is equivalent to "getting the ball back" with an onside kick. the prompt corresponds to our best-performing chatgpt 0-shot prompt found in as text generation can suffer from low throughput and high cost. however, there are many small-scale tasks in academic settings where the poor efficiency of llms for sts are often of lesser concern. in the literature, we find small-scale applications of sts in the fields of psychology @xcite , community question answering @xcite , computational social science @xcite , and propaganda detection @xcite which use generic text encoders for knowledge-intensive/domain-specific problems. in this study, we aim to show that llms are more well-suited than generic text encoders for such tasks. we confirm our intuition that llms like chat-gpt are well-suited to perform sts by conducting the first thorough exploration of sts in the context of text generation. we evaluate two llms (i.e., chatgpt, llama2) for sts in the context of both existing sts benchmarks and domain-specific sts challenge sets. our work identifies sts-specific prompting strategies that significantly outperform prompts from prior works @xcite . specifically, we find that mapping the original [0-5] similarity scale used in sts benchmarks to be between [0-1] significantly improves performance of llms on the sts task. in other words, asking llms to infer similarity as a percentage improves performance vs. asking llms to utilize an arbitrary scale. see figure 1 for an example sts prompt used in this study. on existing benchmarks, we find that a 0-shot chatgpt pipeline provides sota performance on the sts13 and sts15 datasets, with near-sota performance on sts14 and sick-r (i.e. 0.45% and 0.51% difference in correlation respectively) when compared to unsupervised sota models. given the opaque nature of chatgpt's training data, we confirm our results are not the result of memorization by collecting 3 new sts challenge datasets using texts written after may 2023 across three domains: health, sports, and politics. we develop each dataset such that similarity is difficult to quantify without significant world knowledge and demonstrate that chatgpt provides sota performance for challenging domain-specific sts. a summary of our contributions is as follows: ## where does chatgpt fail on sts? in this section, we analyze the top 500 predicted samples from chatgpt with the largest absolute difference between prediction and ground truth across five sts datasets in the 0-shot setting (sts 12-16 ). we aim to surface the types of text pairs ill-suited for semantic similarity modeling with chatgpt. ## conclusion in this study, we show that while smaller llms like llama2 struggle on sts, larger models like chat-gpt are highly capable of performing semantic similarity tasks, as it achieves sota performance on 2/7 standard sts datasets. we additionally show that chatgpt is far superior to existing sts models on world knowledge-dependent comparisons -as chatgpt outperforms existing models by an average of 22.3% on domain-specific sts tasks. in conclusion, chatgpt shows promising results for domain-specific sts tasks.