Predicting Question-Answering Performance of Large Language Models through Semantic Consistency (2023)

Abstract: Semantic consistency of a language model is broadly defined as the model’s ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community.We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction – predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.

## introduction consistency of a model is broadly defined as the invariance of its behavior under meaning-preserving variations of its input @xcite . clearly, consistency is a highly desirable property of large language models, increasing their safety, robustness and trustworthiness. here we address the question of factual consistency of llms in the context of open-domain zero-shot factual question answering. as a concrete example, a consistent model will produce the same answer for the set of questions {"what is stevie cameron's occupation?", "what job does stevie cameron do?", "what does stevie cameron earn a living as?"}. a model's consistency metric is defined to be agnostic to answers' accuracy @xcite , meaning that semantically-equivalent (possibly incorrect) outputs are qualified as consistent. as such, while the correct answer to the questions above is "journalist", three other identical answers (e.g., "politician") will score as perfectly consistent. semantic consistency of masked language models (mlms) has been studied by @xcite , who inspected masked tokens as predicted by encoder models, for alternations of word tuples, using a dataset of factual statements and their crowd-sourced paraphrases, specifically tailored for working with mlms. @xcite evaluated semantic consistency of decoder models for the task of non-factual question answering, experimenting with a range of consistency metrics. the authors automatically generated paraphrases for questions in the truthfulqa dataset @xcite , and scored a model's consistency as its robustness to paraphrases. however, the sub-optimal quality of automatic paraphrases, along with open and often lengthy nature of answers to questions, foot_0 as well as multiple (occasionally semantically diverse) reference answers, challenge benchmarking of llms' qa consistency using truthfulqa. a benchmark dataset for measuring the robustness of llms to paraphrases in the context of factual qa should satisfy two desirable properties: (1) strictly semantically-equivalent question paraphrases, and (2) questions that call for single short (possibly multi-word) answer, facilitating accurate evaluation. using the recently introduced popqa dataset with over 14k factual questions @xcite , we create its carefully curated extended version-popqa-tp (popqa templated paraphrases)-where 3-10 manually-created alternations were appended for each original question. the final dataset comprises over 118k questions, while preserving metadata (e.g., reference answers) from the original popqa. we further use this dataset for benchmarking factual semantic consistency of multiple encoder-decoder and decoder-only llms. the dataset is made available for the community at https://huggingface.co/ datasets/ibm/popqa-tp. we next demonstrate that robustness to question paraphrases correlates with a model's answer correctness for the given question. practically, this finding means that semantic consistency score is predictive of a model accuracy. combining this predictor with additional metrics suggested in prior work as correlating with llm qa correctness, we perform a comprehensive regression analysis of the predictive power of various metrics on the model's accuracy, as well as interactions between those metrics. moreover, we show that the developed framework can be used for predicting the likelihood of a language model to accurately answer a factual question. collectively, these results pave the way for the extremely challenging, yet highly important, task of question-answering performance prediction, a reference-less evaluation of qa performance, in the absence of ground-truth answers. the contribution of this work is, therefore, twofold: first, we introduce and release a large extension of the popqa dataset (popqa-tp), with high-quality paraphrases, that can be used for benchmarking qa semantic consistency of llms. second, we develop a prototype model for qa performance prediction, allowing for comparative analysis of various metrics, and demonstrating predictive power much higher than baselines. ## dataset benchmarking semantic consistency of llms requires high quality question alternations, eliminating possible confounds that stem from issues in automatic paraphrase generation. despite the immense advances in paraphrasing models during the past few years (e.g., @xcite @xcite , automatic tools still occasionally produce paraphrases that are not meaning-preserving (e.g., "who is the vocalist of 'perfect'?" for the original question "who is the composer of 'perfect'?"), incomplete (e.g., "who is the vocalist of 'perfect'? shape of you"), or violate, albeit infrequently, grammatical rules (e.g., "tap water's safe drinking?" as a paraphrase of "is tap water safe to drink?"). aiming at a high-quality benchmark dataset, we opted to manually construct paraphrase templates specific to each question category in popqa, as detailed below. ## benchmarking semantic consistency we next use popqa-tp, our dataset of manuallyconstructed paraphrase templates for assessing the semantic consistency of multiple contemporary llms. we report both models' accuracy (the ratio of correct answers to questions), as well as their consistency (robustness to question alternations), and further develop hypothesis about the correla-tion of semantic consistency and correctness. ## qa performance prediction we next define and address the task of factual question-answering performance prediction. here we rely on some parallels to the task of query performance prediction (qpp) in ir (search) systemsan established research area @xcite @xcite . qpp is defined as the assessment of the retrieval quality of a search system for a query, without relevance judgments. core differences exist between ir and llm-based systems used for the task of open-domain factual qa; yet, we address a conceptually similar task: assessment of a system's potential answer quality (that is manifested by its correctness) for a question, without relying on ground-truth answers. casting the task as a classification scenario, we train a logistic regression model, where several regressors-variables proven to correlate with llms correctness-carry over predictive power on the outcome variable: the model's likelihood to produce a correct answer for a given question. following @xcite , we define our first predictor-question subject popularity (spop)-as the mean number of monthly views of the subject entity's wikipedia page. in the popqa datset, the spop score varies from 2 to over 15m. ## conclusions we explore the robustness of llms to paraphrases in the context of open-domain zero-shot qa. introducing a large and carefully-curated extension of the popqa dataset (popqa-tp), with high-quality paraphrases, we first benchmark the semantic consistency of diverse llms; next, we develop a framework for qa performance prediction, incorporating semantic consistency, as well as additional aspects, shown to correlate with model's qa accuracy. collectively, our work shows that a model's ability to answer a question accurately can be reliably predicted, in a reference-less setting. our future work includes the exploration of how the semantic consistency metric used in this work can be adapted to additional generative tasks with long(er) answers, e.g., summarization, dialogue.