Analyzing and Improving Coherence of Large Language Models in Question Answering (2025)

Abstract: Large language models (LLMs) have recently revolutionized natural language processing. These models, however, often suffer from instability or lack of coherence, that is the ability of the models to generate semantically equivalent outputs when receiving diverse yet semantically equivalent input variations. In this work, we analyze the behavior of multiple LLMs, including Mixtral-8x7B, Llama2-70b, Smaug-72b, and Phi-3, when dealing with multiple lexical variations of the same info-seeking questions. Our results suggest that various LLMs struggle to consistently answer diverse equivalent queries. To address this issue, we show how redundant information encoded as a prompt can increase the coherence of these models. In addition, we introduce a Retrieval-Augmented Generation (RAG) technique that supplements LLMs with the top- k most similar questions from a question retrieval engine. This knowledge-augmentation leads to 4-8 percentage point improvement in end-to-end performance in factual question answering tasks. These findings underscore the need to enhance LLM stability and coherence through semantic awareness.

## introduction llms have revolutionized the majority of natural language processing tasks in the last years, including question answering (qa) @xcite , chatbot @xcite , coding @xcite , and summarization @xcite to name a few. one of their main limitations is the stability of their output when slightly changing the input. this is reported in several previous works @xcite , especially in the prompt engineering research @xcite . although these works clearly point out the problem, understanding its causes is rather challenging, as the stability of the model, in addition to the fine-tuning step, also depends on pre-training. the latter involves the usage of a huge amount of data, which is typically different in different llms, e.g., using different training techniques, and parameters. this has forced researchers to study the stability of models using a black-box approach: trying different prompts, mostly guided by human cognitive considerations, and observing the impact on the single output or overall performance. in this work, we analyze the coherence of multiple llms, including mixtral-8x7b @xcite , llama2-70b @xcite , phi3mini (3.8b) @xcite , smaug-72b @xcite , in multiple factual qa tasks. specifically, we show that large models fail to provide the same or similar answers for semantically equivalent questions as input. we conjecture that the instability, or lack of coherence, is a symptom of the model not being able to fully understand a request, i.e., accessing its parametric knowledge necessary for generating a correct answer. we show that redundant information encoded in the prompt, as multiple equivalent variations of the same question, can increase the accuracy of the model and its coherence, thus mitigating the understanding issue. intuitively, llms can exploit different semantic patterns from the equivalent questions to better connect the user's request with their parametric knowledge. we exploit this finding to design a principled approach to improve llm coherence and accuracy, leading to greater stability through retrieval augmented generation (rag) @xcite . in short, we built (i) an index of 38m of questions, and (ii) a dense retrieval engine, which, given a target question, retrieves semantically equivalent (or similar) queries. given a question asked to a general llm, our system first, retrieves k similar questions and then provides them to the llm along with the original request. in this work, we refer to this approach as question-rag (q-rag). our results show that all analyzed llms significantly benefit from our approach, with an improvement in both accuracy and coherence, measured as the semantic similarity of answers for equivalent questions. ## related work several llms of diverse size, ranging from a few to hundreds of billion parameters, pre-trained on web-scale corpora, have been recently introduced, e.g.: gpt family @xcite @xcite , llama @xcite , mixtral @xcite , smaug @xcite , falcon @xcite , or phi3-mini @xcite . these models can be successfully applied to various nlp tasks, e.g., question answering @xcite , chatbot @xcite , coding @xcite , and summarization @xcite , achieving high accuracy. however, their parametric knowledge shows some limitations, e.g., @xcite showed that llms, applied to machine reading tasks, may easily fail when the input context is not aligned with the internal knowledge. similar mechanisms produces lack of coherence of their results. for instance, @xcite showed that llms are vulnerable to option position changes in multiple-choice qa tasks. @xcite pointed out llm weaknesses with respect to adversarial attacks, e.g., when attempting to manipulate the output. finally, chatterjee et al. (2024) introduced a measure to quantify the sensitivity of an answer for a given prompt. our analysis of coherence is different from previous work as it is based on observing llm output when semantically equivalent questions are used in their input. moreover, we provide a principle approaches to mitigate this issue. ella rabinovich (2023) used equivalent questions in llms for a different purpose: they introduced popqa-tp, a meticulously curated dataset of 118k high-quality paraphrased questions. popqa-tp expands the original popqa @xcite , creating paraphrasing of each of the 14k initial questions. popqa-tp can be used for benchmarking llms' ability of maintaining semantic consistency across variations of the same question. however, the authors used popqa-tp for designing an automatic evaluation model. this predicts the llm's likelihood of answering a question correctly, using semantic consistency metrics, i.e., co-sine similarity between answer embeddings, with other predictors, such as question subject popularity and answer certainty. ## question-answer coherence we consider question-answer coherence as a proxy to quantify the ability of a llm of retrieving the knowledge necessary to correctly answer a question. intuitively, if a model can correctly answer a specific question, e.g., what is the capital of italy?, it can also answer other semantically equivalent questions, e.g., can you tell me what's the capital of italy?, or name of the city that serves as the capital of italy. conversely, if a model fails to answer a question, it will likely expected to fail equivalent questions, since it lacks the necessary knowledge. a different behaviour of the model indicates an underlying issue since failing to answer an equivalent question answered correctly suggests the model cannot access and use its knowledge. similarly to previous work (ella rabinovich, 2023), we measure the model coherence with respect a question, q, as the average similarity between the answers embeddings generated from a set of questions semantically equivalent to q. we can estimate model coherence on a set of different questions (each associated with different equivalence clusters) by averaging their coherence. formally, let q be a set of open-domain wellformed questions and let c ⊆ q be a set (or cluster) of questions such that ∀(q i , q j ) ∈ c 2 : q i ≡ q j , where ≡ indicates that two questions are semantically equivalent. we used the equivalence definition introduced by @xcite : two questions (q i , q j ) are semantically equivalent iff they have the same information-seeking intent and their answers can be interchanged, more formally: ∀ a : l(q i , a) ↔ l(q j , a), where l is a labeling function such that l(q, a) = 1 if the answer a is correct for q given a certain interpretation of correctness, -1 otherwise. given a model δ, and a set of m clusters {c r } m @xmath0, such that c @xmath1, .., q n }, q i ≡ q j ∀i, j, the coherence is defined as ## conclusions in this paper, we showed that popular llms have poor coherence when answering multiple lexical variations of the same questions. to mitigate the issue, we introduced a prompting technique that uses semantically equivalent questions, retrieved through a query retrieval framework, to improve the coherence the models. multiple experiments on various benchmarks suggest that our approach improves both the coherence of the models and their accuracy. our work highlights the limitations of popular llms and sets foundations for future research on coherence improvement.