The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answering (2023)

Abstract: Large language models are known to produce output which sounds fluent and convincing, but is also often wrong, e.g. “unfaithful” with respect to a rationale as retrieved from a knowledge base. In this paper, we show that task-based systems which exhibit certain advanced linguistic dialog behaviors, such as lexical alignment (repeating what the user said), are in fact preferred and trusted more, whereas other phenomena, such as pronouns and ellipsis are dis-preferred. We use open-domain question answering systems as our test-bed for task based dialog generation and compare several open- and closed-book models. Our results highlight the danger of systems that appear to be trustworthy by parroting user input while providing an unfaithful response.

## introduction with the advent of large language models (llm), question answering systems have become opendomain and conversational, meaning that they are able to generate fluent and informative responses to questions about nearly any topic and over several turns @xcite . however, these systems are also known to produce factually incorrect statements, commonly referred to as hallucinations @xcite . these two properties taken together require the system as well as the user to ensure that they mutually understand each other -a process also known as conversational grounding @xcite . empirical studies of dialogue have shown that people use different kinds of context-dependent linguistic behavior to indicate grounding, including use of fragments, ellipsis and pronominal reference @xcite . other studies show that lexical alignment in a response, i.e. repeating and adopting the interlocutor's lexical items (pickering and @xcite , can play a similar role, see examples in figure 1 . ## study of trust so far we have established that lexically aligned responses coming from gpt-3 variants are not necessarily faithful. the surface form seems to negatively affect users' preferences, obviating their need to check the supporting source, and creating a risk of placing trust to an imperfect system. with this experiment, we investigate a more general trend between linguistic phenomena and user trust. human evaluation experiment annotators are presented with the dialogue only, and are asked to choose the response they trusted more from two possible responses, or none. going beyond just lexical alignment, we selected 15 pairs of responses foot_5 , for every linguistic phenomenon in section 2.2. we modified responses to ensure each specific phenomenon was the only difference between them. we collected 20 preferences for each response pair. results ## conclusions we investigated the performance of different models on the task of ocqa, measuring faithfulness and lexical phenomena. automatic metrics highlighted how gpt-3 variants are less faithful than dpr+fid, as confirmed by annotators in the faithfulness judgment task. we conducted a study on conversational grounding phenomena and a preference task, whose significant results demonstrated an effect of surface form in human preferences towards the more conversational gpt-3, even when unfaithful. another experiment confirmed trust as being effected by high lexical alignment.