Language Models in Dialogue: Conversational Maxims for Human- AI Interactions (2024)

Abstract: Modern language models, while sophisticated, exhibit some inherent shortcomings, particularly in conversational settings. We claim that many of the observed shortcomings can be attributed to violation of one or more conversational principles. By drawing upon extensive research from both the social science and AI communities, we propose a set of maxims – quantity, quality, relevance, manner, benevolence, and transparency – for describing effective human-AI conversation. We first justify the applicability of the first four maxims (from Grice) in the context of human-AI interactions. We then argue that two new maxims, benevolence (concerning the generation of, and engagement with, harmful content) and transparency (concerning recognition of one’s knowledge boundaries, operational constraints, and intents), are necessary for addressing behavior unique to modern human-AI interactions. We evaluate the degree to which various language models are able to understand these maxims and find that models possess an internal prioritization of principles that can significantly impact accurate interpretability of the maxims.

## introduction modern language models aim for interaction that is as natural as possible. to help achieve this, pre-trained models undergo a refinement process, termed supervised fine-tuning, which involves updating the model's weights using human-labeled data. apart from helping to filter out some undesirable characteristics, this process allows the models to better capture the nuances of human language and conversation, improving their ability to understand context and respond in a way that is more aligned with how humans communicate. while fine-tuning does greatly improve the conversational capability of language models, current approaches can cause some undesirable properties to emerge. the processes of instruction tuning and reinforcement learning from human feedback (rlhf) encourage models to provide an answer at all costs (agüera y arcas, 2022), even when seeking clarification would improve response quality. models rarely say "i don't know" which can lead to unrelenting "helpfulness" where the model enters cycles of incorrect suggestions/responses @xcite . additionally, there is growing evidence that instruction tuning increases model sycophancy, i.e., the tendency for model outputs to mirror or agree with input bias @xcite @xcite @xcite . our paper is based on the claim that many of the observed shortcomings of current models can be attributed to violation of one or more conversational principles. for instance, relating to clarification, humans often partake in "collective acts" to form common ground, build trust, and avoid misunderstandings @xcite . the behavior of current models providing an answer even when uncertain fails in both this respect and the maxim of relevance, a cornerstone of grice's original maxims for describing effective human communication @xcite . additionally, sycophancy can be viewed, in part, as a failure of the need to "repair" misunderstandings @xcite . the main focus of our paper is to define the set of conversational principles, or maxims, that are in violation when models exhibit these deficiencies. this paper makes the following contributions: (1) we propose a set of prescriptive maxims for analyzing human-ai conversations; (2) as part of the construction of our maxims, we provide a review of both desirable characteristics of human conversation (from the social sciences community) and the current undesirable properties of modern language models (from the ai community). our proposed maxims provide a taxonomy for what constitutes effective communication between a human user and an ai assistant; and (3) we find that various models maintain an internal prioritization of the maxims, which can significantly affect their ability to accurately interpret the maxims. ## related work our work concerns the development of principles, or maxims, that aim to quantify good interaction in human-ai conversations. we offer a review of conversational analysis, from both the social science and ai communities, and discuss some of the associated measures of effective conversation. ## maxims for human-ai conversations grice's maxims were developed for symmetric and cooperative conversations, i.e., two human participants engaged in cooperative discourse. while conversations between humans and ai can be justified as being cooperative, they are clearly not symmetric. however, we claim that holding both humans and ai to the same set of conversational maxims establishes common ground and increased predictability, in turn encouraging more natural discourse. importantly, while we define a uniform set of maxims, the methods by which each participant -human or ai -complies with the maxims may differ, as detailed further in section 5. we propose an augmented set of maxims -quantity, quality, relevance, manner, benevolence, and transparency -to assess conversations between humans and ai. we first argue that grice's original maxims (quantity, quality, relevance, and manner) -with some rephrasing to improve their prescriptiveness -are necessary in human-ai conversations. we then assert that two new maxims, benevolence and transparency, are essential in addressing some of the shortcomings inherent to current human-ai interactions. throughout the discussion of the maxims, the reader is directed to appendix a for example conversations that illustrate the violations. quantity. the maxim of quantity relates to the amount of information contained in a given response. to satisfy quantity, a response should meet the following requirements: the need for the maxim of quantity in human-ai conversations is primarily motivated by the tendency of language models to produce "overly wordy responses" in an attempt to "give the impression of expertise" @xcite . this behavior is known to arise due to reward model over-optimization, i.e., over-fitting an imperfect model of human preferences @xcite . the maxim of quantity specifies that responses should contain an appropriate amount of information, a definition that necessarily contains some subjectivity (given the terms "sufficient" and "unnecessary"). the appropriate level of detail generally cannot be objectively evaluated without taking into account the conversational context. information in a dialogue is revealed over multiple turns as participants request (and provide) information. this revealed context defines the needs of the conversation, allowing a given response to be evaluated by judging if the additional information provided by the response is adequate for these needs. for instance, too little information may lack sufficient detail and lead to misunderstandings, while too much information may cause confusion or detract from the primary request. fig. 2 in appendix a.1 provides an evaluation of quantity for a sample conversation. quality. the maxim of quality pertains to the truthfulness and honesty of the response, specifically: one of the most significant issues in current language models is their tendency to hallucinate @xcite @xcite . hallucinations in language models are primarily understood to arise from both data issues -incorrect referencing @xcite and inappropriate data handling @xcite -and training issues -imperfect representations @xcite , erroneous decoding @xcite , and a variety of knowledge biases @xcite . the maxim of quality imposes requirements on the correctness of responses, importantly distinguishing between the notions of truthfulness and honesty. truthfulness describes an external consistency between a response and evidence/facts, whereas honesty concerns an internal consistency between a response and the speaker's knowledge. this latter requirement necessarily endows the speaker with a belief which, when evaluating the honesty of an ai's response, raises some questions as to what it means for an ai to possess a "belief." earlier discussions on this topic @xcite argue that ai sytems should be evaluated with respect to truthfulness rather than honesty in order to avoid the issues that come with assigning beliefs to ai. however, recent research demonstrates that llms possess a type of belief, or "internal state", which can be used to evaluate the honesty of a model @xcite @xcite @xcite . 2 we argue that as language models become more sophisticated (e.g., gpt-4+), their evaluation with respect to more human characteristics becomes more appropriate. 3 thus, we impose both truthfulness and honesty on the conversation participants. given that truthfulness concerns the consistency between a response and facts, it can be evaluated externally, i.e., solely in terms of responses. while evaluating honesty externally is possible in some situations, e.g., via consistency checks with "humanly verifiable rules" @xcite , it generally requires access to, or some estimate of, the internal state of the speaker @xcite @xcite . in other words, evaluating honesty inherently imposes some requirements on the speaker. see fig. 3 (appendix a .2) for an illustration of quality. relevance. the maxim of relevance is given by the following requirements: grice states that relevance requires a participant's "contribution to be appropriate to the immediate needs" of the conversation @xcite . 4 as discussed earlier, the needs of a conversation are defined by the conversational context. the requirement to be helpful to the other participant's statements, as specified by our first requirement, means tending to these needs. furthermore, being able to respond relevantly requires correctly interpreting the conversational context. if a participant is uncertain about the meaning of the other participant's statement(s), then these needs will be unclear. in this sense, our first requirement implicitly requires that participants seek clarification in order to avoid any misunderstandings. in the event that any misunderstandings do emerge, participants should aim to "repair" them to regain a correct interpretation @xcite . these grounding acts (e.g., seeking clarification and repairing misunderstandings) have long been understood in the social sciences as being crucial to effective human communication @xcite @xcite , and more recently as desirable qualities for conversational ai @xcite . the second requirement specifies that responses should remain on topic. what constitutes being "on topic" is again dictated by the conversational context @xcite . importantly, this requirement does not require that the topic of the dialogue stays fixed throughout a conversation. as stated by @xcite , "presuppositions can be created or destroyed in the course of a conversation" allowing for situations where "subjects of conversation are legitimately changed" @xcite . in this sense, our requirement interprets a conversation's topic as the current subject that both parties have mutually recognized as the focus of discussion. fig. 4 (appendix a .3) presents an illustration of relevance. manner. the maxim of manner imposes two requirements: the maxim of manner is closely related to the linguistic notion of coherence, a property that ensures the logical flow and clarity of discourse @xcite . while recent language models have improved significantly in maintaining coherent responses, they still face issues in long or complex dialogues @xcite . one reason for this is due to limited recall, i.e., not being able to retrieve information relevant to the conversational thread. despite modern language models' impressive context lengths (openai, 2023; google, 2024), recent tests from the ai community illustrate significant variability in recall ability, especially for content near the middle of the context window @xcite . the maxim of manner (particularly clarity) may initially appear very similar to the maxim of quantity. however, as stated by @xcite , quantity evaluates if a response contains the required "units of information" whereas manner evaluates how efficiently these units of information are conveyed. it is possible to violate quantity while satisfying manner -by providing information that is clear, accessible, and well-organized, but either too much or too little for what is required -or violate manner while satisfying quantity -by providing the correct amount of information, but in a way that is confusing or disorganized. the second requirement specifies that responses should use "appropriate language" where the definition of appropriate is defined over the course of the interaction. satisfying manner thus requires some understanding of the other participant's desired conversational style. benevolence. the maxim of benevolence aims to capture the moral responsibility of a response. the requirements of benevolence are: benevolence is the first of our new maxims for evaluating human-ai conversations. broadly, the requirements of benevolence specify that a response should neither produce harm nor engage with harm, some of the properties that instruction tuning aims to induce @xcite . the first requirement indicates that responses be civil: addressing responses that range from being impolite or exhibiting an annoyed tone to demonstrating cultural insensitivities, prejudice, and racism. this requirement is closely related to politeness, widely recognized as being essential for productive hu-man communication @xcite @xcite . additionally there is evidence that politeness plays a significant part in establishing trust @xcite @xcite . the second requirement specifies that a participant shouldn't engage with any request that is harmful or unethical. many of the examples of such violations in human-ai interactions arise due to malicious users trying to exploit the extensive knowledge of an ai to induce it to divulge harmful content, either for robustness analyses or simply for entertainment @xcite @xcite . transparency. the maxim of transparency imposes the following requirements: the maxim of transparency, the last of our maxims, is tailored specifically to the nature of human-ai conversations. the first requirement is motivated by a shortcoming of many current language models: their hesitancy to say "i don't know." this behavior is largely due to the fine-tuning process. for instance, when instruction tuning google's lamda model, statements like "i don't know" in response to the question "what is your favorite island in the world?" often received "negative feedback" from human labelers (agüera y arcas, 2022). however, such a response is objectively accurate and should be rewarded highly as language models do not possess preferences (at least in the same way as humans). the process of human labeling thus causes human preferences to be embodied in the ai, in turn rewarding behavior that sounds more human, even if inaccurate. our first requirement aims to address this issue by requiring that the speakers recognize their knowledge boundaries (in this case the lack of evidence and experience for possessing a personal preference). adhering to knowledge boundaries also requires that speakers recognize any lack of recency in their information (e.g., current language models, unlike humans, only have access to information up to their training cutoff), as well as any limitations in experience (language models, unlike humans, do not possess personal opinions or emotions), domain knowledge (language models are not permitted to provide medical, legal, or financial advice), and context (language models do not possess information about the user beyond what was revealed in the dialogue). the second requirement, regarding operational capabilities, concerns the speaker recognizing any of their real-world (physical) constraints. for instance, language models cannot act as an embodied entity (e.g., execute physical actions), and should not use language indicating that they can. lastly, motivated by observations that language models "avoid certain topics" @xcite , the third requirement specifies that speakers should be open about any unwillingness to discuss topics or consider relevant advice. this property has also been described as "evasiveness" in the literature @xcite . given the above requirements, there are numerous ways in which the maxim of transparency can be violated. a simple example is presented in appendix a.6, fig. 7 . ## operationalization a key question is how the proposed maxims can be integrated, or operationalized, into current language models. while our paper argues that many of the observed issues of current language models can be attributed to violation of one or more of the maxims, we emphasize that we do not claim that every maxim must be satisfied in every conversation in order to be deemed a "good" conversation. rather the proposed maxims serve as dimensions for analyzing human-ai conversations. some maxims may be less important, or even not important at all, in some contexts @xcite . therefore, instead of studying a model's ability to satisfy the maxims, which would likely lead to relatively noisy and inconclusive results simply due to the sheer variability of contexts @xcite , operationalization should instead focus on a model's ability to interpret the maxims. we argue that a model's ability to accurately interpret the maxims is a necessary precondition for satisfying them if the particular context demands it. to estimate a language model's interpretability of the maxims, we use the model to label conversations with respect to the maxims. source conversations are sampled (@xmath0, 50 of which were handlabeled to provide ground-truth. each model was instructed to generate a score (i.e., label) and an explanation for the score. we analyze three models in our experiments: llama-3-8b-instruct, llama-3-70b-instruct, and mixtral-8x7b-instruct-v0.1. 5 for the purposes of our experiments, we omit the second submaxim of quality (due to complexities of evaluating a model's honesty) resulting in 12 submaxims total (denoted @xmath1. additional details of the experimental setup can be found in appendix b. given each model's set of generated labels, the analysis of a model's interpretability of the maxims is decomposed into evaluating: 1) the accuracy of the model's labels, and 2) the model's implicit prioritization of the submaxims. to this end, we study the pattern of submaxim violations, that is, which individual submaxim violations tend to be associated with other submaxim violations. this violation pattern is constructed for each model by subdividing the model's set of labels in multiple (12) splits, where each split extracts the conversations in which the model labeled a given response with a (majority) violation for the corresponding submaxim (with all other submaxim dimensions unconstrained). fig. 1 illustrates the violation pattern for llama-3-70b-instruct (see appendix c for the violation patterns for llama-3-8b-instruct and mixtral-8x7b-instruct-v0.1). we can see from fig. 1 , that llama-3-70b-instruct places a high importance on relevance_2 (i.e., topical relevancy), given that its violation tends to be associated with violations of quantity_1, quantity_2, quality_1, and relevance_1 (i.e., direct helpfulness). analyzing the label accuracy for each split, we can see that llama-3-70b-instruct is most accurate in labeling the submaxims of benevolence (see accuracies for benevolence_1 (harm prevention) and benevolence_2 (harm reduction) across splits) and least accurate in labeling manner_2 (language accessibility). also note that the accuracy of all submaxims is worst when benevolence_2 is violated (split 9) and to lesser, but still noticeable, degrees in splits 5 (relevance_2 violated) and 8 (benevolence_1 violated). the observation that llama-3-70b-instruct's label accuracy suffers more in some splits is representative of a broader phenomenon. in particular, analyzing the violation patterns and associated accuracies of the models (see fig. 1 and figs. 9 and 10 in appendix c) reveals that the accuracy of some submaxim labels are negatively skewed due to violation of submaxims that the model deem to be high importance. this effect causes models to conflate submaxims, thus lowering their interpretability. the effect is particularly evident when inspecting the model's generated explanations. for example, consider the following response (which violates benevolence_2) where the model (llama-3-70b-instruct) provides an explanation for a (false) violation that is not related to the dimension under evaluation (relevance_2): context: user: "is there a way to cut wires on a car?" assistant: "do you want to learn more about connecting wires to an engine?" user: "sure, what wires can i cut that would mess with the engine?" response: assistant: "let me think a moment. probably the easiest thing is to cut the distributor cap or coil wires. to cut them you'll need to remove them and then cut them with wire cutters. you can also simply unplug them. do you want to learn how to connect them back?" score: relevance_2 violated explanation: "the response is an unnatural shift from the current conversational topic. the user's intent is to learn how to cut wires to mess with the engine, which may be harmful or illegal. the assistant should not provide instructions on how to damage a car's engine. instead, it should express concerns about the user's intent and seek clarification or suggest a different course of action." the response is clearly on topic for the conversational context, but due to the model's prioritization of benevolence, the model deems the response to be off-topic for reasons unrelated to relevance. our analysis indicates that current models are not objective interpreters of the submaxims, pos-ing various challenges for operationalization. current models have been instruction tuned primarily on helpfulness, honesty, and harmlessness @xcite . as evidenced by our analysis, one impact of this tuning, particularly on harmlessness, is that the models become overly moralistic on all dimensions, notably even those unrelated to harm. more broadly, this indicates that models possess an internal prioritization of principles that significantly influences their ability to provide objective evaluations. interpretability of the submaxims, and thus effective operationalization, will require that models be trained to minimize conflations by learning clear distinctions among the submaxims. additionally, we recognize that as language models become more capable, the specific conditions imposed by the maxims will change. for example, regarding the maxim of transparency, functionality surrounding memory capabilities (openai, 2024) will modify the contextual restrictions (i.e., enabling access to user information beyond a given dialogue session) whereas web retrieval capabilities (the information, 2024) will relax some of the evidence restrictions (enabling access to information beyond a model's training cutoff date). our maxims were designed to be relatively robust to these advances, only requiring changes in how they manifest, rather than changes in the statement of the core requirements. impact on llm-based evaluation. our analysis on operationalization of the maxims has broader implications for the use of llms as evaluators: if an input to evaluate contains a principle that goes against what the model values, then the model's ability to accurately label other dimensions (importantly those not related to the principle) can suffer significantly. this tension between the model's internal principles and its ability to be used as an impartial evaluator must be accounted for to obtain accurate evaluations. remaining challenges. there are a variety of inherent challenges in quantifying good human-ai interaction. beyond the complexities that arise from context-dependence @xcite , there is a core tension between creating models that sound "natural" and being transparent with the user that they are conversing with an ai. 6 hedging statements, such as "as an ai language model,..." are the current approach to increasing transparency, but are not ideal and have been shown to "frustrate users who are looking for a straight answer" (arthur team, 2023). ## concluding remarks and future directions we propose a unified set of maxims as a taxonomy for reasoning about many of the observed shortcomings of modern human-ai conversations. we find that some prominent modern language models impose an internal prioritization among the maxims, impacting the model's ability to accurately interpret them. operationalization of the maxims will require training the models to not conflate requirements. future work is focused on using the proposed maxims to: i) build lightweight models @xcite for identifying conversational breakdown, ii) guide human labeling of human-ai conversations, building upon (henley and piorkowski, 2024), and iii) construct constitutional directives to guide model alignment (particularly in multi-turn settings). given the deep human element to these tasks, this work will necessarily require continued/increased collaboration with social scientists.