Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning (2024)

Abstract: This study addresses the challenges of assessing and enhancing social-pragmatic inference in large language models (LLMs). We first highlight the inadequacy of current accuracy-based multiple choice question answering (MCQA) formats in assessing social-pragmatic reasoning, and propose the direct evaluation of models’ free-form responses as measure, which correlates better with human judgment. Furthermore, we explore methods to improve pragmatic abilities in LLMs, advocating for preference optimization (PO) over supervised finetuning (SFT), given the absence of a definitive “gold” answer in social contexts. Our results show that preferential tuning consistently outperforms SFT across pragmatic phenomena and offers a near-free launch in pragmatic abilities without compromising general capabilities. Lastly, we examine the internal structure of LLMs, revealing that the significant boost in pragmatic reasoning is tied to deeper layer representations, analogous to human high-level thinking. Our experiments span a variety of pragmatic and social reasoning datasets, as well as an image referential game requiring a multimodal theory of mind (ToM). With our refined paradigms for evaluating and enhancing pragmatic inference, this paper offers key insights into building more socially aware language models.

## introduction social-pragmatic inference is a key aspect of human communication, involving the ability to understand and respond to implied meanings, intentions, and emotional states behind literal utterances @xcite @xcite , as well as shared social conventions @xcite . pragmatics spans a broad range of phenomena, including implicatures, irony, humor, * work done during internship at peking university. and metaphor, along with higher-level cognitive skills like theory of mind (tom) @xcite , which are essential for interpreting non-literal language and context-sensitive messages. for example, a friend's statement, it's chilly in here, might be a polite request to close a window, rather than a simple observation about the temperature. the importance of social-pragmatic intelligence in human communication underscores the need for large language models (llms) to develop similar capabilities in order to interact more naturally with users. however, current approaches to enhancing pragmatic abilities in llms face two lines of limitations: 1) on the evaluation front, typical methods rely on measuring classification accuracy on benchmarks formatted as multiple (if not binary) choice question answering (mcqa) @xcite @xcite @xcite . while a model might correctly select the option label, it may still fail to respond pragmatically by itself. for example (see fig. 1 ), a model might pick the right answer in an mcqa task without fully understanding the social complexity of changing the subject. moreover, real-world social interactions seldom have a single "gold" answer. relying on accuracy in selecting predefined responses undermines the assessment of a model's true pragmatic capability in flexible generations. 2) on the side of improving pragmatic abilities, while inference-time methods such as few-shot prompt engineering @xcite and external graph modules @xcite have been proposed to improve llm performance in pragmatic tasks, little effort has been made to directly invoke the model's internal social-pragmatic intelligence, enabling it to autonomously generate pragmatically appropriate responses. in this paper, we propose paradigm shifts on both fronts: 1) for evaluation, we argue for an open-figure 1 : an example of llm outputs when queried about a social-pragmatic scenario, taken from @xcite . on the right, a llama2-13b-chat @xcite model correctly identifies the gold response id in the mcqa format but fails to fully grasp the underlying pragmatic meaning when generating its own response. on the left, a smaller llama2-7b-chat model preference-tuned to contrast the gold answer with less pragmatic alternatives, produces an open-ended response that is equally good and as pragmatically sound as the provided "gold" answer. to better understand how the internal components of a transformer-based llm @xcite are responsible for invoking social-pragmatic abilities, we explored finetuning specific transformer layers. our results indicate that pragmatic understanding is closely tied to deeper-down layers in the model, which hints at a potential parallel with how human pragmatic inference relies on higherlevel cognitive processes. overall, the main contributions of this paper are: ## improving pragmatic abilities on top of the open-ended evaluation paradigm that more closely reflects real-world scenarios, we also aim to explore how to intrinsically enhance the social-pragmatic capabilities of llms. different from previous works ( §5) that primarily focus on adding external modules for better cognitive abilities @xcite or rely on few-shot prompt engineering @xcite , our approach is centered on aligning the model's intrinsic representation toward a more socially and pragmatically grounded distribution. let p θ represent an llm parameterized by θ. in our context, p θ takes a question q as input, which describes a pragmatics-involved social scenario, and a gold is the annotated correct response. ## conclusion this paper addresses two lines of challenges related to social-pragmatic abilities in llms. first, we argue for a shift from the traditional mcqa format to open-ended evaluation that directly measures the soundness of the model's generated responses in social scenarios. second, we propose to enhance llms' intrinsic pragmatic abilities via preference optimization (po) over supervised finetuning (sft). through po, models learn to capture the subtle nuances between preferred and dispreferred social interactions. our experiments across multiple pragmatic datasets, coupled with human evaluation, and further examined within a multimodal theory of mind setting through the image referential game, all effectively demonstrate both the advantages of our free-form evaluation protocol and the superiority of po over sft in pragmatic scenarios. additionally, we also reveal the impact of trainable layer depth on the model's pragmatic performance gains, suggesting a potential mirroring with the higher-level cognitive processes involved in human social reasoning.