PUB : A Pragmatics Understanding Benchmark for Assessing LLM s’ Pragmatics Capabilities (2024)

Abstract: LLMs have demonstrated remarkable capability for understanding semantics, but their understanding of pragmatics is not well studied. To this end, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely; Implicature, Presupposition, Reference, and Deixis. We curate high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA). PUB includes a total of 28k data points, 6.1k are newly annotated. We evaluate nine models varying in the number of parameters and type of training. Our study reveals several key observations about the pragmatic capabilities of LLMs: 1. chat-fine-tuning strongly benefits smaller models, 2. large base models are competitive with their chat-fine-tuned counterparts, 3. there is a huge variance in performance across different pragmatics phenomena, and 4. a noticeable performance gap between human capabilities and model capabilities. We hope that PUB will enable comprehensive evaluation of LLM’s pragmatic reasoning capabilities.

## introduction pragmatics, within linguistics, examines how context shapes language understanding in communication @xcite . it centers on real-life language use, considering context, speaker intentions, presuppositions, and implied meanings to derive interpretations beyond literal words. human's proficiency in pragmatics stems from their inherent cognitive skills and social awareness. our minds adeptly process not only spoken words but also context and implied messages. in natural language processing (nlp), large language models (llms) @xcite @xcite ., * first author † equal contribution in coding and experiments 2023 ) have emerged as a transformative force in recent years. llms have shown remarkable abilities on many downstream tasks like natural language understanding @xcite ), text generation @xcite , code synthesis @xcite , question answering @xcite @xcite and reasoning @xcite @xcite , etc. while semantics involves the study of words and their meanings in a language, pragmatics extends this inquiry by considering word's meanings within the context in which they are used. given llms' increased interaction with humans via practical, real-world applications like chatbots, search engines, and web browsers, the following research question arises: do llms understand pragmatics in conversations? most benchmarks until now deal only with abilities like problem-solving @xcite or semantic understanding @xcite where llms have started approaching human performance. however, due to lack of reliable benchmarks, it is still unclear whether an llm understands pragmatics or not. to facilitate this research, we propose a pragmatic understanding benchmark (pub) 1 over four major pragmatic phenomena: 1. implicature: understanding what is implied in a statement even though it is not literally expressed. 2. presupposition: an implicit assumption that is taken for granted before the use of a statement. 3. deixis: a phenomenon in which certain phrases within a sentence rely on contextual cues, such as the speaker, the listener, or the surrounding context, to convey their meaning effectively. 4. reference: how language points to things, people, place, time, etc. in accordance with the content and structure outlined in the handbook of pragmatics @xcite . pub includes 22,000 examples, leveraging existing data, and introduces three new datasets with 6,100 newly annotated examples. human evaluation of a subset of these datasets is conducted to assess performance against established llms. the benchmark comprises fourteen tasks that evaluate pragmatics as an mcqa task since mcqa evaluation is more closely related to question-answering abilities in conversations @xcite . we carefully curate the existing datasets to balance them and formulate prompts for these tasks, which are more natural and better suited to evaluate llms. following @xcite , (robinson and wingate, 2023)), we evaluate the pragmatic abilities of llms using multiple choice prompting (mcp) and cloze prompting (cp). to validate the model's confidence in its choices we also calculate the proportion of plurality agreement (ppa) on 3 tasks similar to @xcite . our contributions are: (1) a comprehensive and 1 the benchmark is available at huggingface.co/datasets/cfilt/pub unified dataset for 14 distinct tasks in pragmatics (figure : 3), containing 28k data points; to the best of our knowledge this is the first datasetlinguistically motivated and well-grounded-to test pragmatic capabilities of llms. (2) a systematic evaluation of 6 variations of llama-2, t5, flan-t5, and gpt-3.5, on the fourteen mentioned tasks. (3) a study of human performance on a sample of the dataset to highlight the performance gap between llms and humans. (4) insight emerging from (3) to uncover strengths and weaknesses of llms vis-a-vis humans. these contribution points-we hope-will assist researchers in improving the interactive abilities of llms. ## related work pragmatics is very crucial in the domain of linguistics, where it plays a critical role in understanding meaning @xcite . in linguistic terms, pragmatics deals with the study of context-dependent aspects of meaning that are systematically abstracted away from, in the construction of content or logical form @xcite . some of the basic subfields of pragmatics include implicature, presupposition, speech acts, reference, deixis, definiteness, and indefiniteness. over the years, many researchers have devoted their research to studying such pragmatic phenomena for machine learning. to study implicatures, @xcite employ indirect answers in polar questions, @xcite utilize hierarchical grammar models for understanding implicature and deictic reference in simple conversations, @xcite employ natural language inference (nli) to grasp scalar implicatures, @xcite leverage implicature rules for optimizing sentiment detection, and @xcite develop a sentence-level corpus with implicature ratings. whereas for presupposition, @xcite use search engine queries that may contain questionable assumptions that are closely related to presupposition. @xcite also reveals that transformer models exploit specific structural and lexical cues as opposed to performing some kind of pragmatic reasoning. @xcite introduced a computational approach to detecting adverbial presupposition triggers, emphasizing the importance of pragmatics in language understanding. @xcite focused on predicting scalar inferences using linguistic signals, while @xcite explored neural sentence encoders for predicting scalar inferences. @xcite investigated presupposition verification in question-answering, and @xcite studied the impact of pre-finetuning bert models on nli involving presuppositions. these studies collectively highlight the advancements in computational pragmatics. a recent comparison of pragmatic understanding between humans and models, conducted by @xcite , shows that language models struggle to comprehend humor, irony, and conversational maxims @xcite . in the most recent work, @xcite have studied implicature recovery in polar questions and answers. these approaches have offered only a restricted understanding of the shortcomings exhibited by these models by either evaluating only a single phenomenon or using a smaller number of samples to make it quantifiable. other existing works @xcite @xcite do not comprehensively cover all important domains of pragmatics to evaluate llms. to the best of our knowledge, we are the first ones to combine major aspects of pragmatics to create a quantifiable benchmark. ## datasets and tasks we describe the datasets used and curated for creating pub in section 3.1. various tasks for evaluation of llms is introduced in section 3.2. ## evaluation strategy we have selected two evaluation methods namely length normalized cloze prompting @xcite and multiple choice prompting (mcp) @xcite . since mcp is also dependent on the multiple choice symbol binding ability of llms, we have computed the proportion of plurality agreement (ppa) @xcite to ensure the model's consistency across possible orders of answer options. the mod-els under investigation include flan-t5-xxl (chung et al., 2022); llama-2 : 7b, 7b-chat, 13b, 13b-chat, 70b, 70b-chat @xcite ; t5 @xcite and gpt-3.5 (brown et al., 2020) ## results and analysis the results of our experiments are presented in figures 4, 5. only the maximum across all evaluation strategies is reported in these figures. detailed results are given in appendix e. the results for ppa are presented in figure 2 . based on these results, the following section aims to answer the questions regarding the pragmatic capabilities of llms. ## conclusion in this study, we introduce the pragmatic understanding benchmark (pub) designed to assess pragmatic comprehension in llms. we offer a detailed analysis, providing insights into various aspects of pragmatic understanding within llms. our observations reveal that pragmatic understanding in llms can be enhanced through instructiontuning of these models. interestingly, even without specific fine-tuning, language models at scale exhibit equivalent performance. notably, smaller models, particularly the instruction-tuned variants, outperform their base counterparts, but this advantage diminishes as models scale up, with base and instruction-tuned models showing comparable performance. despite advancements, llms are yet to attain human-level performance, especially in tasks requiring a deep understanding of language context. the observed variability in model performance across different tasks within the same dataset highlights the complexity of achieving human-like pragmatic understanding in llms. the pub benchmark thus provides a clear indication of where llms currently stand and the strides still needed to reach human parity in language understanding. we hope that this benchmark will aid researchers in improving llms' conversational abilities with humans.