I mplica TR : A Granular Dataset for Natural Language Inference and Pragmatic Reasoning in T urkish (2024)

Abstract: We introduce ImplicaTR, a linguistically informed diagnostic dataset designed to evaluate semantic and pragmatic reasoning capabilities of Natural Language Inference (NLI) models in Turkish. Existing Turkish NLI datasets treat NLI as determining whether a sentence pair represents \textit{entailment} , \textit{contradiction} , or a \textit{neutral} relation. Such datasets do not distinguish between \textit{semantic entailment} and \textit{pragmatic implicature} , which linguists have long recognized as separate inferences types. ImplicaTR addresses this by testing NLI models’ ability to differentiate between \textit{entailment} and \textit{implicature} , thus assessing their pragmatic reasoning skills. The dataset consists of 19,350 semi-automatically generated sentence pairs covering \textit{implicature, entailment, contradiction,} and \textit{neutral} relations. We evaluated various models (BERT, Gemma, Llama-2, and Mistral) on ImplicaTR and found out that these models can reach up to 98% accuracy on semantic and pragmatic reasoning. We also fine tuned various models on subsets of ImplicaTR to test the abilities of NLI models to generalize across unseen implicature contexts. Our results indicate that model performance is highly dependent on the diversity of linguistic expressions within each subset, highlighting a weakness in the abstract generalization capabilities of large language models regarding pragmatic reasoning. We share all the code, models, and the dataset.

## introduction natural language inference (nli) tasks are generally designed as three-way classification problems between sentence pairs @xcite . given a sentence pair consisting of a premise (p) and a hypothesis (h), the task is to classify the relation between p and h as one of entailment, contradiction, or neutral. some of the most commonly used nli datasets such as snli @xcite and mnli @xcite contain three way annotations of sentence pairs and recently @xcite translated both datasets into turkish to create the combined nli-tr dataset. although these nli datasets have been useful in testing the sentential understanding and reasoning capabilities of language models, they fall short of detecting the precise nature of reasoning, i.e. semantic vs. pragmatic, due to the coarseness of their labeling schemas. in particular, these datasets conflate various implicational relations such as entailment, implicature, and presupposition under the same label i.e. entailment. however, linguists have long observed that entailments differ from implicatures and presuppositions specifically in terms of what kind of reasoning mechanisms underlie such implicational relations @xcite @xcite @xcite . a key distinction between entailments and implicatures is that of reasoning over what is said and what is not said. entailment relations are inferences based on what is said and they arise as a consequence of the meanings of expressions in a sentence and the general laws of logic. the defining characteristic of an entailment relation between a premise (p) and a hyopthesis (h) is truth. p entails h if and only iff whenever p is true h must be true as well. the p-h pair in (1) illustrates entailment. this is a logical corollary of the subsetsuperset relation between fluffy cats and cats. implicatures on the other hand are inferences based on what is not said and they follow from general cooperativeness principles of conversation @xcite . in (2), the relation between p and h is implicature but not entailment. unlike entailments, implicatures are not logical consequences of their premises. instead, they arise through pragmatic reasoning. implicatures can be distinguished from ordinary entailments by means of various tests such as cancellation, suspension, and reinforcement. for example, implicatures can be cancelled without leading to a contradiction but entailments cannot as illustrated in (3) and ( 4 ). to test the pragmatic reasoning capabilities of language models in turkish, we introduce implicatr, the first fine-grained turkish nli dataset consisting of premise-hypothesis pairs containing entailment, implicature, contradiction, and neutral labels. we test various types of large language models (llms) using implicatr and observe that llms are capable of carrying out both semantic and pragmatic reasoning with success rates of up to 98% accuracy. despite their high levels of success, our ablation studies reveal that llms do not form a high level abstraction for pragmatic reasoning as they cannot generalize across various types of implicature contexts. recent nli research started to pay attention to more granular inference types that can help evaluate the precise reasoning capabilities of language models by distinguishing inference types such as implicature, entailment, presupposition. implicature (george and mamidi, 2020) and big-bench (srivastava and others, 2022) datasets were created for particulatized implicatures. similarly, grice @xcite offers conversational reasoning and implicature data in the form of open dialogues devised by an automated grammar. the imppressive dataset @xcite ) consists of semi-automatically generated scalar implicatures and presuppositions as premise-hypothesis pairs, where authors show that models can do pragmatic reasoning for some types of scales in their dataset. this brief review of the literature reveals that the nli literature needs more work in the areas of pragmatic reasoning and we aim to help fill this gap by investigating implicatures, which present a distinct line of work for the nli research with its more granular comprehension of the pragmatic inferences. in addition, nli research in turkish has a limited scope, totally lacking an investigation into implicatures to the best of our knowledge. with its rich morphology and agglutinative nature especially reflected on the verbs, turkish presents a peculiar case for probing into how implicatures are handled by nli models. way nli classes (entailment, contradiction, neutral). the dataset comprises five different linguistic categories (quantifiers, adjectives, verbs, modals, and numerals) with varying number of scalar pairs for each category. ## experiment 1 experiment 1 aims to explore whether llms exhibit pragmatic reasoning, specifically in scalar implicature resolution. we fine-tuned a series of models on implicatr and observed that language models can successfully identify implicatures. ## experiment 2 upon observing that llms are capable of learning pragmatic inferences in the form of scalar implicatures, we conducted a second experiment where we perform an ablation study to test the generalization abilities of llms with respect to pragmatic reasoning in a supervised fashion. this experiment consists of two phases. in the first phase, we train five models by eliminating one of the linguistic categories entirely from training split in each model training and then test the model on the eliminated linguistic category. the goal is to test whether llms can create a sufficiently abstract generalization of scalar implicatures that can be used independent of the linguistic structures. in the second phase, we develop a sixth model by eliminating some scalar pairs from each linguistic category and test the model on the eliminated pairs. the goal in this second phase is to test the generalization abilities of llms within each category. the ablation study is followed by a feature analysis to inspect which linguistic features are influential in llm performance in textual entailment and implicature reasoning. ## conclusion we presented implicatr, a diagnostic dataset to test the pragmatic reasoning abilities of language models. implicatr contains nli-style sentence pairs with four distinct inference types, entailment, contradiction, neutral and implicature. we evaluated various llms and showed that they are capable of doing pragmatic reasoning and distinguishing entailments from implicatures with a high degree of accuracy. our results also indicated that the models we tested cannot make sufficiently abstract generalizations across various linguistic structures for pragmatic reasoning and the type frequency of the scalar items is inversely correlated with the model success.