CHA mbi: A New Benchmark on C hinese Ambiguity Challenges for Large Language Models (2024)

Abstract: Ambiguity is an inherent feature of language, whose management is crucial for effective communication and collaboration. This is particularly true for Chinese, a language with extensive lexical-morphemic ambiguity. Despite the wide use of large language models (LLMs) in numerous domains and their growing proficiency in Chinese, there is a notable lack of datasets to thoroughly evaluate LLMs’ ability to handle ambiguity in Chinese. To bridge this gap, we introduce the CHAmbi dataset, a specialized Chinese multi-label disambiguation dataset formatted in Natural Language Inference. It comprises 4,991 pairs of premises and hypotheses, including 824 examples featuring a wide range of ambiguities. In addition to the dataset, we develop a series of tests and conduct an extensive evaluation of pre-trained LLMs’ proficiency in identifying and resolving ambiguity in the Chinese language. Our findings reveal that GPT-4 consistently delivers commendable performance across various evaluative measures, albeit with limitations in robustness. The performances of other LLMs, however, demonstrate variability in handling ambiguity-related tasks, underscoring the complexity of such tasks in the context of Chinese. The overall results highlight the challenge of ambiguity handling for current LLMs and underscore the imperative need for further enhancement in LLM capabilities for effective ambiguity resolution in the Chinese language.

## introduction ambiguity is a prevalent and noteworthy linguistic phenomenon @xcite . differences in context or individual interpretations will lead to different understandings of ambiguous sentences, complicating the comprehension and communication of information @xcite . large language models (llms) * corresponding author. there are a few works pay attention to ambiguity problem @xcite @xcite @xcite . @xcite ambignq, offering multiple potential answers for ambiguous open-domain questions and supplying disambiguation questions corresponding to each answer. they also built a baseline model for generating multiple answers to open-domain questions @xcite . pavlick and kwiatkowski conduct a thorough examination of disagreement in human judgments on the natural language inference (nli) task. their findings reveal that numerous disagreements remain after augmenting the number of annotators and the contextual information provided @xcite . all these efforts contribute to a deeper understanding and resolution of language ambiguity. however, they mainly focus on the ambiguity problem in english. chinese, as a language significantly distinct from english @xcite , presents greater intricacy at the lexical and morphemic levels, involving a wider range of ambiguity types. for example, if a, b and ab are all possible words, then ab exhibits combinatorial ambiguity, such as "才"(just), "能"(be able to) and "才能"(talent). in the chinese context, there is a lack of annotated datasets to comprehensively evaluate the ability of llms to identify and resolve ambiguity. to address this gap, we introduce chambi 1 , a chi-nese multi-label disambiguation dataset for natural language inference. the dataset contains 4,991 examples, including 824 ambiguous instances, each annotated with two or more disambiguation labels, representing different interpretations for resolving the ambiguity. enumerating all possible interpretations of a ambiguous sentence is challenging. therefore, we define ambiguity using the format of premise-hypothesis pairs in nli dataset inspired by ambigqa @xcite . natural language inference is a task to identify whether a hypothesis is true, false, or uncertain given a premise @xcite . ambiguity in premise and (or) hypothesis may affect the determination of labels, leading to multi-label example. we only need to fully analyze the potential multiple relationships between the premise and the hypothesis, and list a different interpretation for each relationship. to standardize ambiguity identification, we define seven categories of ambiguity, including spoken ambiguity (accent ambiguity, pause ambiguity), written ambiguity (vocabulary, grammar, semantics, reference ambiguity) and incomplete ambiguity. through the chambi dataset, we design various tests to evaluate the ambiguity identification and disambiguation abilities of popular chinese-supporting generative models, including the gpt series model, baidu large model ernie-bot, the chinese large model atom 2 , and the chinese-english bilingual large model chatglm2 @xcite . our main contributions are: label variation on nli. nli is the task of determining the logical relationship between a given premise and a hypothesis, which can be divided into three types: entailment, contradiction and neutral. entailment means that the premise can imply the truth of the hypothesis. contradiction means the direct conflict between premise and hypothesis, including opposition and mutual exclusion. neutral means that there is no clear entailment or contradiction between premise and hypothesis, and they can exist independently @xcite . the research on label variation in nli has seen notable contributions. pavlick and kwiatkowski @xcite initiate groundbreaking work in this domain, subsequently, nie et al. @xcite expand this by collecting more annotations. additionally, efforts are made to model label variation @xcite ; zhang example rewrites i rewrites ii category p: 他们正在看电影。 p: 他们正在电影院里看电影。 p: 他们正在家里看电影。 incomplete (they are watching a film.) (they are watching a film in the cinema.) (they are watching a film at home.) h: 他们正在电影院里。 h: 他们正在电影院里。 h: 他们正在电影院里。 (they are in the cinema.) (they are in the cinema.) (they are in the cinema.) l: entailment,contradiction l: entailment l: contradiction p: 小明邀请了她，小暗也邀请了 她，她犹豫了一下，然后接受了 他的邀请。 p: 小明邀请了她，小暗也邀请 了她，她犹豫了一下，然后接 受了小明的邀请。 p: 小明邀请了她，小暗也邀请 了她，她犹豫了一下，然后接 受了小暗的邀请。 reference (xiaoming invited her, and xi-aoan also invited her. she hesitated, and then accepted his invitation.) (xiao ming invited her, and xiao an also invited her. she hesitated, and then accepted xiaoming's invitation.) (xiao ming invited her, and xiao an also invited her. she hesitated, and then accepted xi-aoan's invitation.) h: 她接受了小明的邀请。 h: 她接受了小明的邀请。 h: 她接受了小明的邀请。 (she accepted xiaoming's invitation.) (she accepted xiaoming's invitation.) (she accepted xiaoming's invitation.) l: entailment,contradiction l: entailment l: contradiction p: 她慢慢地放下了手中的书，转 身面对我。 p: 她慢慢地放下了手中的书， 同时转身面对我。 p: 她先慢慢地放下了手中的 书，然后转身面对我。 incomplete (she slowly put down the book in her hand and turned to face me.) (she slowly put down the book in her hand and turned to face me at the same time.) (she slowly put down the book in her hand and then turned to face me.) h: 她转过身来，慢慢地放下了 手中的书。 h: 她转过身来，同时慢慢地 放下了手中的书。 h: 她先转过身来，然后慢慢 地放下了手中的书。 (she turned around and slowly put down the book in her hand.) (she turned around and slowly put down the book in her hand at the same time.) (she turned around and then slowly put down the book in her hand.) l: entailment,contradiction l: entailment l: contradiction ## dataset construction here, data collection, annotation, and validation process are described. the final dataset contains 4991 nli examples, of which 824 are ambiguous. each example has a premise sentence and a hypothesis sentence. each ambiguous example is annotated with a set of labels, reflecting the ambiguity in the premise and (or) hypothesis. additionally, it includes ambiguity category and the disambiguation rewrites of the premise and (or) hypothesis for each label, as shown in we employ two methods to collect ambiguous nli examples: automatic generation and keyword crawling. automatic generation (section 3.1) takes advantage of text generation and pattern replication capabilities of llms to obtain potentially ambiguous examples. crawling and manual curation (section 3.2) involves acquiring ambiguous sentences from various sources as premises, followed by manual formulation of hypotheses. then we manually annotate and validate collected examples, and obtain high-quality examples and disambiguation rewrites of ambiguous examples. ## evaluation of llms with the proposed dataset chambi, we evaluate the capabilities of existing popular language models in two aspects: ambiguity identification and ambiguity resolution. two tests evaluate ambiguity identification capabilities, including evaluating whether llms can recognize ambiguities in premises or hypotheses (section 4.1), and whether they can recognize ambiguity and perform correct multi-label classification for examples (section 4.2). additionally, two tests evaluate disambiguation capabilities, including evaluating whether llms can directly generate disambiguation rewrites (section 4.3), and whether they can recognize the effectiveness of disambiguation rewrites (section 4.4). for these tests, we only focus on two-label and three-label examples from chambi. the llms we select for evaluation are gpt-3 (davinci), instructgpt (text-davinci-003), chat-gpt (gpt-3.5-turbo), gpt-4, baidu large model ernie-bot (ernie-bot-turbo-0704), the opensource large pre-trained model atom-7b based on llama2 for chinese, and the bilingual open-source conversational model chatglm2-6b. to better evaluate models' capabilities, we also compare human performance as a baseline. due to the high time and labor costs, for each test, we randomly select 50 examples and ensure that three participants participated. ## conclusion we present chambi dataset, the first chinese multi-label disambiguation dataset, serving as a benchmark for evaluating ambiguity identification and disambiguation capabilities of llms. leveraging this dataset, we conduct detailed evaluations of several state-of-the-art llms that support chinese, revealing the challenges they face in effectively handling ambiguity. additionally, chambi can serve as a resource for fine-tuning llm to create an ambiguity detector. in our experiment, the finetuned chatglm2, with its improved ambiguity identification capabilities, demostrates its potential for handling ambiguous language in real world applications. we hope this new benchmark will serve as a foundation for research in ambiguity identification and disambiguation in chinese, and foster the development of large language models in this area.