Consistency Analysis of C hat GPT (2023)

Abstract: ChatGPT has gained a huge popularity since its introduction. Its positive aspects have been reported through many media platforms, and some analyses even showed that ChatGPT achieved a decent grade in professional exams, adding extra support to the claim that AI can now assist and even replace humans in industrial fields. Others, however, doubt its reliability and trustworthiness. This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour, focusing specifically on semantic consistency and the properties of negation, symmetric, and transitive consistency. Our findings suggest that while both models appear to show an enhanced language understanding and reasoning ability, they still frequently fall short of generating logically consistent predictions. We also ascertain via experiments that prompt designing, few-shot learning and employing larger large language models (LLMs) are unlikely to be the ultimate solution to resolve the inconsistency issue of LLMs.

## introduction ai systems can be more reliable and trustworthy, provided that they behave similarly to humans @xcite . in this regard, chatgpt, a llm that simulates human-like conversations @xcite , is gaining widespread popularity, reaching 100 million users only two months after its launch @xcite . in addition to many convenient functions that it provides, chatgpt has performed astoundingly well on various professional examination cases, including passing the united states medical licensing examination @xcite , achieving passing grades in four real exams at the university of minnesota law school @xcite , and providing decent answers to operation management exam questions, which is a core mba course @xcite . these surprising results make people be-lieve that llms can assist humans even in professional areas and greatly influence diverse academic and industrial fields. others, however, question chatgpt's reliability, pointing out its overconfidence in generating factually incorrect information @xcite , the inability to comprehend the complexity of human language @xcite , and imperfect mathematical abilities @xcite . even though these mistakes may appear insignificant in normal daily tasks, they can provoke crucial concerns in conservative and risk-sensitive domains, such as law, medicine, and finance. a correct behaviour is a crucial aspect in deciding models' trustworthiness by improving the certification foot_0 process @xcite . in this regard, we mainly investigate the trustworthiness of chatgpt in terms of logically consistent behaviour. by using the becel dataset @xcite , which is designed to ascertain whether language models satisfy various types of consistency, we analyse chatgpt's ability to generate logically consistent predictions based on the four properties below: ## related works the consistency of language models has been an important topic in natural language processing (nlp) but conducted under various definitions. the idea of semantic consistency is the most widely used concept in consistency analysis, meaning that a model should make consistent decisions in semantically equivalent contexts @xcite . semantic consistency is an indispensable property that should be satisfied in every textual data and nlp task. @xcite observed that plms are likely to generate different masked language modelling predictions when an object in queries is replaced with its plural form. @xcite , on the other hand, found that plms generate different masked language modelling predictions when given paraphrased queries. @xcite proposed an enhanced framework that facilitates the assessment of semantic consistency of natural language generation (nlg) outputs by introducing agreement functions. another line of work employed the idea by introducing a consistency regularisation term for training, which penalises the violation of semantic consistency, to train more robust nlp models @xcite @xcite . symmetric consistency is a consistency type based on symmetric inference, defined as f (x, y) = f (y, x). this implies that a model should be input-order invariant for tasks where the symmetric property holds. regarding the natural language inference (nli) task, @xcite believed that symmetric consistency applies to data points with "not entailment", i.e., "contradiction" and "neutral", as a label. they showed that many deep-learning-based nli models change their predictions when the premise and hypothesis are switched. on the other hand, @xcite only considered "contradiction" labels for their analysis and ascertained that nli models based on bert @xcite are likely to violate symmetric consistency. @xcite performed a symmetric consistency analysis on nli and semantic textual similarity (sts) tasks in a more conservative manner, arguing that a model should generate not only the same predictions but also the same confidence scores if it is truly input-order invariant. they also observed that plm-based models violated symmetric consistency and introduced a consistency regularisation term to compensate for the issue. the fundamental idea lying in negation consistency is the logical negation property (p is true ⇔ ¬p is false @xcite ). intuitively, the main idea is that a model's prediction should differ for text inputs delivering the opposite meaning. several studies investigated the negation consistency of bert and found that the model often generates the same outputs when asked negated and non-negated masked queries, e.g., "birds can lay [mask]" and "birds cannot lay [mask]" (kassner and schütze, 2020; ettinger, 2020). hossain et al. (2020) created negated versions of nli datasets and also observed the violation of negation consistency, suggesting that plms lack the understanding of negation expressions. to alleviate the issue, several works adopted data augmentation to train a model with abundant data containing negation expressions @xcite . @xcite introduced the meaning-matching task to enhance plms' textual understanding ability and observed performance improvements. transitive consistency is a consistency type that can measure the deductive reasoning ability. it is derived from transitive inference, represented as @xmath0, y, and z @xcite . in the nli task, @xcite employed the concept to generate four transitive inference rules. for three sentences p , h, and z, the rules are defined as: ## experimental results we now present our experimental results on the consistency performance of llms. the primary analysis target is the comparison between chatgpt's performance with eleuther ai's prompt design and those of fine-tuned plms. the becel dataset performances of two plms (electra-large @xcite and t5 @xcite ) are taken from @xcite . the performance of the prompt design devised by @xcite , gpt-4, and few-shot learning (2-shots in our experiments) will be mainly discussed in section 5. ## discussion can prompt design be a solution? prompts are input text consisting of a task demonstration and, for a few-shot task, some examples @xcite . prompt design has been shown to be an effective method of regulating the behaviour of gpt-3 @xcite . hence, one might argue that searching for an optimal prompt for each task can improve consistency. however, our experimental results provide sceptical evidence for this claim. when comparing the performance of chatgpt using eleuther ai's prompts and those designed by @xcite , there was no statistically significant difference in performance across all consistency types and downstream tasks at a confidence level of 0.05. in the case of gpt-4, similar results were observed, where a statistically significant difference at a confidence level of 0.05 was found only in symmetric consistency in the rte task. we believe that a primary cause contributing to this phenomenon is an inherent characteristic of machine learning: inductive reasoning. the underlying idea behind prompt design is that prompts created by experimenters are not optimal, because language models might have acquired target information from completely different contexts @xcite . that is, it can resolve the matter of inconsistency if and only if numerous consistency properties are reflected in chatgpt's inductive bias, which is technically not feasible. moreover, consistency improvements with prompt design can be considered another violation of semantic consistency, because the prompts will deliver identical semantic meaning, i.e., task description. can few-shot learning be a solution? it is widely known that providing few-shot examples generally leads to higher accuracy compared to the zero-shot setting. to ascertain whether this principle extends to consistency, we conducted additional few-shot experiments on chatgpt by providing two-shot examples and using eleuther ai's prompt. however, the findings suggest that providing few-shot examples is not beneficial to improving consistency. when compared with the result of chatgpt employing the same prompt design but under a zero-shot setting, we observe statistically significant increases in inconsistencies across the majority of test cases. the most substantial rise in inconsistency values is evident in the context of symmetric consistency and the wic task. we even identified several instances where the model altered its decision when the order of two-shot examples was switched. these experimental outcomes indicate that employing a few-shot approach does not represent a definitive solution for enhancing the consistency of llms. can increasing data and model size be a solution? another possible way to improve consistency is training a larger llms with more abundant training data, as larger models generally outperform smaller ones in numerous nlps downstream tasks. however, our experimental results reveal the limitations of this approach. through a comprehensive comparison of chatgpt and gpt-4 performance employing identical prompt designs, it was ascertained that gpt-4 does not necessarily exhibit superior performance in comparison to chatgpt from a consistency perspective. while gpt-4 did demonstrate enhanced consistency in several test scenarios, including symmetric and transitive consistency in the wic task across both prompt designs, symmetric consistency in the rte task, and negation consistency in the snli task using prompts devised by @xcite , no discernible improvements were observed in the remaining test cases, which constitutes a portion of 77% of among the total test scenarios. in addition, gpt-4 also exhibited a high level of self-contradiction, just like chatgpt. moreover, increasing the size of the data and model is a technically unsustainable strategy. first, the data collection procedure requires tremendous effort and is challenging to cover all possible variations, especially for consistency types that demand abundant linguistic resources, such as semantic consistency. second, even if we successfully expand the data, it is doubtful whether we can afford to update an llm with each new dataset iteration. considering the ever-changing information, the data expansion and update of an llm should be performed continuously, as neglecting to do so may raise the concern of outdated information @xcite . however, training an llm entails tremendous financial and environmental costs @xcite . for instance, training a bert-base model without hyperparameter tuning requires a co2 emission of 650kg, which is comparable to flying from new york to san francisco for one passenger @xcite . a simple expectation of co2 emission for re-training chatgpt and gpt-4 would amount to 1,033t and 1,0240t, respectively, 3 while a human is responsible for 5t co2 emission per year. the continuous emission of such a substantial volume of greenhouse gases would have a detrimental impact on the environment of modern society facing the global climate crisis. 3 chatgpt and gpt-4 are approximately 1,590 and 16,000 times larger than bert-base. ## summary and outlook the advent of chatgpt is accelerating the developments in the nlp field driven by llms. its outstanding performance captured considerable attention, resulting in many articles, posts, and analyses highlighting chatgpt's positive aspects across numerous media. there are others, however, who question its reliability based on the model's faulty behaviours. to this end, this study aims to examine the trustworthiness of chatgpt in terms of the language model's consistency. in this paper, we have investigated the consistency behaviour of chatgpt across four consistency types and downstream tasks. our experimental results demonstrate that chatgpt achieves a certain level of enhanced language understanding ability, especially in negation expressions and antonyms. it also exhibits improved deductive reasoning ability with lower transitive inconsistencies compared to the earlier version of plms. however, while chatgpt exhibits enhanced negation and transitive consistency, this does not mean that the model is perfectly consistent, i.e., it still makes mistakes that violate the logical properties, and the frequency of such occurrences is non-negligible. also, contrary to the widespread belief regarding the outstanding performance of chatgpt, its performance across various consistency types falls short of expectations. it frequently changes its decision when an input text is replaced with a paraphrased sentence, even though it is generated from chatgpt itself. moreover, in input-order invariant tasks, chatgpt is prone to make a different decision when the order of the input sentences is switched. these issues are also observed in gpt-4 or with the use of different prompt designs and few-shot learning, indicating that these approaches are unlikely to be a fundamental remedy. given how simple and natural the symmetric and semantic consistencies are in human reasoning, violating these consistencies can be a huge blow to llms' trustworthiness. these fallacious behaviours are especially lethal to domains operating conservatively and at high risk. although llms are a revolutionary technique that brought an unprecedented era to nlp, such issues should be resolved before these models are applied in real applications, particularly considering the huge economic and environmental costs consumed for developing llms.