Is C hat GPT a General-Purpose Natural Language Processing Task Solver? (2023)

Abstract: Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot—i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.

## introduction large language models (llms) have been shown to be able to solve a variety of natural language processing (nlp) tasks zero shot-i.e., without relying on any training data for a given downstream task-by conditioning the model on appropriate prompts @xcite . the ability to perform new tasks based on instructions can be seen as an important step towards artificial general intelligence @xcite . despite achieving reasonable performance in some cases, current llms are still prone to various mistakes in zero-shot learning. in addition, the format of the prompt can have a substantial impact-for example, simply adding "let's think step by step" @xcite has been shown to significantly improve the performance of in-structgpt @xcite on reasoning tasks. these limitations illustrate that current llms are not truly general-purpose language systems. recently, the chatgpt llm released by ope-nai has attracted a great deal of attention from the nlp community. chatgpt was created by training a gpt-3.5 series model through reinforcement learning from human feedback (rlhf) @xcite (similarly to instructgpt). rlhf mainly includes three steps: training a language model with supervised learning, collecting comparison data based on human preferences and training a reward model, and optimizing the language model against the reward model using reinforcement learning @xcite . through rlhf training, chatgpt has been observed to have impressive capabilities in various aspects, including generating high-quality responses to human input, rejecting inappropriate questions, and self-correcting previous errors based on subsequent conversations @xcite . while chatgpt shows strong dialogic capabilities, it still remains unclear to the nlp community whether chatgpt attains better zero-shot generalization compared with existing llms. to fill in this research gap, we systematically study the zeroshot learning capability of chatgpt by evaluating it on a large collection of nlp datasets covering 7 representative task categories, including reasoning foot_0 , natural language inference, question answering (reading comprehension), dialogue, summarization, named entity recognition, and sentiment for each reasoning dataset, the better result between zero-shot and zero-shot chain-of-thought is shown. measures of samsum, conll03, and the rest are rouge-1/2/l average, f1, accuracy, respectively. to answer these questions, we empirically compare the performance of chatgpt (gpt-3.5-turbo) and the previous gpt-3.5 model (text-davinci-003). in addition, we report zero-shot, fine-tuned, or few-shot fine-tuned results from recent work such as flan @xcite , t0 @xcite , and palm @xcite . ## related work this work mainly explores the zero-shot learning capability of chatgpt on a diverse collection of datasets including reasoning and classic nlp tasks. in light of this, we review three lines of research that form the basis of this work: large language models, zero-shot learning, and chain-of-thought prompting for reasoning. ## methodology as mentioned in section 1, we mainly compare the zero-shot learning performance of chatgpt (gpt-3.5-turbo) and gpt-3.5 (text-davinci-003) on different tasks. given a task instruction p and a test problem x that are concatenated as the input, the model f is expected to generate a target text @xmath0, x) to address the test problem. the instructions and input formats of different tasks are shown in figure 2 and 3. for example, when the model performs sentiment analysis tasks, the task instruction p is "for each snippet of text, label the sentiment of the text as positive or negative. the answer should be exact 'positive' or 'negative'.". after reading the instruction p and the input x "it 's a stunning lyrical work of considerable force and truth.", the model is expected to generate the output y "positive". different from this single-stage prompting method, we use the same two-stage prompting as @xcite for zero-shot-cot. in the first stage, we adopt "let's think step by step." as the instruction p 1 to induce the model to generate the rationale r. in the second stage, we use the selfgenerated rationale r along with the original input x and the instruction p 1 as the new input to guide the model to generate the final answer. a new instruction p 2 , e.g., "therefore, among a through e, the answer is", serves as the trigger sentence for extracting the answer. all task instructions are taken from or inspired by brown et al. ( 2020) ## experiments in this section, we first describe the tasks and datasets, and then present the experimental results. ## conclusion we have empirically studied the zero-shot learning capabilities of chatgpt on a large, diverse collection of datasets covering representative task categories. extensive experimental results and analysis model zero-shot fine-tuned chatgpt gpt-3.5 flan t5-11b all 93.7 88.8 94.6 97.5 pos 90.8 88.1 --neg 96.7 89.5 --