Prompting Implicit Discourse Relation Annotation (2024)

Abstract: Pre-trained large language models, such as ChatGPT, archive outstanding performance in various reasoning tasks without supervised training and were found to have outperformed crowdsourcing workers. Nonetheless, ChatGPT’s performance in the task of implicit discourse relation classification, prompted by a standard multiple-choice question, is still far from satisfactory and considerably inferior to state-of-the-art supervised approaches. This work investigates several proven prompting techniques to improve ChatGPT’s recognition of discourse relations. In particular, we experimented with breaking down the classification task that involves numerous abstract labels into smaller subtasks. Nonetheless, experiment results show that the inference accuracy hardly changes even with sophisticated prompt engineering, suggesting that implicit discourse relation classification is not yet resolvable under zero-shot or few-shot settings.

## introduction pre-trained language models have demonstrated superior performance in various nlp tasks for years, and recently prompt-tuning instead of fine-tuning has become the dominant framework to make efficient use of large language models (llms). llms such as chatgpt have demonstrated human-level performance in various reasoning tasks under zeroshot or few-shot settings using natural language prompts as inputs (see e.g., @xcite . this has led to a wave of research in prompt engineering to elicit the prediction potential of llms (such as @xcite . in order to create metadata for textual analysis or to train models for specific nlp tasks, researchers have been relying on the annotation performed by trained annotators or crowdsourced workers. recently, chatgpt was shown to outperform crowdsourced workers in annotating political topics, affil-iation, and policy frames @xcite . however, it is not yet clear whether a similar prompting approach can also be successful for classifying what discourse relation holds between two text spans. discourse relations (drs) are semantic-pragmatic links between clauses and sentences. they can be explicitly marked by discourse connectives (dcs), such as however and in addition, or they can be inferred from the text without relying on a specific marker -such cases are referred to as implicit relations. for example, there is a causal relation between the following sentences: mary lost her keys. therefore, she could not enter her office., and the same relation can still be inferred without the dc therefore. discourse relation analysis is useful for various downstream tasks, such as summarization @xcite and relation extraction @xcite , and discourse-annotated data serves as the basis of various linguistic research (e.g. @xcite . however, classifying implicit drs involves cognitive processing that is difficult even for humans in different languages @xcite @xcite @xcite and poses a challenge for nlp (e.g., 64.58% accuracy and 49.03% f1 on pdtb 2.0 in @xcite , even with powerful llms. ## methodology prompting llms to classify among specific labels typically requires listing all valid options. the input becomes even longer when an example per class is included for in-context learning. instead, we propose several methods to break down the 14way dr classification task into smaller sub-tasks, which are described in details below. ## discussion and conclusion we set out to test chatgpt's ability to infer implicit dr senses with the latest model and carefully engineered prompts. unfortunately, the low performance of implicit dr recognition could not be improved by sophisticated prompt engineering techniques that were successful in other tasks. this points to the fact that either other prompting tech-niques are needed, or that implicit dr recognition simply cannot be solved under zero-shot or fewshot settings. knowledge acquired in other reasoning tasks does not seem to be transferrable to this task and supervised guidance to map the semantics of the arguments to the ambiguous and abstract dr labels is necessary. we also performed smaller-scale experiments with other llms such as llama @xcite but the performance was substantially worse even than gpt-3.5. the training data of these other llms do not include pdtb nor discogem. we found that chatgpt is able to produce pdtb 2.0 labels even when the options are not provided in the prompt, suggesting that its training data should have at least included texts related to pdtb-style dr analysis (e.g., possibly an annotation manual or research article). therefore, strictly speaking, the inference made by chatgpt is not completely zeroshot because it is informed about the dr labels. this may explain why the two-step dc insertion prompt, which does not involve any dr labels at all, totally failed in the task. the underperformance of the per-class binary prompt suggests that prompting the discriminative comparison among all possible options at once is more accurate than separate detection of individual dr sense. too many relation senses were rejected when the model was presented with the binary choice of yes/no; some of these rejected senses have been accepted when compared with an even more unlikely sense. the per-class approach, nevertheless, provides a framework to collect multi-label annotations, which is not only important to dr annotations but also to other tasks like natural language inference and sentiment analysis. we also experimented with running the mc prompts multiple times with a higher temperature setting, or explicitly asking for multiple labels in the prompt. chatgpt only occasionally produced multiple labels in these cases, possibly due to the dominance of single-label annotated data in its training history. the better performance of the per-class verification approach compared with binary questions shows that the verification questions actually worked. this approach is related to chain-ofthought prompting @xcite ; the identification of the arguments of the level-3 sense justifies the presence of the level-2 relation. we will experiment using this approach to refine the mc prompt. another direction is to develop other approaches to disassemble the dr annotation task. breaking down the multi-way classification task into smaller tasks was successful in dialogue structure annotation @xcite , using a heavily engineered step-by-step scheme (e.g. > 6 steps, each asking for specific features of the input). such a tailored annotation scheme might also be necessary to prompt implicit dr annotations.