Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (2022)

Abstract: Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive.Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought.Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning.With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations.Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration.Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

## introduction large-scale language models @xcite @xcite have shaped the nlp scene by introducing in-context learning (icl) @xcite as a novel approach to adapt language models for downstream tasks without explicit finetuning. icl enables language models to learn and predict from task-specific prompts that contain demonstrations in the natural language format, * equal contributions. # co-corresponding authors. ## looking deeper into ground-truth labels demonstrations of ground-truth labels foot_0 , correctly paired with inputs, have been known to be a crucial factor of supervised learning, but a recent work by @xcite purportedly revealed the possibly counter-intuitive nature of label demonstrations in in-context learning (icl). specifically, the findings implied that the correctness of input-label correspondence in in-context demonstrations is not as important as we have thought. we name this phenomenon input-label insensitivity. although the finding was supported by reasonably large-scale experiments, covering various experimental variables such as datasets, language models, in-context learning types, etc., we found that, through deeper analysis of the experiments, input-label insensitivity is not consistent across all experimental settings. the results from the prior experimental settings @xcite ) is shown on the left. "no demo" refers to the result without demonstrations and "random label" refers to the result with label demonstrations replaced with a random label uniformly sampled from the label space. minor variations in the experimental settings could result in a large difference in the degree of which the icl performance responds to the label corruption. more details on the experiment is described in appendix a. this section highlights the limitations of the existing work, proposes new metrics to quantify the impact of input-label correspondence, and finally presents deeper analyses of the icl experiments utilizing the newly proposed metrics. @xcite showed that replacing groundtruth labels in prompt demonstrations with incorrect labels marginally affects the mean-aggregated overall performance on selected datasets. although the input-label insensitivity phenomenon was less prominent on gpt-j with the direct icl method, the icl still performed better when entirely incorrect labels were given than the absence of demonstrations (the zero-shot baseline), allegedly supporting the input-label sensitivity idea @xcite . however, we argue that there are mainly two limitations to the existing claim. ## discussion this section provides additional evidence that the demonstration of ground-truth labels can be more important than the previous finding suggests and that existing interpretation of the experimental results may have been obfuscated by the entanglement of various aspects of demonstrations. ## related work as the scale of language models becomes larger @xcite @xcite , fine-tuning becomes prohibitively expensive due to the space and time complexities. as an alternative, in-context learning (icl) @xcite has shown to be an effective parameter-free learning strategy by prompting language models with task-specific prompt templates. since then, a plethora of works has investigated both the properties of the learning mechanism @xcite @xcite @xcite . although numerous efficient fine-tuning strategies have been proposed in the past @xcite @xcite , the absence of an explicit training step in icl has enabled it to retain its own class of adapting large-scale language models. ## conclusion and future work in this work, we took a closer look at how inputlabel relationships affect the in-context learning performance. to quantitatively analyze the impact of input-label mappings in in-context learning, we proposed novel metrics, gler and input-label sensitivity. through extensive experiments, we found that the integrity of the input-label mapping is a crucial factor in performing icl. we also conducted ablation studies to reveal various conditions that allow icl to improve insensitivity towards label corruptions (while still maintaining a healthy performance). for future work, based on the current findings, we will investigate whether we could exploit data augmentation for extremely low-resource situations for icl.