Understanding Demonstration-based Learning from a Causal Perspective (2023)

Abstract: Demonstration-based learning has shown impressive performance in exploiting pretrained language models under few-shot learning settings. It is interesting to see that demonstrations, even those composed of random tokens, can still improve performance. In this paper, we build a Structural Causal Model (SCM) to understand demonstration-based learning from causal perspectives and interpret random demonstrations as interventions on the demonstration variable within the causal model. We investigate the causal effects and find that the concurrence of specific words in the demonstration will induce bias, while randomly sampled tokens in the demonstration do not. Based on this finding, we further propose simple ways to construct random demonstrations, which even outperform hand-crafted, meaningful demonstrations on public sequence labeling benchmarks.

## introduction large pretrained language models (plms) have recently shown great progress @xcite @xcite @xcite . these models, such as gpt-4 @xcite , palm @xcite , and llama @xcite , have shown human-level capability with only a few illustrative examples @xcite . specifically, demonstration-based learning has been introduced to augment the input with demonstrations, i.e., the input and expected output pairs. brown et al. (2020) simply picked up to a small number of sampled instances and directly concatenated them with the input to perform in-context learning. @xcite concatenated the input with task demonstrations to create augmented input and fed them into plms to obtain improved token representations to do sequence labeling in a classifier-based fine-tuning way. to better understand demonstration-based learning, we take a deeper dive into the random construction of demonstrations. specifically, we first build a structural causal model (scm) to understand demonstration-based learning from a causal perspective. a causal view is developed to explore the spurious correlations between demonstrations and few-shot training samples. based on the intervention on the demonstration variable in the scm, we design multiple simple and effective ways to construct random demonstrations. these methods are evaluated on structured prediction tasks with carefully designed experiment setups. empirical results show that carefully designed random demonstrations can outperform meaningful demonstrations under the few-shot learning setting. this finding suggests that meaningless demonstrations can still provide valid information for plms. moreover, random demonstrations allow the learning algorithm to identify important features and patterns in the data more effectively than homogeneous handcrafted demonstrations. sentence: the algerian war of independence marked the end of french colonial rule in north africa . ## demonstration-based learning from a causal perspective in this section, we give a specific example to show the potential bias and understand demonstrationbased learning from a causal perspective. specifically, we first introduce a structural causal model (scm) @xcite to describe the mechanism and identify the induced bias. then, we perform demonstration variable intervention and propose multiple simple and effective random demonstration templates inspired by our causal model. we observe that the frequent co-occurrence of tokens in the classical demonstrations generate harmful superficial patterns which is misleading to the model and leads to biased predictions @xcite . a specific example with different demonstrations is provided in ## conclusions in this paper, we present a casual view to understand demonstration-based learning. based on the structural causal model we constructed, we investigate the causal effects and discover that the concurrence of specific words in the demonstration can induce bias. to address this issue, we perform interventions by constructing random demonstrations. our empirical results indicate that carefully designed random demonstrations consistently outperform meaningful demonstrations on public sequence labeling benchmarks.