The effects of distance on NPI illusive effects in BERT (2024)

Abstract: Previous studies have examined the syntactic capabilities of large pre-trained language models, such as BERT, by using stimuli from psycholinguistic studies. Studying well-known processing errors, such as NPI illusive effects can reveal whether a model prioritizes linear or hierarchical information when processing language. Recent experiments have found that BERT is mildly susceptible to Negative Polarity Item (NPI) illusion effects (Shin et al., 2023; Vu and Lee, 2022). We expand on these results by examining the effect of distance on the illusive effect, using and modifying stimuli from Parker and Phillips (2016). We also further tease apart whether the model is more affected by hierarchical distance or linear distance. We find that BERT is highly sensitive to syntactic hierarchical information: added hierarchical layers affected its processing capabilities compared to added linear distance.

## introduction the recent proliferation of large language models, such as bert @xcite , have inspired investigations into these models' linguistic behavior. bert, a ubiquitous baseline model in nlp experiments, provides context-based representation of text data. numerous studies have attempted to reveal how accurately these language models simulate human behavior. one of the significant challenges in sentence processing is the resolution of long-distance syntactic dependencies between words or phrases. here we study the processing of long-distance dependencies in bert to better understand the limitations and strengths of transformer-based language models. we focus on the illusion effect in negative polarity item (npi) licensing. we evaluate bert on a psycholinguistic dataset that examines the effect of distance on npi illusive effects @xcite , and additionally investigate the nature of the distance that affects npi illusive effects. we find that bert is more affected by hierarchical than linear information. ## experiment 1 4.1 stimuli to set a baseline for the effect of distance on npi illusive effects, we used the stimuli from the psycholinguistics study by @xcite . one example from the data set is in if bert is to behave similarly to human subjects, we expect it to output the same surprisal scores for illusive and ungrammatical sentences in the long distance condition, but not in the short distance condition. ## experiment 3 in experiment 2, we found that hierarchical distance affected bert's capacity to distinguish between grammatical and ungrammatical sentences, more so than linear distance when looking at accuracy, but not when comparing average surprisal scores. we suspect that the size of the illusive effect in the linear distance condition was almost undetectable due to the hierarchical distance between linear hierarchical @xmath0.2% @xmath1.2% 44.4% ## discussion our study shows three main results. first, we have replicated the results in parker and phillips (2016) that have shown that npi illusive effects are modulated by the distance between the licensor and the npi. ## conclusion in this paper, we have conducted experiments inspired by psycholinguistic studies to examine the sensitivity of the pre-trained bert model to hierarchical information. in particular, we studied the effect of distance on npi licensing illusions in pre-trained bert and designed our own stimuli to tease apart whether it is the hierarchical or linear distance that mattered. we found that bert in fact displays some illusive effects, meaning that it did not perfectly learn the correct structural generalization for npi-licensing, but at the same time remained sensitive to hierarchical distance and not linear distance.