Can Language Models Induce Grammatical Knowledge from Indirect Evidence? (2024)

Abstract: What kinds of and how much data is necessary for language models to induce grammatical knowledge to judge sentence acceptability? Recent language models still have much room for improvement in their data efficiency compared to humans. This paper investigates whether language models efficiently use indirect data (indirect evidence), from which they infer sentence acceptability. In contrast, humans use indirect evidence efficiently, which is considered one of the inductive biases contributing to efficient language acquisition. To explore this question, we introduce the Wug InDirect Evidence Test (WIDET), a dataset consisting of training instances inserted into the pre-training data and evaluation instances. We inject synthetic instances with newly coined wug words into pretraining data and explore the model’s behavior on evaluation data that assesses grammatical acceptability regarding those words. We prepare the injected instances by varying their levels of indirectness and quantity. Our experiments surprisingly show that language models do not induce grammatical knowledge even after repeated exposure to instances with the same structure but differing only in lexical items from evaluation instances in certain language phenomena. Our findings suggest a potential direction for future research: developing models that use latent indirect evidence to induce grammatical knowledge.

## introduction recent advances in language models, such as those from the gpt and llama families @xcite , have shown remarkable progress in various tasks. these models are trained on extremely large datasets, on a scale thousands of times greater than the amount of data children are exposed to in developing grammatical knowledge comparable to that of adults @xcite . this suggests substantial potential for improving their learning efficiency. however, it remains still unclear how the degree of indirectness in observed instances affects the number of occurrences required for language models to induce grammatical knowledge. pre-vious work has investigated how language models learn grammatical knowledge based on the appearance of items in training data focusing on the word frequency effect @xcite or generalization to unseen instances @xcite @xcite through few-shot learning or pretraining on corpora filtered by specific linguistic constructions. however, those methods face a limitation in identifying ways to enhance the model's learning efficiency. in this work, we explore the degree of indirectness and the amount of data needed for language models to induce linguistic generalization. to address this question, we introduce the wug indirect evidence test (widet), a dataset containing additional indirect training and evaluation instances. we train language models on pretraining data incorporating the indirect training instances. we then evaluate their linguistic generalization across seven different phenomena, including anaphor agreement, transitivity, and subject-verb agreement. these phenomena require language models to comprehend diverse properties and multiple parts of speech of specific words to judge their acceptability. to control the number of observed indirect training instances, we inject synthetic instances with newly coined words into pretraining data. following @xcite , we refer to these words that do not appear in the original vocabulary and data as wug words. 1 we use various synthetic data as additional indirect training instances, each differing in the degree of lexical and syntactic indirectness as well as the number of observations. we find that language models generalize linguistic knowledge from training instances identical to correct evaluation instances, though their data efficiency varies across different linguistic phenomena. this variation is likely influenced by the number of words between the wug and the words that act as cues for the model to learn its properties. we surprisingly observe that the language models do not induce grammatical knowledge in certain phenomena, even in instances that only differ in lexical items. syntactically indirect instances rarely induce the model's generalization. given that the distances between the wug and the cue words to learn its properties might cause inefficiency in the models' learning, we conduct a detailed analysis of indirect instances with complex interference, using anaphor gender agreement as a case study. we examine whether these instances affect the generalization, considering three factors related to attractors and distance, finding that when the language models are trained on the instances with complex interference, they hit a plateau in learning after sufficient observations. those findings from our controlled and comprehensive experiments suggest that, at least in our small-scale settings, language models do not generalize in a human-like manner even from the data with a degree of indirectness that seems intuitively manageable for humans, depending on language phenomena. our work contributes insights into language models' capacity to use indirect evidence for learning. to advance this in future research direction: implement a model that can use indirect evidence, enabling data-efficient language acquisition comparable to that of humans. foot_1 2 background ## wug indirect evidence test (widet) this section describes how we construct additional training and evaluation instances, which comprise our dataset, widet. following targeted syntactic evaluation @xcite @xcite , we employ minimal pair paradigm where pairs of sentences minimally differ in target words. the examples of instances are listed in ## analysis with more indirect instances in section 5, de induced the model's linguistic generalization but its data efficiency varies by linguistic phenomena. for anaphor agreement, the models' learning is more apt to reach a plateau in 25 -75 observations compared to other phenomena (see the figure for anaphor agreement in figure 2 ). this stagnation might be caused by the longer distance between the wug and the reflexives, whereas the relevant items are adjacent to each other in other phenomena such as transitive. to corroborate this negative effect of long distance on learning, we employ more indirect agreement instances to investigate whether the long distance hinders linguistic generalization on ana.gen.agr in language models. the difficulty of long-distance agreement is caused by attractors and distance @xcite . agreement attractors indicate the intervening words that distract the learner from judging the correct agreement @xcite . when language models judge the gender agreement, they would check if the word "<wug#@xmath0. distance refers to the number of the words intervening between the antecedent "<wug#@xmath1. attractor indicates the competing words (e.g., "man" in the case of at1 in the language models' grammatical knowledge concerning long-distance dependencies has been investigated in previous studies @xcite , and these studies argue that the models can indeed acquire the knowledge of long-distance agreement. however, the overall results on anaphor agreement in this study suggest that further investigation is required to reveal the relationship between models' performance and the distance of items relevant to correct judgment. for this purpose, we conduct a fine-grained analysis using synthetic sentences varying the distance between wugs and reflexive pronouns. ## conclusion we investigate the degree of indirectness and the amount of data required to induce human-like linguistic generalization in language models. we found that language models do not induce humanlike linguistic generalization even with a degree of indirectness that seems intuitively manageable for humans, depending on language phenomena. this limitation indicates a direction for future studies: implementing a model that can use indirect evidence, which will lead to data-efficient language acquisition comparable to that of humans.