Generalizations across filler-gap dependencies in neural language models (2024)

Abstract: Humans develop their grammars by making structural generalizations from finite input. We ask how filler-gap dependencies (FGDs), which share a structural generalization despite diverse surface forms, might arise from the input. We explicitly control the input to a neural language model (NLM) to uncover whether the model posits a shared representation for FGDs. We show that while NLMs do have success differentiating grammatical from ungrammatical FGDs, they rely on superficial properties of the input, rather than on a shared generalization. Our work highlights the need for specific linguistic inductive biases to model language acquisition.

## introduction human learners use their linguistic environment to acquire a grammar. at the same time, they come to generalizations that are not obviously signaled in the input. the central puzzle in language acquisition is to characterize the system that allows for human-like generalizations from finite input. linguists posit that these generalizations are achieved through shared representations that allow learners to treat superficially distinct phenomena as a class @xcite @xcite @xcite . the recent success of neural language models (nlms) has caused many to question the necessity of linguistically-specific representational systems in language learning @xcite @xcite . we address this renewed controversy by conducting two experiments to uncover whether nlms posit a shared representation for a particular syntactic dependency: filler-gap dependencies. we consider whether an nlm recognizes fillergap dependencies in superficially distinct con-structions, as humans do @xcite @xcite @xcite . we further ask whether the nlm posits a shared representation for filler-gap dependencies, and thus systematically applies constraints across them. recent research shows nlms can differentiate between grammatical and ungrammatical instances of filler-gap dependencies in individual constructions, but our study asks whether fillergap dependencies are treated as a class by the nlm. if a shared structural relation is learnable by an nlm, which lacks language-specific biases, then, in principle, a learner does not need to have such biases to learn that relation. although one could learn the correct pattern through piecemeal learning of each construction individually (given enough input), a shared representation across filler-gap dependencies would allow a learner to generalize from only a subset of constructions containing filler-gap dependencies. whether an nlm posits this shared representation is the question. we provide an nlm with direct evidence for a filler-gap dependency in one construction and test whether it generalizes to other constructions. in our first experiment, we augment an nlm's training data with specific instances of clefting, and in our second, topicalization. we compare performance on four constructions containing filler-gap dependencies: wh-movement, clefting, toughmovement, and topicalization. the nlm treating filler-gap dependencies systematically would be evidence that this shared representation is learnable without language-specific inductive biases. ## filler-gap dependencies filler-gap dependencies share a set of properties across superficially distinct constructions, including sensitivity to islands @xcite . these properties persist across constructions, despite variation in semantic contribution and discourse function @xcite . in psycholinguistic experiments, humans have been shown to be sensitive to gaps across filler-gap dependencies, including wh-movement @xcite , tough-movement @xcite , and clefting @xcite , though see @xcite for variation in english relative clauses. these effects are sensitive to locality constraints, and appear to be mediated by the presence of islands @xcite @xcite . generalizing from surface forms on the basis of a shared representation could be critical to learning, especially if some constructions containing filler-gap dependencies do not occur frequently in the input. clefting (1) is one construction that contains a filler-gap dependency. in (1a), the filler these snacks forms a dependency with the gap site, marked with __ for readability, but silent in natural language. the filler is interpreted as the object of bought, despite not appearing linearly beside bought in the string. strings lacking a filler but containing a gap (1b) are ungrammatical, and when the gap is filled (i.e., an object, such as cheese, immediately follows the verb), the acceptability pattern reverses (1c-d). in other words, neither a filler nor a gap can occur without the other. importantly, clefts are superficially similar to sentences like (1d) which lack a filler-gap dependency, and thus are structurally quite distinct. a learner must distinguish between instances of clefting (1a) and other superficially similar sentences (1d). it is these snacks that mary bought __ today. b. * it is apparent that mary bought __ today. c. * it is these snacks that mary bought cheese today. d. it is apparent that mary bought cheese today. filler-gap dependencies occur in many constructions, including wh-movement (2), topicalization (3), and tough-movement (4), which differ in surface form but share the filler-gap dependency and its properties. i know what mary bought __ today. while it might initially appear that learning could occur from simply expecting a gap when presented with a filler, properties of this dependency also include specific constraints on when they can be formed. some structural configurations, called islands, block the formation of a filler-gap dependency. for example, a filler-gap dependency cannot be formed inside a relative clause (e.g., that carried __) despite the fact that carried lacks an object (i.e., is followed by a gap). the relative clause blocks the dependency, and so a gap is unacceptable regardless of the presence of a filler. examples (5)-( 8 ) show that all filler-gap dependencies are subject to this same restriction. one task for a learner is to recognize, on the basis of grammatical examples only (e.g., (1a) and (1d), but not (1b), (1c), or (5)), when each fillergap dependency can and cannot occur. a further task is to recognize that the same properties apply to each construction containing a filler-gap dependency (2-4), and thus posit a shared representation underlying all filler-gap dependencies. an alternative method to generalizing would be a piecemeal learning process: learning each construction separately. for the piecemeal process to work, sufficient examples of each construction must occur in the input, and similar constraints across these constructions would arise from distinct observations. nlms here provide an opportunity to test whether a shared representation can in principle be extracted from the input without linguistic biases. ## discussion in this paper, we test whether nlms can generalize knowledge of filler-gap dependencies across different constructions when their input is augmented with only one construction containing a filler-gap dependency. the pattern of knowledge of the pretrained rnn potentially reflects piece-meal learning, where the frequency of particular constructions modulates the model's recognition of the filler-gap dependency for each construction type individually @xcite . however, based on the pretrained results alone we cannot determine whether the nlm's inferences for one construction are based on others, hence the need for an augmentation-based procedure. experiment 1 found that while cleft-rnn behaves differently than the pretrained rnn on clefting, wh-movement, and tough-movement, it fails to generalize systematically across all the types of filler-gap dependencies we test. cleft-rnn's failure to learn the relevant dependency for simple topicalization sentences further confirms that these models do not arrive at their knowledge of fillergap dependencies through a shared representation. cleft-rnn does improve its representation of island constraints in clefting, the construction it was augmented with. however, this improvement still preserves the incorrect prediction for grammaticality. in this case, positive evidence of grammatical forms is still insufficient for humanlike learning. this finding supports a conclusion drawn by @xcite : that given sufficient evidence of a construction, nlms can arrive at a correct representation of the constraints. however, the ability to generalize from one construction appears weak at best. cleft-rnn's filler effect for wh-movement is the only instance among our findings where an nlm achieves the most stringent measure of islands: a confidence interval overlapping with zero. however, we are cautious to over-interpret this finding: our test set for wh-movement was far smaller than that of the other construction types. further, the failure of cleft-rnn to capture islands in tough-movement relative to the pretrained rnn highlights a more pressing issue with nlms that rely only on surface distributions: exposure to one construction type can cause a degradation in an nlm's knowledge of a different type, when the two share superficially similar characteristics at odds with the dependency. the learner would then need even more positive direct evidence of the other type to offset such erroneous conclusions. we found that the nlm we tested fares worse at recognizing island constraints than past studies would suggest: both the pretrained rnn and our augmented models failed to arrive at the most stringent measure of islandhood in all cases but wh-movement in cleft-rnn. we suspect this is greatly influenced by the frequency of surface forms of particular constructions, as hypothesized by @xcite . why, then, did @xcite 's method of using the presence of fillers, gaps, and islands as predictors of surprisal yield a significant interaction for islandhood in instances where the filler effect suggested otherwise? we believe that the interaction collapses effects across different combinations of features. our nlms succeed with uges, which likely obscures their corresponding failure to recognize fges. however, the failure in fges suggests that the nlm is recognizing neither grammaticality nor the presence of an island. for fges, the filler effect is in the wrong direction; grammatical continuations (i.e., those without a filler or a gap) are more surprising than ungrammatical ones (a filler and no gap). this contradicts both the measures for islands proposed by @xcite and @xcite , who suggest that rather than no difference at the gap site, surprisal should align with grammaticality. here we find that surprisal does not align with either measure and is in fact showing the reverse pattern for grammaticality. the results of experiment 1 strongly suggest that the nlm arrives at its knowledge of filler-gap dependencies through piecemeal learning and that positive direct evidence of a filler-gap dependency in each construction is required to learn the dependency for that construction. we conducted experiment 2 to explore whether positive evidence of simple topicalization sentences is sufficient for topic-rnn to make predictions consistent with a human-like understanding of both the simple dependency and islands. topic-rnn learns to expect a gap given a filler, but fails to learn the other direction of the dependency: that in the absence of a filler, there should be no gap. the model's failure to learn the simple topicalization dependency even in the face of direct evidence is an additional challenge to claims that language-specific biases are not necessary to learn such dependencies. taken together, the results from experiments 1 and 2 show that nlms do not generalize from a shared representation to learn filler-gap dependencies. instead, they rely heavily on input that closely aligns with individual constructions. further, in cases such as topicalization, nlms appear to struggle with learning the dependency. our findings are particularly important as researchers consider in what ways nlms might and might not serve as good proxies for language learners. our work reiterates the importance of specific linguistic inductive biases to model language acquisition.