Neural reality of argument structure constructions (2022)

Abstract: In lexicalist linguistic theories, argument structure is assumed to be predictable from the meaning of verbs. As a result, the verb is the primary determinant of the meaning of a clause. In contrast, construction grammarians propose that argument structure is encoded in constructions (or form-meaning pairs) that are distinct from verbs. Two decades of psycholinguistic research have produced substantial empirical evidence in favor of the construction view. Here we adapt several psycholinguistic studies to probe for the existence of argument structure constructions (ASCs) in Transformer-based language models (LMs). First, using a sentence sorting experiment, we find that sentences sharing the same construction are closer in embedding space than sentences sharing the same verb. Furthermore, LMs increasingly prefer grouping by construction with more input data, mirroring the behavior of non-native language learners. Second, in a “Jabberwocky” priming-based experiment, we find that LMs associate ASCs with meaning, even in semantically nonsensical sentences. Our work offers the first evidence for ASCs in LMs and highlights the potential to devise novel probing methods grounded in psycholinguistic research.

## introduction pretrained transformer-based language models (lms) such as bert @xcite and roberta @xcite have recently achieved impressive results on many natural language tasks, spawning a new interdisciplinary field of aligning lms with linguistic theory and probing the linguistic capabilities of lms @xcite . most probing work so far has investigated the linguistic knowledge of lms on phenomena such as agreement, binding, licensing, and movement @xcite ) transitive bob cut the bread s v o s acts on o ditransitive bob cut joe the bread s v o1 o2 s transfers o2 to o1 caused motion bob cut the bread into the pan s v o path s causes o to move via path resultative bob cut the bread apart s v o state s causes o to become state with a particular focus on determining whether a sentence is linguistically acceptable @xcite . relatively little work has attempted to determine whether the linguistic knowledge induced by lms is more similar to a formal grammar of the sort postulated by mainstream generative linguistics @xcite @xcite , or to a network of form-meaning pairs as advocated by construction grammar @xcite . one area where construction grammar disagrees with many generative theories of language is in the analysis of the argument structure of verbs, that is, the specification of the number of arguments that a verb takes, their semantic relation to the verb, and their syntactic form @xcite . lexicalist theories were long dominant in generative grammar @xcite @xcite . in lexicalist theories, argument structure is assumed to be encoded in the lexical entry of the verb: for example, the verb visit is lexically specified as being transitive and as requiring a noun phrase object @xcite . in contrast, construction grammar suggests that argument structure is encoded in form-meaning pairs known as argument structure constructions (ascs, figure 1 ), which are distinct from verbs. the argument structure of a verb is determined by pairing it with an asc @xcite . to date, a substantial body of psycholinguistic work has provided evidence for the psychological reality of ascs in sentence sorting @xcite , priming @xcite , and novel verb experiments @xcite . here we connect basic research in ascs with neural probing by adapting several psycholinguistic studies to transformer-based lms and show evidence for the neural reality of ascs. our first case study is based on sentence sorting (bencini and @xcite ; we discover that in english, german, italian, and spanish, lms consider sentences that share the same construction to be more semantically similar than sentences sharing the main verb. furthermore, this preference for constructional meaning only manifests in larger lms (trained with more data), whereas smaller lms rely on the main verb, an easily accessible surface feature. human experiments with non-native speakers found a similarly increased preference for constructional meaning in more proficient speakers @xcite , suggesting commonalities in language acquisition between lms and humans. our second case study is based on nonsense "jabberwocky" sentences that nevertheless convey meaning when they are arranged in constructional templates @xcite . we adapt the original priming experiment to lms and show that roberta is able to derive meaning from ascs, even without any lexical cues. this finding offers counter-evidence to earlier claims that lms are relatively insensitive to word order when constructing sentence meaning @xcite . our source code and data are available at: https://github.com/spoclab-ca/ neural-reality-constructions . ## case study 1: sentence sorting this section describes our adaptation of the sentence sorting experiments to transformer lms. ## case study 2: jabberwocky constructions we next adapt the "jabberwocky" priming experiment from johnson and @xcite to lms, and make several changes to the original setup to better assess the capabilities of lms. priming is a standard experimental paradigm in psycholinguistic research, but it is not directly applicable to lms: existing methods simulate priming either by applying additional fine-tuning @xcite , or by concatenating sentences that typically do not co-occur in natural text @xcite . therefore, we instead propose a method to probe lms for the same linguistic information using only distance measurements on their contextual embeddings. ## conclusion we find evidence for argument structure constructions in transformer language models from two separate angles: sentence sorting and jabberwocky construction experiments. our work extends the existing body of literature on lm probing by taking a constructionist instead of generative approach to linguistic probing. our sentence sorting experiments identified a striking resemblance between humans' and lms' internal language representations as lms are exposed to increasing quantities of data, despite the differences between neural language models and the human brain. our two studies suggest that lms are able to derive meaning from abstract constructional templates with minimal lexical overlap. both sets of experiments were inspired by psycholinguistic studies, which we adapted to fit the capabilities of lms -this illustrates the potential for future work on grounding lm probing methodologies in psycholinguistic research.