CAL a M o: a Constructionist Assessment of Language Models (2023)

Abstract: This paper presents a novel framework for evaluating Neural Language Models’ linguistic abilities using a constructionist approach. Not only is the usage-based model in line with the un- derlying stochastic philosophy of neural architectures, but it also allows the linguist to keep meaning as a determinant factor in the analysis. We outline the framework and present two possible scenarios for its application.

## introduction over the years, linguists have given a lot of thought to what language is, and how it can be best formally described. different approaches with sometimes contradictory aims have produced an extremely rich array of conceptual tools to describe linguistic phenomena. such tools play diverse roles in explaining the phylogenetic, ontogenetic or historicalcultural facets of language and are often heavily interlaced with one another. in this research landscape, computational modelling has largely been used to simulate and investigate speaker behaviour at various levels of granularity. a specific area within the computational community is known as (neural) language modelling, which aims at reproducing linguistic surface structure by means of (pseudo)-probabilistic models. neural architectures have played a special role in this subfield of research, due to their flexibility. the extreme complexity of theoretical tools found in linguistics gets cut down by order of magnitudes when it comes to the analysis of language processing using computational modelling. for instance, when the term language is mentioned in relation to artificial neural networks, it seems that the word is often used as a mere synonym of grammar: while it is clear from a broader theoretical perspective that the two objects do not overlap, the distinction gets blurred in many computational studies. that is, assumptions which would be clearly stated in theoretical linguistics (e.g. how grammatical abstraction fits into the concept of language), are not explicitly discussed by computational studies: it is often the case that a specific set of choices concerning the description of language are taken as default. most current work also seems to implicitly make a number of assumptions about what kind of grammar is supposed to emerge from neural language models (henceforth, nlms), and this underlying choice is often echoed in the most common evaluation settings and in the conclusions that are being drawn from such experiments. most of these default assumptions are inherited from the nativist chomskian tradition and the universal grammar (ug) framework @xcite , which has pervaded a lot of the computational work on grammar, and continues to do so in the recent literature on neural models. ironically, the nativist assumptions that permeate the mainstream computational methodology are at odds with the very nature of the models created by the field. neural models are essentially based on pattern learning and are completely agnostic about the nature of the data they are made to process. the idea that language can be abstracted from a general purpose statistical mechanism is more akin to usage-based (henceforth, ub) approaches @xcite @xcite , and nlms would provide a much more natural testbed for that theoretical strand. in the cognitive and ub accounts, the exploitation of predictability during language development (and again we refer to development at all the three tiers of philogeny, ontogeny and cultural evolution) is the root of a number of fundamental mechanisms such as schematization, entrenchment and distributional analysis @xcite . in the light of these processes, language, seen as a structured inventory of constructions, gets build through generations @xcite and throughout a speaker's lifetime: shared linguistic material among utterances, such as morphological markers for instance, enable the identification of particular patterns or constructions as units bearing meaning @xcite . the perceived gap between the nativist and nonnativist traditions with respect to computational modelling probably stems from historical factors. the chomskian school and its formal approach offered a definition of language that, in the past, could easily be interpreted and implemented by emergent computational approaches. but there is no reason for this bias to perdure. in this paper, we argue in favour of a usage-based framework to analyse language acquisition in anns. we first point out the aspects of nativist theories that have so far influenced the evaluation of nlms ( §2). we then introduce a framework for a quantitative and qualitative analysis of nlms linguistic abilities within the constructionist perspective ( §3). we finally show some preliminary analyses performed with the proposed formalization ( §4). ## nativist vs. non-nativist approaches to language acquisition all theories of language use and development recognize that at the root of human linguistic ability is the capacity to handle symbolic structures. but they disagree on the specific content of speakers' linguistic knowledge, the mode of acquisition of such content, and the extent to which linguistic productivity is affected by this stored knowledge @xcite . theories diverge with respect to three aspects: input, stability and systematicity. the perspective taken on each of these aspects has consequences for the conclusions drawn from nlms' responses to the evaluation setting. in the following, we consider each aspect in turn and specifically highlight how the evaluation of computational models becomes biased due to a lack of explicitness in relating experimental and theoretical aspects of the research question. input. one of the main arguments introduced by nativist frameworks is the poverty of the stimulus: the input children are exposed to is underdetermined and does not explain acquisitional generalizations observed in learners @xcite . such theories assume that children navigate a hypotheses space defined by innate constraints @xcite . constructionist approaches, instead, posit that language emerges from the input through domain-general mechanisms: this implies that the input is shaped and skewed in a specific way in order to enhance learnability (boyd and @xcite . a well established line of research has shown how children are proficient statistical learners @xcite @xcite . the emergence of language-like structure from purely linear signal has also been shown in recent experiments such as @xcite , which demonstrated how important aspects of the sequential structure of language may derive from adaptations to the cognitive limitations of human learners and users @xcite . the crucial difference between the nativist and the non-nativist approach here is how strict the relation between the received input and the acquired linguistic structure is: if we commit to a view in which the input only serves as a trigger of an almost pre-determined cognitive structure, we are naturally driving our attention far from the features of the input and primarily to the features of the structure. on the other hand, deriving the linguistic structure from the input structure itself requires investigating the two aspects together. so far, most studies on nlms have disregarded the effect of the input on experimental results @xcite . stability. the continuity assumption was first introduced by @xcite in order to reconcile aspects of developmental language with the generative framework. it posits that the differences between adult and children linguistic structures is negligible and merely due to performance factors. in contrast, what we can refer to as the developmental hypothesis claims that the mechanisms underlying acquisition remain the same throughout a life-long acquisition process, but the structures and abstractions they generate evolve over time. ub models also put emphasis on the linear and time-dependent nature of the linguistic signal @xcite . according to the ub account, generalizations appear gradually, as productivity emerges from item-specific knowledge @xcite . another aspect of stability is inter-speaker differences. ug posits that all speakers eventually converge to the same grammar @xcite . individual differences have however been found in almost every area of grammar, depending on a variety of factors including environmental ones @xcite . the 'sameness' assumption pervades the computational linguistics literature, where evaluation is performed according to a single 'gold standard' per task. for traditional tasks such as sentiment analysis or word similarity ratings, the annotations of human subjects are averaged, and the system is evaluated against the average. for language modeling, model perplexity is computed with respect to the statistical features of a large corpus, which aggregates the writing styles and linguistic habits of thousands of speakers. while this state of affairs has started to be criticised by various researchers, it remains for now the status quo. when considering language development as a speaker-dependent process, strongly affected by the nature of the input, an evaluation based on an 'average speaker' becomes truly unsatisfactory. we cannot assume the existence of a ground truth, and must rely on softer evaluation measures: it is clear that the linguistic behaviours of different speakers must overlap sufficiently to allow for communication, but that we also want to observe in the output of the network the kind of variability that is seen in humans. systematicity. the ability to understand and generate an unbounded number of novel sentences, using finite means, is considered one of the hallmarks of our language faculty. the boundaries of this systematicity remain however largely unclear: provided that we agree on what the finite means at our disposal are, not all the possibilities are actually realised by speakers and not all realised possibilities share the same cognitive or linguistic status. one way to look at systematicity is that of compositionality, for which the most widely known version is probably due to @xcite , that port chomsky's innateness theory to semantics: a set of rules or constraints is needed in order to systematically build the meaning of sentences by integrating meaning of words. even the montagovian formal approach to compositionality (montague, 1970) relies on chomskian-derived ideas of a stable lexicon that stores meanings, and the existence of a set of precise interpretation rules that allow for those meanings to be mixed and modulated through the filter of syntax. the core of both visions is still very much syntax-centered (to which semantics has to be isomorphic) and very little space is left for indeterminacy, negotiation between speakers and other aspects related to the interactive and communicative nature of language (different individuals can retain in fact quite differ-ent concepts associated to the same lexical label for instance, @xcite . in a nutshell, if we see systematicity from the standpoint of compositionality, the quasi-regularities of linguistic structure represents a major hurdle to surpass. quasi-regularity is instead the engine of productivity, as in the ability of speakers to use all the available linguistic means to cue the intended meaning. just like compositionality, productivity deals with the domain in which a grammatical pattern can be employed in a linguistic context without losing interpretability, and it deals with what is actually possible in the language and where to draw the boundaries of acceptability. the shift has not just been syntactic: in the formal representation of these two aspects of systematicity in semantics, for instance, composition-oriented @xcite or productivity-oriented @xcite theories have conceptualized the idea of selectional constraints differently. knowledge on systematicity is in both cases considered as implicit knowledge that the speaker has about their language. nativist approaches have however primarily dealt with compositionality, and so are nlms often evaluated: given grammar rules and lexicon, what are the computational mechanisms that allow them to combine? ub theories, on the other hand, have primarily been dealing with productivity: how far can meaning boundaries be forced? what are the mechanisms that allow for linguistic creativity? this of course entails, in the ub community, a relation to surface properties of the input as well: @xcite , for instance, note how the maximally schematic constructions, such as sbj verb obj, are also the most productive ones, and that this has a relation to their frequency too, both as a type and for each of their instantiations. ## calamo in our proposed methodology, calamo (constructionist assessment of language models), we incorporate the ub perspective across all three aspects: input, stability and systematicity. as far as input is concerned, calamo differs from standard approaches by considering input data an important factor in determining the shape of the learner's grammatical knowledge. in traditional scenarios, the input only serves as a triggering factor and its features play little to no role in the analysis. from a ub perspective, instead, the relation between the abstract grammatical structure of the input and the acquired grammar, which then constrains the production of the learner, is strict. regarding stability, depending on the view that is taken on the continuity hypothesis, we can see nlm's grammatical competence either as a binary or as a gradient property. in the first case, we test whether the network is able or not to handle some linguistic phenomenon, while in the second case, as advocated by calamo, we are interested in seeing how and why some linguistic aspect becomes more and more salient to the network during training. the compositionality vs. productivity perspectives, finally, entail a different organization of linguistic knowledge: the mainstream compositionality perspective tends to set meaning aside, and treat the lexicon as an organized repository of meanings (it makes sense therefore to test nlm's capabilities on semantically nonsensical sentences or to extend the known rules to completely unknown lexical items). in the productivity perspective, instead, meaning is intrinsically part of the process and is treated as a systematic aspect of grammar, too. ## exploratory experiments in order to explore the potential applications of the framework described in §3, we built a simple instance using the childes corpus @xcite as input data i and a vanilla character-based lstm @xcite as computational mechanisms c. with this simple setting, we explored two aspects: (i) we tested whether distributional similarities in λ i would influence the acquisition of constructicons λ 1 , • • • , λ n , and (ii) we tried to describe grammar as it emerges from a population of speakers. constructions were approximated through catenae @xcite : subtrees extracted from a dependency parsing syntactic representation (see figure 2 ). ## concluding remarks the nature of linguistic representations is a core issue in linguistic theories of language development. we feel this aspect has been overlooked in the nlms literature and propose an approach that brings back theoretical insights into the picture. we commit here to the ub constructionist framework, not as an ideal model of human language acquisition, but rather as a set of tools and categories that suffice to explain nlms' generated language. since learning a language largely overlaps with learning to process the input, there must be a relation between processing biases relating to certain types of constructions and the distribution of those constructions in the linguistic input @xcite . as experience grounds linguistic knowledge, distributional properties become a key aspect to determine the content of linguistic representations. in this framework, language is not considered as an autonomous cognitive system. rather, the acquisition of grammar is regarded as any other conceptualization process and knowledge of language emerges from use. to conclude, the observation of nlms linguistic abilities would benefit from a constructionist approach. the evaluation can take place at multiple levels and includes properties of the situation described by the linguistic signal, but also properties of the linguistic signal itself. the ub framework may in fact provide useful categories to analyze the statistical properties of artificial language learners, and most importantly allows us to examine the semantic and the syntactic layers in parallel, both in the input received by the learner and in the stochastic output it generates.