Learned Construction Grammars Converge Across Registers Given Increased Exposure (2021)

Abstract: This paper measures the impact of increased exposure on whether learned construction grammars converge onto shared representations when trained on data from different registers. Register influences the frequency of constructions, with some structures common in formal but not informal usage. We expect that a grammar induction algorithm exposed to different registers will acquire different constructions. To what degree does increased exposure lead to the convergence of register-specific grammars? The experiments in this paper simulate language learning in 12 languages (half Germanic and half Romance) with corpora representing three registers (Twitter, Wikipedia, Web). These simulations are repeated with increasing amounts of exposure, from 100k to 2 million words, to measure the impact of exposure on the convergence of grammars. The results show that increased exposure does lead to converging grammars across all languages. In addition, a shared core of register-universal constructions remains constant across increasing amounts of exposure.

## exposure and convergence the central question that this work aims to answer is whether register-specific grammars converge onto shared representations when exposed to more training data. variation in the context of production, called register variation, has a significant impact on the frequency of constructions. for example, imperative and wh-question constructions are much more frequent in informal or conversational speech, while declarative constructions are much more frequent in formal written usage @xcite . at the same time, usage-based grammar views language as a complex adaptive system that emerges given exposure to usage @xcite . thus, a language learner is expected to be strongly influenced by the observed frequency of constructions. given this wide variance in the frequency of constructions across registers, it is conceivable that learners exposed to different registers learn different constructions. this paper simulates the language acquisition process from a usage-based perspective by learning construction grammars, called cxgs @xcite @xcite . a constructional approach to language focuses on symbolic formmeaning mappings that are potentially idiomatic. previous work on computational cxg has explored how to discover potential constructions @xcite @xcite , the process of construction learning @xcite , and whether constructional information is implicitly encoded in models like @xcite ). a commonly discussed example of a construction is the ditransitive in (a1). cxgs use a constraint-based formalism in which each slot in the construction is defined by a particular slotconstraint; in (a1) these are syntactic constraints. one of the important ideas in cxg is that constructions themselves carry a meaning. for example, the ditransitive construction carries a meaning of transfer regardless of the meaning of the particular verb that is used in the ditransitive. in some cases, this notion of transfer also follows from the meaning of the verb, as in (a2) with sold. but, in other cases, utterances like (a3) can take on a meaning of transfer that is not present in the verb smile. constructions can also have idiomatic children, such as (a4), in which an item-specific slotconstraint defines a sub-class of the construction, like (a5), whose meaning is not entirely predictable or transparent given the parent construction. register is a distinct pattern of usage that is associated with the context of production. a substantial body of research has shown that register is a major source of linguistic variation @xcite . recent work has shown that the impact of register variation exceeds the impact of geographic variation in many cases @xcite . the result of register variation is that large corpora often contain a number of distinct sub-corpora, each with their own unique patterns of usage @xcite . in other words, a gigaword web-crawled corpus is not simply a flat collection of many written documents: there is, instead, a register-based grouping of sub-corpora which often contain significantly different linguistic forms. this paper simulates the acquisition of constructions by incrementally increasing the amount of exposure: 100k words, 200k words, 300k words and so on up to 2 million words @xcite @xcite . this provides a series of grammars, each representing a different state in the learning process. this experiment is repeated across three registers, each with a unique set of constructional frequencies: wikipedia (formal), twitter (informal), and web (mixed). each register has a progression of 20 register-specific grammars, with each grammar representing different levels of exposure. is there a level of exposure at which registerspecific grammars reach a stable shared representation of the language? on the one hand, it is possible that register-specific grammars are maintained as unique sub-sets of linguistic behaviour. in this case, grammars would not converge given increased exposure. on the other had, it is possible that registerspecific constructions fade away as increased exposure leads to more generalized grammars. in this case, grammars would converge, becoming more similar and less register-specific as they are learned through exposure to more training data. to avoid language-specific generalizations, this experiment is repeated across six germanic languages (danish, dutch, english, german, norwegian, swedish) and six romance languages (catalan, french, italian, portuguese, romanian, spanish). each grammar contains a set of constructions that have been learned to best represent the training data. thus, for each stage in the learning process, we can measure the overlap between register-specific grammars (twitter to web, twitter to wikipedia, and wikipedia to web). when register-specific grammars have a higher overlap this means that they share more of their constructional representations. in other words, higher overlap means that the grammars are more similar. we say that grammars converge when they have a higher degree of overlap or similarity. this paper develops two measures of grammar similarity to capture different aspects of convergence: a fuzzy jaccard similarity that captures convergence across even rare constructions and a frequency-weighted jaccard similarity that focuses on the core constructions. these two measures of convergence allow us to model the degree to which construction grammars learn register-specific representations. there are three possible outcomes for this experimental framework: first, it is possible that grammars converge as exposure increases. this convergence would indicate that register variation becomes less important given more data: the grammars contain the same constructions regardless of the input register. in other words, this would indicate that more data overall is able to compensate for register variation. second, it is possible that grammars do not converge as the amount of exposure increases. this outcome would indicate that each register represents a unique sub-grammar, a distinct set of linguistic behaviours. in this case, more data overall would never compensate for register variation. these two competing hypotheses are tested in experiment 1 (section 5). this experiment finds that increased exposure does, in fact, lead to converging grammars. this finding supports the first hypothesis that is described above. ## experimental design the basic experimental approach is to learn grammars over increasing amounts of exposure: from 100k words to 2 million words in increments of 100k words (thus creating 20 grammars per condition). this series of grammars simulates the accumulation of grammatical knowledge as the amount of exposure increases. this approach is repeated across each of the three registers that represent different contexts of production. the register-specific data used to progressively learn grammars is collected from three sets of corpora: tweets (tw), wikipedia articles (wk), and web pages (cc for common crawl). this dataset is summarized in the pairwise similarity relationships between grammars differ, in part, because some sources of data are more similar to one another. for example, cc and wk are more similar registers and thus their grammars have a baseline similarity that is higher than wk and tw. in other words, the wikipedia grammars (cxg-wk) are more similar to the web grammars (cxg-cc) than to the twitter grammars (cxg-tw). what matters, then, is the degree to which the relative similarity between grammars changes as the amount of exposure increases. this approach controls for the underlying similarity between registers. ## learning constructions the grammar induction algorithm used to learn constructions is taken from previous work @xcite . at its core, this model of cxgs has three main components: first, a psychologicallyplausible measure of association, the ∆p , is used to measure the entrenchment of particular representations @xcite @xcite . second, an association-based beam search is used to identify constructions of arbitrary length by finding the most entrenched representation for each training sentence in reference to a matrix of ∆p values @xcite . third, a minimum description length measure is used as an optimization function, balancing the trade-off between increased storage of item-specific constructions and increased computation of generalized constructions @xcite . we briefly review each of these components of the algorithm in this section. constructions are constraint-based syntactic representations in which individual slots are limited to particular syntactic, semantic, or lexical items @xcite . construction grammars are usage-based in the sense that constructions range from very idiomatic phrases (like "give me a hand") to very abstract sequences (like np -> det adj np). one of the many advantages of cxgs is that they represent actual usage, which means that they are capable of identifying syntactic variation across dialects @xcite and even across individuals @xcite . but the disadvantage is that the induction algorithm must learn both the units of representation (i.e., semantic categories) as well as these multi-dimensional slotconstraints. for example, the constructions in (b) and (c) in the grammar induction algorithm starts by defining the types of representation. lexical constraints are based on word-forms, without lemmatization. these are the simplest and most idiomatic types of constraints. syntactic constraints are formulating using the universal partof-speech tagset @xcite and implemented using the ripple down rules algorithm @xcite . semantic constraints are based on distributional semantics, with k-means clustering applied to discretize pre-trained fasttext embeddings @xcite . the semantic constraints in (b) and (c) are formulated using the index of the corresponding clusters. a complete list of semantic domains used in this paper, along with a grammar for english, are available in the supplementary material. each sentence in the input corpus is transformed into these three parallel dimensions of representation (lexical, syntactic, semantic). in the first stage of the algorithm, a co-occurrence matrix is produced that represents the association between all pairs of representations using the ∆p measure, shown below. what distinguishes the ∆p from more common measures like ppmi @xcite is that it has direction-specific variants that take ordering into account, thus helping to capture syntactic patterns. the measure, calculated left-to-right, is the probability that two units occur together (x and y ) adjusted by the probability that x occurs alone. in this notation, y p indicates that unit y is present and y a that unit y is absent. given (i) a three-dimensional representation of each sentence and (ii) a co-occurrence matrix with the directional association for each pair of representations, a beam search algorithm is used to find the most entrenched sequence of constraints for each sentence in the training corpus. the basic idea behind this search is to traverse all possible paths of constraints, ending each path when the cumulative ∆p falls below a threshold @xcite . for each sentence, the sequence of slot-constraints with the highest cumulative association is added to a provisional grammar. in cxg, some representations are very entrenched (grammaticalized) and others are only slightly entrenched @xcite . the optimum sub-set of constructions is then selected from this provisional grammar. the grammar induction model itself is based on the minimum description length paradigm @xcite @xcite . in this kind of model, observed probabilities are used to calculate the encoding size of a grammar (l 1 ) as well as the encoding size of a test corpus given that grammar (l 2 ). usage-based grammar posits a trade-off between memory and computation; this is modelled by mdl's combination of l 1 and l 2 encoding size @xcite . the best grammar is the one which minimizes this metric on a test corpus. in (2), g refers to the grammar being evaluated and d refers to the test corpus. for example, this is used to choose the parameters of the beam search algorithm described above. in practice, the use of mdl to evaluate grammars is quite similar to the use of perplexity to evaluate language models; for example, the mdl metric is specific to each test corpus. the advantage of the mdl metric for usage-based grammar is that it distinguishes between the complexity of the grammar (l 1 ) and the fit between the grammar and the test corpus (l 2 ). this induction algorithm provides a grammar of constructions to describe the training corpus, where the grammar is chosen to minimize the mdl metric. the grammar is a set of constructions, each of which is a sequence of slot-constraints. and each slot-constraint is formulated using the basic inventory of lexical, syntactic, and semantic fillers. in previous work, the induction algorithm used alternating training and testing sets to refine grammars. a large background corpus was used to estimate the ∆p matrix that guides the selection of slot-constraints. the experiments here, however, depend on limiting the amount of training data as a means of controlling for different levels of exposure. in each condition, then, the same training data is used for each stage in the algorithm. in other words, the 200k word exposure condition has access only to the 200k word training corpus (with the implicit exception of the pre-trained embeddings). each model is trained on the same underlying corpus, so that the 500k word condition is given the same data as the 400k word condition plus an additional 100k words of new data. ## measuring grammar similarity a grammar in this context is a set of constructions, where each construction is a sequence of slot-constraints. our central measure of overlap between grammars, then, is based on the jaccard similarity, where values close to 1 represent very similar grammars and values close to 0 represent very different grammars. one of the challenges with cxgs is that two different representations could use different slotconstraints to capture a similar set of utterances, essentially providing two versions of the same construction. consider the construction in (d), a part of the english web-based grammar. the tokens in (d1) through (d4) are tokens of this construction. an alternate constraint that specifies the rather than det might be chosen, leading to a different representation for the same underlying construction. the challenge for calculating convergence using the jaccard similarity between grammars is that similar constructions could capture the same set of tokens. for example, the syntactic constraint (det) could be replaced with a lexical constraint (the) in the construction in (d). the jaccard similarity on its own would not capture the similarity between these two alternate formulations of what is ultimately the same underlying construction. our first measure is thus a fuzzy jaccard similarity, in which the definition of set membership is extended to very similar constructions. in this measure, a sub-sequence matching algorithm is used to find how many slot-constraints are shared between two constructions, taking order into account. any two constructions above a threshold of 0.71 shared sub-sequences are considered a match. this threshold is chosen because it allows one slot-constraint to differ between most constructions while still considering them to be similar representations. for example, two six-slot constructions must share five constraints in order to count as a match at this threshold. this measure thus provides a better approximation of construction similarity, focusing on constructions with slightly different internal constraints or with an added slot-constraint in one position. our second measure is a frequency-weighted jaccard similarity, in which the importance of each construction is weighted relative to its frequency in an independent corpus. for each language, a background corpus is created from a mix of different registers: open subtitles and global voices (news articles) @xcite and bible translations @xcite . this background corpus represents usage external to the three main registers used in the experiments. the frequency of each construction is derived from 500k words of this background corpus, so that very common constructions are given more weight in the similarity measure. the basic idea is that some constructions are part of the core grammar, thus being frequent in all registers. this weighted measure captures convergence within this core grammar by focusing on those constructions which are most frequent in an independent corpus. these two measures based on jaccard similarity provide three values for each condition: cc-wk, cc-tw, and wk-tw. each of the values represents a pairwise similarity between two register-specific grammars. the higher these values, the more the learner is converging onto a shared grammar. we have thus formulated two measures of grammatical overlap. the first, fuzzy jaccard, captures the overall similarity between grammars. the second, frequency-weighted jaccard, captures the similarity between grammars with a focus on the core constructions that are frequent across registers (ignoring the long tail of register-specific forms). the following sections apply these measures of grammatical overlap to the cxgs exposed to increasing amounts of usage. in each case, higher values represent increased convergence between grammars. ## experiment 1: constructional overlap to what degree do grammars converge onto shared representations as they are exposed to increasing amounts of data from different registers? the first experiment uses the fuzzy jaccard similarity to model convergence. the results are shown in figure 1 and figure 2 . in both cases, the y axis represents the similarity between register-specific grammars, with higher values representing more convergent sets of constructions. note that each line represents the similarity between two registerspecific grammars. and the x axis represents the amount of exposure, moving from 100k words up to 2 million words. languages are labelled using their language code, as listed in we notice, first, that there is a baseline difference in the similarity between registers. in every language, for example, wk and tw are the least similar. and, in every language, cc and wk are the most similar. this pattern is shared across all 12 languages because the underlying contexts of production have this similarity, with wikipedia the most formal and twitter the least formal. the distance between registers does not matter here. what does matter is the relative change in distance as the algorithm is exposed to more data (i.e., cc-tw at 200k words and cc-tw at 2 million words). we notice, second, that in every language grammars converge with increased exposure. the overall similarity between constructions increases as more training data is observed. this is true for both germanic and romance languages, showing this to be a robust generalization. languages do differ, however, in (i) the overall amount of similarity and (ii) the rate of convergence. in the first case, within germanic languages the range of grammar similarity is generally comparable, starting at approximately 0.2 and ending at approximately 0.4. some languages (like english) reach a higher level of convergence. other languages have register-specific patterns: in norwegian the similarity between cc-wk is quite high throughout, but in swedish the similarity between cc-wk is the same as the similarity between cc-tw. in other words, the ordering of register similarity is constant across languages but the distance is not. romance languages have a generally higher overall rate of similarity, although there is a wide gap between french (the highest) and romanian (the lowest). this means that there is some variation across languages in terms of how similar the register-specific grammars become. there is also variation in the rate of convergence. some languages (like swedish) have a somewhat flat rate of convergence while other languages (like english) have a steeper curve. a flat curve represents a slow convergence while a steep curve represents a rapid convergence as exposure increases. some languages have bursts of convergence at specific amounts of exposure. for example, norwegian has a steep increase until about 300k words, then remains flat until about 1 million words before beginning to converge again. in other words, if we think about the growth curve of similarity, the pattern of convergence differs across languages. the cause of these differences is a matter for future research. the basic conclusion here is that all languages show increasing convergence with increasing exposure. in other words, register-specific syntactic patterns generalize as the induction algorithm encounters more training data. ## experiment 2: the core constructions although many constructions have varying frequency across different registers, we would expect that the grammar of each language also has a core set of very frequent constructions which are shared across all registers. in other words, register variation itself should not be strong enough to erase the syntactic generalizations provided by a grammar. the frequency-weighted jaccard similarity measure is used to find this core set of constructions: how much do grammars change with increasing exposure when we focus on the most frequent constructions? as explained above, the frequency weighting is derived from independent corpora that represent a different set of registers. the weighted similarity measures are shown in figure 3 and figure 4 , again with the y axis showing similarity (higher values mean more overlap between grammars) and the x axis showing increasing amounts of exposure. each line represents the similarity between two register-specific grammars. the overall level of similarity is significantly higher here. the germanic languages range from approximately 0.7 (dutch) to approximately 0.8 (swedish). the romance languages have the same range, with spanish the lowest and portuguese the highest. this overall increase in similarity shows that, when focusing on the core constructions, the register-specific grammars converge quickly. we notice, second, that the growth curve for the frequency-weighted measure shows very little change after a certain point. for many languages, like portuguese and italian, there is a sharp increase after several hundred thousand words of exposure. this indicates that the initial grammars (based on low exposure) are not adequate. after more exposure, however, the stable core of constructions is acquired. once that initial burst of acquisition is complete, there are no significant changes. this is not true for norwegian (which continues to show a continuous growth of convergence) or for romanian (which has an initial decline and a much later burst of similarity). but the overall pattern across all languages is that the core set of constructions remains stable after a small amount of exposure. ## conclusions these experiments show that register-specific grammars converge onto shared constructions as they are exposed to more training data. this is observed across 12 languages and three registers. at the same time, each language has a core set of frequent constructions which is not influenced by register variation. this core cxg is acquired for each language given a limited amount of exposure and does not change significantly as exposure increases. these results are important for describing the interaction between syntactic generalizations (the core grammars) and syntactic variation (the register-specific grammars).