Production vs Perception: The Role of Individuality in Usage-Based Grammar Induction (2021)

Abstract: This paper asks whether a distinction between production-based and perception-based grammar induction influences either (i) the growth curve of grammars and lexicons or (ii) the similarity between representations learned from independent sub-sets of a corpus. A production-based model is trained on the usage of a single individual, thus simulating the grammatical knowledge of a single speaker. A perception-based model is trained on an aggregation of many individuals, thus simulating grammatical generalizations learned from exposure to many different speakers. To ensure robustness, the experiments are replicated across two registers of written English, with four additional registers reserved as a control. A set of three computational experiments shows that production-based grammars are significantly different from perception-based grammars across all conditions, with a steeper growth curve that can be explained by substantial inter-individual grammatical differences.

## methods: computational cxg the grammar induction experiments in this paper draw on computational construction grammar @xcite . in the construction grammar paradigm, a grammar is modelled as an inventory of symbols of varying complexity: from parts of words (morphemes) to lexical items (words) up to abstract patterns (np -> det n). construction grammar thus rejects the notion that the lexicon and grammatical rules are two separate entities, instead suggesting that both are similar symbols with different levels of abstraction. in the same way as other symbols, the units of grammar in this paradigm consist of a form combined with a meaning. this is most evident in the case of lexical items, but also applies to grammatical constructions. for example, the abstract structure np vp np np, with the right constraints, conveys a meaning of transfer (e.g. kim gave alex the book). in order to extract a grammar of this kind computationally, an algorithm must focus on the form of the constructions. for example, computational construction grammars are different from other types of grammar because they allow lexical and semantic representations in addition to syntactic representations. on the one hand, this leads to constructions capturing item-specific slot-constraints that are an important part of usage-based grammar. on the other hand, this means that the hypothesis space of potential grammars is much larger. representing the meaning of these constructional forms is a separate problem from finding the forms themselves. the grammar induction algorithm used here employs an association-based beam search to identify the best sequences of slot-constraints @xcite . while a grammar formalism like dependency grammar @xcite must identify the head and attachment type for each word, a construction grammar must identify the representation type for each slot-constraint. this leads to a larger number of potential representations and the beam search has been used to explore this space efficiently. previous work has used the minimum description length (mdl) paradigm @xcite to describe the fit between a grammar and a corpus as an optimization function during training. with the exception of the use of semantic representations for slot-constraints, the meaning of constructions is not taken into account here. this is a necessary simplification. nonetheless, it is important to remember that -to the extent that these patterns are strong manifestations of association across slots -it is likely that they each possess a distinct meaning as well as a distinct form. the experiments in this paper are centered on sub-sets of corpora containing 100k words. this is significantly less data than previous work @xcite . the idea is to measure the degree to which the grammar itself changes when the induction algorithm is exposed to a more realistic amount of linguistic usage. because the impact of training size is not clear on the mdl metric, the grammars in this paper are based on the beam search together with an mdl-based metric for choosing the optimum threshold for the ∆p association measure @xcite used in the beam search. but a final mdl-based selection stage is not employed. previous work represented semantic domains using word embeddings clustered into discrete categories. to provide better representations for less common vocabulary items, the embeddings here are derived from fasttext @xcite , using k-means (the number of clusters is set to 1 per 1,000 words). thus, the assumption is that each lexical item belongs to a single domain. drawing on the universal part-of-speech tag-set @xcite , semantic domains are only applied to open-class lexical items, on the assumption that more functional words do not carry domain-specific information. the codebase for grammar induction is open source. foot_0 ## data and experimental design the basic experimental framework in this paper is to apply grammar induction to independent subsets of corpora drawn from different registers. we find the growth curve of grammars and lexicons by measuring the increase in representations as these individual subsets are combined. in this case, we examine the representations learned from between 100k and 2 million words in increments of 100k, for a total of 20 observations per condition. further, we measure the convergence of grammars by quantifying pairwise similarities within each condition. in this framework, a condition is defined by the data used for learning representations. for example, we examine the convergence of grammars learned from news articles by measuring pairwise similarity across 200 randomly selected combinations of unique sub-sets of the corpus of news articles. because of variation in registers, or varieties associated with the context of production @xcite , some grammatical constructions are incredibly rare in one type of corpus but quite common in another type @xcite . along these same lines, some registers have more technical terms and thus a larger lexicon with more rare words. both of these factors mean that the relationship between grammar and the lexicon could be an artifact of one particular register. to control for this possibility, the experiments in this paper are replicated across six registers, as shown in first, corpora representing unique individuals are taken from academic articles and from project gutenberg. in this condition, each additional increment of data represents a new speaker (e.g. dickens, followed by austen, followed by james). second, corpora representing aggregations of individuals are taken from the same registers; the difference here is that each additional increment of data does not represent a unique new speaker, only an increased amount of language use. third, background corpora representing other aggregations of individuals are taken from tweets, web pages, wikipedia articles, and news articles. these background corpora provide a baseline against which we compare variation in production-based vs perception-based models. does any observed difference between the production and perception conditions fall within the expected range observed within this baseline? in the first condition, production, each increment of data (100k words) represents the production of a single individual. in other words, a model trained on this sub-set of the corpus is a representation of only that one individual's production. a corpus of academic articles is drawn from the field of history @xcite . this corpus represents the ac-ind condition, meaning the academic register representing individuals. a corpus of books from project gutenberg is drawn from 20th century authors. this corpus represents the pg-ind condition, meaning the project gutenberg data organized by individuals. each grammar and lexicon in this condition is trained on the production of a single speaker. in the second condition, perception, these production-based corpora are contrasted with data from the same registers in which each increment of 100k words represents many unknown individuals aggregated together. in other words, a model trained on this sub-set of the corpus reflects the perception of a single individual exposed to many other speakers. the academic register is represented by the british academic written english corpus @xcite , drawn from proficient student writing. this provides the ac-agg condition, representing the academic register but with each increment an aggregation of many unknown individuals. the register of books is drawn from the same project gutenberg corpus, this time with at most 500 words in each increment representing a single author. this ensures that there is little individual-specific information present in the corpus. this variant provides the pg-agg condition, representing project gutenberg data as an aggregation of many individuals. to provide a baseline, these paired corpora are contrasted with four further sources which represent an aggregation of many unknown individuals: social media data from tweets (tw-agg), web data from the common crawl (cc-agg), wikipedia articles (wi-agg), and news articles, with no more than 10 articles from the same publication per increment (nw-agg). this range of sources ensures that the experiments do not depend on the idiosyncratic properties of a single register. each id in the growth of grammars and lexicons is simulated by creating the union of these independent sub-sets: for example, the grammar from dickens plus the grammar from austen plus the grammar from james. this means that, after observing 2 million words, the production-based condition has observed the union of 20 different individuals. this design is required to represent the productionbased condition because of the difficulty of finding 2 million words for many different individuals. this means that the perception-based condition at 2 million words samples from potentially tens of thousands of speakers while the production-based condition samples from just 20 speakers. thus, the growth curves potentially depend on the order in which different samples are observed. in other words, there is a chance that differences between growth curves are artifacts of particular orders of observation and not actual differences between corpora. to test this possibility, we simulate growth curves from 100 random samples for each condition. for each sample, we calculate the coefficient of the regression between the amount of the data and the number of representations, a measure of the growth curve. this provides a population of growth curves for each condition. we then use a t-test to determine whether this sample of growth curves represents a single population. in every case, there is no difference. this gives us confidence that the order of observations has no influence on the final results; the curves reported here are averaged across these 100 samples. ## measuring growth curves and grammatical overlap the growth of the lexicon is expected to take a power law distribution in which the number of lexical items is proportional to the total number of words in the corpus, as shown in (1). the challenge in understanding the rate of growth, then, is to estimate the parameter α. the simplest method is to undertake a least-squares regression using the log of the size of the corpus and number of representations, as show in (2). on some data sets, this method is potentially problematic because fluctuations in the most infrequent representations can lead to a poor fit at certain portions of the curve @xcite . we validated the experiments in this paper by conducting comparisons between estimated α parameters and synthesized data following heap's law. these comparisons confirm that the traditional least-squares regression method provides an accurate measure of the growth curve. the first question is the degree to which there is variation in the α parameter across representation type (grammar vs lexicon) or condition (production vs perception). for each case, such as perceptionbased grammar induction from news articles, we calculate the growth curve as described above using least-squares regression on the mean growth curve. we then report both the estimated α and the confidence interval for determining whether differences in the parameter values are significant. the second question is the degree to which the representations from individual sub-sets of a corpus agree with one another. to measure this, we use the jaccard distance between grammars, shown in (3). to calculate the jaccard distance, we first form the union of the two grammars being compared and, second, create a vector for each with binary values indicating whether a particular item is present or not present. the jaccard distance then measures the difference between these binary vectors, with higher values indicating that there is more distance between grammars and lower values indicating that the grammars are more similar. ## experiment 1. growth curves across grammar and the lexicon we begin by measuring the difference between growth curves for the lexicon and for grammars. here we compare each of the six perception-based conditions, to see the range of behaviours across registers. this is shown in figure 1 , with the x axis showing the increasing amount of data (from 100k words to 2 million words) and the y axis showing the increasing number of representations (to a max of 80k lexical items). the red line represents the grammar and the blue line represents the lexicon. each of the perception-based conditions (i.e., each register) is represented by a separate plot. this figure shows that the lexicon grows much more quickly than the grammar. this is somewhat expected because, even though both of them are symbols in the construction grammar paradigm, they are symbols of different complexity and may have different behaviors. the other important observation is that lexical items can only be terminal units in the slots of grammatical constructions, which again suggests that the number of different terminal units should be larger than the number of grammatical constructions. the growth of both lexicon and grammar is visualized by the slope of the lines, with a steeper curve showing quicker growth. further, the grammar generally levels off, with the rate of growth slowing more quickly as the amount of data increases. in other words, as we observe new data, we are less likely to continuously encounter new constructions as we are to encounter new lexical items. there is general agreement across registers, except that the corpus of news articles shows a grammar that grows much more quickly, reaching a total of 37k constructions. this is a significantly larger grammar than any of the other registers. we also see variation in the lexicon, with the vocabulary on wikipedia growing at the quickest rate. ## experiment 2. perception vs production in growth curves our next experiment takes a closer look at the difference in the growth curves under our two conditions, production (structured around individuals) and perception (structured around aggregations of individuals). the results are shown in figure 2 , again with the growth in number of representations (types) on the y axis and the amount of data observed (tokens) on the x axis. the top row presents the lexicon and the bottom row the grammar. finally, the blue line represents the perception condition while the red line represents the production or individual condition. the growth of the lexicon does not show any striking differences. in the academic register (ac), the perception condition shows a faster growth rate; but in the book register (pg) the reverse is true. but the growth of the grammar shows a marked difference: the production-based grammar (in red) grows more quickly in both conditions. this is formalized in it is important to remember that the grammar induction algorithm is applied independently to each sub-set of the data. what this result shows, then, is that there are considerable individual differences or idiosyncrasies in the grammar but not in the lexicon. in both registers, grammar induction based on the production of individuals acquires more constructions given the same amount of exposure. this is important because most computational approaches to language learning assume that speakers generalize toward a single shared grammar. this implies, incorrectly, that the presence of many speakers in the training corpora is irrelevant, perhaps with the further constraint that each training corpus should represent a single community and register (like written british english). ## experiment 3. perception vs production in grammar similarity the previous experiments have focused on the size and growth of the grammars without focusing on the presence of individual representations (i.e., constructions). to what degree do the grammars from each sub-set of a corpus overlap? is there a significant difference between the overlap of perceptionbased and production-based representations? the basic idea in this experiment is to take a closer look at the higher growth curve in production-based grammars identified in the previous experiment: it is possible that a few of the grammars are unique, thus contributing to a higher growth curve, without a pervasive uniqueness distributed across all of the production-based grammars. this experiment consists in creating pairs of grammars under the two conditions. first, we sample 200 pairs drawn from each condition/register: for example, a pair from different sub-sets of the corpus of news articles. second, we use jaccard distance to measure the similarity of each pair. each comparison is made within a single register, thus controlling for the possibility of register variation. this provides a broader population of pairwise similarities, allowing us to measure the uniqueness of individual grammars in each condition. we visualize the distribution of grammar similarities using a violin plot in figure 3 . the distance measure ranges from 1 (no overlap) to 0 (complete overlap). the violin plot here shows the distributions, with width representing the density for a particular value and height representing the range of values. this shows, for example, that the ac-ind condition is not normally distributed. rather, it has a large range of values with two slight peaks. the ac-agg condition, however, is normally distributed, with a large peak at its mean (shown here by the dotted line in the center). the values for the jaccard distances show that, independently of condition, these pairs of grammars are relatively dissimilar. there are many reasons why this is the case, ranging from the amount of data used to train each grammar to the possibility that constructional representations overlap with slightly different slot-constraints. putting aside the baseline similarity that is observed using this particular measure, the larger point is that there is a clear distinction between production-based and perception-based grammars. this figure shows a clear distinction between the production-based (ind) and perception-based (agg) conditions. the grammars learned from individuals vary widely among themselves: some pairs have a high overlap but others a low overlap. furthermore, the most similar pairs in the individual conditions are as similar or less similar than the average pair for the aggregated condition. this indicates that there are individual differences in these grammars, the same phenomenon that resulted in the higher growth curves identified in the second experiment above. the perception-based grammars, however, have a low degree of variation: the similarity measures are centered densely around the mean because most grammars have the same degree of similarity. this means that the aggregated or perception-based condition is forcing the induction algorithm to converge onto more stable representations by exposing it to many individuals. the inverse of this generalization is that individuals have unique or idiosyncratic constructions which are only revealed when the training corpus is centered around that individual. this finding fits well with studies in variation @xcite , dunn2019a which reveal the high degree of syntactic differences across speech com- we also notice in figure 3 that the news register, although part of the perception-based condition, is not as densely centered as the other background registers. this shows the importance of including many registers in a study like this. the likely reason is that different publications enforce their own stylistic conventions. this data set is balanced to ensure that no single publication venue accounts for more than 10 of the articles in any sub-set of the corpus. it remains the case, however, that the presence of a publication-specific style may simulate a different distribution of grammar overlap. we formalize this violin plot in these bayesian estimates reinforce the visualization and show that there is more variance and thus more individual differences within grammars that are trained from the production of a single individual. this experiment thus confirms what is suggested by the increased growth curves seen in the second experiment: production-based grammars diverge into more individual-specific representations. ## discussion and conclusions the three computational experiments in this paper have shown that there is a significant difference between perception-based and production-based grammar induction, even when these conditions are contrasted across many registers. grammars based on individuals (i) have a significantly steeper growth curve and (ii) a significantly more longtailed distribution of pairwise similarity. we have also seen that the growth curve of the grammar in general does not have the same α parameter as the lexicon, but does still conform to the generalizations provided by heap's law. this supports the idea of a continuum between grammar and the lexicon, with the symbolic representations in the grammar more complex and more abstract, thus showing a slower growth curve. the results obtained by the three experiments overall reveal that, given a certain number of word tokens, the number of constructions extracted is higher if the sample is taken from one unique individual as opposed to a set of unknown individuals. for example, 100k words of data from academic prose written by the same individual contain 1845 construction types, while the same amount of data from a combination of individuals contains about 1512 construction types, a difference of 333. this is not a trivial result: as a counter-factual, it would also be plausible to expect that the aggregated data would contain a wider variety of constructions because it represents a wider variety of individuals. these results therefore suggest that the constructions that are normally observed in traditional (aggregated) corpora are just the tip of the iceberg: there are many individual-specific constructions that are never observed in aggregated production. in other words, the uniqueness of individual construction grammars is disguised when observing the aggregated usage of many individuals. these findings are consistent with the usagebased proposal that the general grammatical representation of a language emerges as a complexadaptive system @xcite . the grammars learned in the perception-based condition contain fewer construction types and are relatively similar to each other. however, these seemingly homogeneous grammars are in fact formed from the shared usage across a number of different individuals. and, as shown in the production-based condition, these aggregated individuals on their own are likely to use very different grammars.