Towards More Natural Artificial Languages (2022)

Abstract: A number of papers have recently argued in favor of using artificially generated languages to investigate the inductive biases of linguistic models, or to develop models for low-resource languages with underrepresented typologies. But the promise of artificial languages comes with a caveat: if these artificial languages are not sufficiently reflective of natural language, then using them as a proxy may lead to inaccurate conclusions. In this paper, we take a step towards increasing the realism of artificial language by introducing a variant of indexed grammars that draw their weights from hierarchical Pitman-Yor processes. We show that this framework generates languages that emulate the statistics of natural language corpora better than the current approach of directly formulating weighted context-free grammars.

## introduction in the world atlas of linguistic structures, @xcite reports that the plurality of world languages follow a subject-object-verb (sov) word order. however, relatively few sov languages (japanese, turkish, persian) have a significant internet footprint. today, the internet is dominated by subjectverb-object (svo) languages like english, spanish, and chinese. the resulting paucity of non-svo data makes it difficult to study whether linguistic models have an inductive bias towards particular word orders, or to develop models that perform well on low-resource languages from underrepresented linguistic families. in recent work, wang and @xcite , @xcite and white and @xcite argue that artificial languages could be an effective tool for addressing challenges like these, enabling researchers to create large corpora that manifest targeted linguistic phenomena. an obvious objection presents itself: what if the models aren't realistic enough? if not, then conclusions drawn from artificial languages may not transfer to natural languages. one response to this objection would be to abandon the entire enterprise, and with it the potential advantages of simulated data. an alternative is to follow the tradition of other disciplines who model natural systems (e.g. physics, geology, meteorology) and iterate on these models until they are sufficiently good predictors of observed phenomena. in this spirit, this paper builds upon the framework of white and @xcite , who used weighted context-free grammars to construct artificial languages for studying the inductive biases of neural language models towards particular word orders. observing that their framework did not account for selectional preference (the linguistic phenomenon that head words and their syntactic dependents are not probabilistically independent), we generalize weighted context-free grammars by introducing the weighted random-access indexed grammar, which facilitates the development of artificial languages that manifest selectional preference. we also present a methodology for building grammars that emulate statistical relationships observed in natural language corpora. inspired by @xcite , we use hierarchical pitman-yor processes @xcite as the token-generating distributions for open-class categories (like noun, verb, and adjective). we set the hyperparameters by matching the statistics of the produced artificial languages with natural language corpora. as a pilot experiment for our framework, we partially replicate an experiment performed by white and @xcite that studied the inductive bias of transformer and lstm-based language models towards languages featuring various syntactic parameter configurations @xcite . finally, we accompany this paper with a python package called testperanto foot_0 , to allow researchers to use and refine our framework for further linguistic studies. ## related work both wang and @xcite and @xcite constructed artificial languages by manipulating sentences from existing natural language corpora. both approaches made use of a dependency parser (or a gold parsed corpus) to inform these manipulations, altering syntactic constituent order @xcite or token morphology @xcite . white and @xcite argued that manipulated natural language corpora have downsides. based on a series of negative results @xcite , they suggested that it may not be possible to remove confounding linguistic features from an existing corpus, making it difficult to isolate typological features for study. to maximize the ability to run a controlled experiment, they generated fully artificial languages from hand-built weighted context-free grammars. however, although their grammars modeled certain syntactic dependencies (e.g. conjugating a verb with its subject), they did not model semantic dependencies. we assert that it is prohibitively difficult to directly formulate weighted context-free grammars that model semantic dependencies (e.g. selectional preference), motivating our extension -the weighted random-access indexed grammar. ## motivation white and @xcite generated artificial language using a weighted context-free grammar (wcfg). a wcfg augments a context-free gram-mar (cfg) with a function q that assigns a nonnegative weight q(r) to each grammar rule r. this induces a weight for each derivation: the product of the weights of the rules used in the derivation. more formal details can be found in @xcite . wcfgs produce terminal symbols (words) according to probability distributions that depend exclusively on the grammar nonterminals. consider the following cfg: by using plain nonterminals like vb and nn, the respective probabilities of sentences it drank water and it drank food depend only on the probability of the rules @xmath0. crucially, the verb choice does not differentiate the sentence probabilities. this is unrealistic -it is more common to drink water than to drink food, whereas it is more common to eat food than to eat water. this phenomenon (that linguistic arguments are not independent of their predicates) is known as selectional preference. one way to detect selectional preference @xcite is to collect dependency relationships from a parsed natural language corpus (e.g. amod, nsubj, dodj) and extract the dependency bigrams (e.g. for amod, the first three dependency bigrams in europarl are internal market, european citizens, and cultural exception). then, as we stream through the dependency bigrams, we plot either the number of observed bigram types -yor process. nn[] is the global noun distribution. nn[1] and nn[2] respectively represent the likelihood that a noun is the subject or object of a verb. nn[2, 27] and nn[2, 28] respectively represent the likelihood that a noun is the object of verb 27 (eat) or verb 28 (drink) of the vocab. we would like to generate artificial languages such that the dependencies have similar statistics to naturally observed dependencies. rather than independently generating open-class words, @xcite suggests using a hierarchical pitman-yor process @xcite ) -a tree-structured set of distributions over the same domain, in which child distributions are resamplings of their parents. figure 2 shows an example. a hierarchical pitman-yor process allows us to model context-specific word distributions (e.g. food is more likely to appear as the object of the verb eat than water, i, or me) that 2 to generate figure 1 , we shuffled the europarl sentences and extracted the adjective-noun dependencies using spacy. the shuffling smooths irregularities caused by topic shift. variable b n+1 determines whether the (n + 1)th sample is set to the value of a previous assignment (b n+1 = 0) or the next unused y i sample (b n+1 = 1). now define t 1 = 1 and consider j, @xmath1. if b n+1 = 0, then let t n+1 = j with probability: otherwise, if b n+1 = 1: the nth sample drawn from the pitman-yor process is x @xmath2. a pitman-yor process, for all practical purposes, can generate an "open-class" of words by using a uniform base distribution p unif with a sufficiently large vocabulary size v (for our experiments, we use the space of all 32-bit integers). a hierarchical pitman-yor process is simply a pitman-yor process that uses another pitman-yor process as its base distribution. for instance, we could define a global adjective distribution p @xmath3.4, 500, p unif ), and then for noun y 1 of our vocabulary, we could define a noun-dependent adjective distribution p adj,y 1 = py(d, θ, p adj ). ## approach the main challenge: how do we construct a wcfg that derives its weights from the linked distributions of a hierarchical pitman-yor process? concerned with the induction of better n-gram language models, previous work @xcite ) mainly focused on how to incorporate hierarchical pitman-yor processes into sequential models like hidden markov models. here, our concern is how to incorporate these distributions into a generative syntactic model convenient for engineering artificial languages with specific linguistic typologies. there exist many syntactic models to choose from, including dependency grammars (eisner, 1996), tree-adjoining grammars @xcite , lexical functional grammars @xcite , ccgs @xcite , hpsgs @xcite and gpsgs @xcite . in this work, we choose to extend context-free grammars, partly because of their popularity and partly to facilitate comparison with (white and @xcite , who used wcfgs -however, our approach can be adapted to other syntactic formalisms. ## demo: simple german syntax with selectional preference to demonstrate how linguistic phenomena can be modeled by a wrig, we present a small example in figure 5 , whose distribution table is given by figure 6 . it models various aspects of german syntax: word order (independent clauses are svo, whereas dependent clauses are sov), verb conjugation (present singular and present plural), and case roles (nominative and accusative). figure 7 shows the first five sentences of a corpus generated by the wrig. to interpret the indexed nonterminals, note that subject count (1=singular, 2=plural) and case (1=nominative, 2=accusative) are encoded as integer indices: • s[y 1 , y 2 ], ic[y 1 , y 2 ], dc[y 1 , y 2 ]: respectively produce a sentence, independent clause, and dependent clause with subject count y 2 , whose head is the y 1 th verb of the vocabulary • np[y 1 , y 2 , y 3 ]: produces a noun phrase with count y 2 and case y 3 , whose head is the y 1 th noun of the vocabulary • vp(d)[y 1 , y 2 ]: produces a (dependent clause) verb phrase with subject count y 2 , whose head is the y 1 th verb of the vocabulary • nn[y 1 , y 2 , y 3 ]: produces the y 1 th noun of the vocabulary, declined for count y 2 and case y 3 • vb[y 1 , y 2 ]: produces the y 1 th verb of the vocabulary, conjugated for subject count y 2 • dt[y 1 , y 2 ]: produces a determiner for a noun with count y 1 and case y 2 terminal rules for open-class nonterminals nn[y 1 , y 2 , y 3 ] and vb[y 1 , y 2 ] are generated by a voicebox that randomly concatenates german syllables to create new words, and adds german morphological endings based on count and case. for the closed-class dt[y 1 , y 2 ], the voicebox generates the german definite determiner for the specified count and case. for instance (see figure 7 ), the noun hunghub foot_4 appears as den hunghub when it is accusative singular and die hunghuben when it is accusative plural. by associating the noun distributions with the distributions of a hierarchical pitman-yor process, we also model selectional preference. by assigning a pitman-yor process of very low strength (0.1) to the verb-dependent object distributions, we enforce unusually strong selectional preference between verbs and objects, allowing us to see its manifestation of in just a small sample of generated sentences (figure 7 ). in particular, the invented verb milchsichkeiten frequently takes the noun hunghub as its object. ## experiment: word order bias as a pilot study of our framework, we re-created an experiment performed by @xcite , who used wcfgs to investigate the inductive biases of neural language models for various word orders exhibited by natural language. we created a wrig based on their wcfg description, which produces simple declarative sentences with relative clauses, prepositional phrases, and clausal complements. we used a voicebox that assigned concatenations of random syllables to each generic noun, verb, and adjective. it used english prepo-singleton type-token proportion ratio dev test dev test amod base 0.099 0.094 0.23 0.23 ours 0.0074 0.013 0.016 0.018 nsubj base 0.045 0.057 0.083 0.12 ours 0.0044 0.010 0.014 0.041 dobj base 0.081 0.088 0.18 0.22 ours 0.0081 0.014 0.036 0.054 sitions, determiners, and morphology (e.g. verbs with a singular subject were suffixed with the letter "s"). we set the parameters of our pitman-yor processes by specifying discount and strength parameters so that our produced sentences closely matched the type-token ratio and singleton proportion curves of the english side of the wmt 2014 german-english parallel corpus @xcite for the following dependency bigrams: adjective-noun (amod), verb-subject (nsubj), verbobject (dobj). figure 8 compares the singleton proportion curves of verb-object dependencies for our generated corpus, versus the development corpus (wmt 2014 ger-eng) and a held-out test corpus: the english side of the jparacrawl 3.0 jpn-eng corpus @xcite . we also compare our corpus statistics to a baseline that attempts to replicate (white and cotterell, 2021), using independent adjective, noun, and verb distributions rather than tied hierarchical pitman-yor distributions. visual inspection shows that the independent baseline is an outlier, unrepresentative of the statistics manifested by natural corpora. we can distill these curves into a single numeric indicator by averaging the absolute difference between an artificial corpus curve (ours or base) and a natural corpus curve the transformer produces lower-perplexity language models for the artificial languages that follow a japanese word order, while the lstm produces lowerperplexity language models for the artificial languages that follow an english word order. we created two variants of the wrig, corresponding to the standard word orders of english and japanese. for instance, as a head-final language, the japanese wrig included the rule 6 : following (white and cotterell, 2021), the wrigs also differed in: we generated 1,000,000 sentences for each wrig variant, and divided these into ten evenly sized corpora. each corpus of 100,000 sentences was further 6 a brief guide to the referenced indexed nonterminals of the wrig: vp[y1.y2] produces a verb phrase with subject count y2, whose head is the y1 th verb of the vocabulary. np[y1, y2] produces a noun phrase with count y2, whose head is the y1 th noun of the vocabulary. vb[y1, y2] produces the y1 th verb of the vocabulary, conjugated for subject count y2. for each architecture (transformer and lstm) and word order (english and japanese), figure 10 visualizes the test perplexity over the ten trials using a point plot foot_6 . for transformer lms, we obtained lower perplexity on the languages that followed a japanese word order. for lstm lms, we observed the opposite: a (statistically significant) lower perplexity on the languages that followed an english word order. while these results generally support the findings of white and @xcite , @xcite did not find significant differences between the lstm lms. we find it encouraging that our results do not differ wildly from white and @xcite (it would be troubling for the prospects of artificial languages if each iterative improvement dramatically reversed the conclusions of the previous iteration). at the same time, we also find it encouraging that the differences between their results and ours offer a possible reconciliation between white and cotterell (2021) and @xcite , who reported, based on experiments with naturally-derived corpora, that lstm lms performed better on svo (versus sov) languages. ## conclusion with this work, our goal is to enable researchers to more easily develop models for typologically diverse languages, and to investigate under what conditions such models perform effectively. by demonstrating that rigs (weighted by hierarchical pitman-yor processes) can model realistic syntactic and semantic dependencies, we hope to provide some confidence that the framework can prove a useful proxy for real-world data, when such data is not readily available. to facilitate adoption of our framework, we are also releasing an open-source python package called testperanto for building wrigs, providing fellow researchers with a means to generate artificial languages that emulate the typology of the natural languages they seek to study.