Emergence of Syntax Needs Minimal Supervision (2020)

Abstract: This paper is a theoretical contribution to the debate on the learnability of syntax from a corpus without explicit syntax-specific guidance. Our approach originates in the observable structure of a corpus, which we use to define and isolate grammaticality (syntactic information) and meaning/pragmatics information. We describe the formal characteristics of an autonomous syntax and show that it becomes possible to search for syntax-based lexical categories with a simple optimization process, without any prior hypothesis on the form of the model.

## introduction syntax is the essence of human linguistic capacity that makes it possible to produce and understand a potentially infinite number of unheard sentences. the principle of compositionality @xcite states that the meaning of a complex expression is fully determined by the meanings of its constituents and its structure; hence, our understanding of sentences we have never heard before comes from the ability to construct the sense of a sentence out of its parts. the number of constituents and assigned meanings is necessarily finite. syntax is responsible for creatively combining them, and it is commonly assumed that syntax operates by means of algebraic compositional rules @xcite and a finite number of syntactic categories. one would also expect a computational model of language to have -or be able to acquire -this compositional capacity. the recent success of neural network based language models on several nlp tasks, together with their "black box" nature, attracted attention to at least two questions. first, when recurrent neural language models generalize to unseen data, does it imply that they acquire syntactic knowledge, and if so, does it translate into human-like compositional capacities @xcite @xcite ? second, whether research into neural networks and linguistics can benefit each other @xcite ; by providing evidence that syntax can be learnt in an unsupervised fashion @xcite , or the opposite, humans and machines alike need innate constraints on the hypothesis space (a universal grammar) @xcite ? a closely related question is whether it is possible to learn a language's syntax exclusively from a corpus. the poverty of stimulus argument @xcite suggests that humans cannot acquire their target language from only positive evidence unless some of their linguistic knowledge is innate. the machine learning equivalent of this categorical "no" is a formulation known as gold's theorem @xcite , which suggests that the complete unsupervised learning of a language (correct grammaticality judgments for every sequence), is intractable from only positive data. @xcite argue that gold's paradigm does not resemble a child's learning situation and there exist algorithms that can learn unconstrained classes of infinite languages @xcite . this ongoing debate on syntax learnability and the poverty of the stimulus can benefit from empirical and theoretical machine learning contributions @xcite @xcite . in this paper, we argue that syntax can be inferred from a sample of natural language with very minimal supervision. we introduce an information theoretical definition of what constitutes syntactic information. the linguistic basis of our approach is the autonomy of syntax, which we redefine in terms of (statistical) independence. we demonstrate that it is possible to establish a syntax-based lexical classification of words from a corpus without a prior hypothesis on the form of a syntactic model. our work is loosely related to previous attempts at optimizing language models for syntactic performance @xcite and more particularly to @xcite because of their use of mutual information and the information bottleneck principle @xcite . however, our goal is different in that we demonstrate that very minimal supervision is sufficient in order to guide a symbolic or statistical learner towards grammatical competence. ## language models and syntax as recurrent neural network based language models started to achieve good performance on different tasks @xcite , this success sparked attention on whether such models implicitly learn syntactic information. language models are typically evaluated using perplexity on test data that is similar to the training examples. however, lower perplexity does not necessarily imply better syntactic generalization. therefore, new tests have been put forward to evaluate the linguistically meaningful knowledge acquired by lms. a number of tests based on artificial data have been used to detect compositionality or systematicity in deep neural networks. lake and @xcite created a task set that requires executing commands expressed in a compositional language. @xcite design a task of logical entailment relations to be solved by discovering a recursive compositional structure. @xcite propose a semi-artificial probing task of mathematics problems. @xcite initiated a different line of linguistically motivated evaluation of rnns. their data set consists in minimal pairs that differ in grammaticality and instantiate sentences with long distance dependencies (e.g. number agreement). the model is supposed to give a higher probability to the grammatical sentence. the test aims to detect whether the model can solve the task even when this requires knowledge of a hierarchical structure. subsequently, several alternative tasks were created along the same concept to overcome specific shortcomings @xcite , or to extend the scope to different languages or phenomena @xcite . it was also suggested that the information content of a network can be tested using "probing tasks" or "diagnostic classifiers" @xcite . this approach consists in extracting a representation from a nn and using it as input for a supervised classifier to solve a different linguistic task. accordingly, probes were conceived to test if the model learned parts of speech @xcite , morphology @xcite , or syntactic information. @xcite evaluate contextualized word representations on syntactic and semantic sequence labeling tasks. syntactic knowledge can be tested by extracting constituency trees from a network's hidden states @xcite or from its word representations @xcite . other syntactic probe sets include the work of @xcite and @xcite . despite the vivid interest for the topic, no consensus seems to unfold from the experimental results. two competing opinions emerge: deep neural language models generalize by learning human-like syntax: given sufficient amount of training data, rnn models approximate human compositional skills and implicitly encode hierarchical structure at some level of the network. this conjecture coincides with the findings of, among others bowman et al. (2015); linzen et al. (2016); giulianelli et al. (2018); gulordava et al. (2018); adhiguna et al. (2018). moreover, some shortcomings of the above probing methods make it more difficult to come to a conclusion. namely, it is not trivial to come up with minimal pairs of naturally occurring sentences that are equally likely. furthermore, assigning a (slightly) higher probability to one sentence does not reflect the nature of knowledge behind a grammaticality judgment. diagnostic classifiers may do well on a linguistic task because they learn to solve it, not because their input contains a hierarchical structure @xcite . in what follows, we present our assessment on how the difficulty of creating a linguistic probing data set is interconnected with the theoretical problem of learning a model of syntactic competence. ## syntactic and contextual categories in a corpus as we have seen in section 2, probabilistic language modeling with a likelihood maximization objective does not have incentive to concentrate on syntactic generalizations. in what follows, we demonstrate that using the autonomy of syntax principle it is possible to infer syntactic categories for a probabilistic language. a stochastic language l is a language which assigns a probability to each sequence. as an illustration of such a language, we consider the empirical distribution induced from the sample in section 3. we will denote by p l (v i 1 . . . v in ) the probability distribution associated to l. definition 6. let v be a vocabulary. a (probabilistic) partition of v is defined by @xmath0, c, π : @xmath1. example. the following probabilistic partitions correspond to the non-probabilistic partitions (contextual and syntactic, respectively) defined in section 3. we will now consider these partitions in the context of the probabilistic language l. from a probabilistic partition @xmath2, c, π) as defined above, one can map a stochastic language l to a stochastic language π(l) over the sequences of categories: as in the non-probabilistic case, the language l will be defined as the language obtained by shuffling the sequences in l. definition 7. let l be a stochastic language over vocabulary v . we will denote by l the language obtained by shuffling the elements in the sequences of l in the following way: for a sequence v 1 . . . v n , one has one can easily check that π(l) = π(l). example. the stochastic patterns of l over the two partitions are, respectively: we can now define a probabilistic contextual partition: definition 8. let l be a stochastic language over vocabulary v , and let @xmath3, c, π) be a probabilistic partition. p will be considered as contextual if π(l) = π(l) we now want to express the independence of syntactic partitions from contextual partitions. the independence of two probabilistic partitions can be construed as an independence between two random variables: definition 9. consider two probabilistic partitions @xmath4, c, π) and @xmath5, c , π ). we will use the notation and the notation p • @xmath6, c × c , π • π ) p and p are said to be independent (with respect to l) if the distributions inferred over sequences of their categories are independent: a syntactic partition will be defined by its independence from contextual information: definition 10. let p be a probabilistic partition, and l a stochastic language. the partition p is said to be syntactic if it is independent (with respect to l) of any possible probabilistic contextual partition in l. example. the partition p 1 is contextual, as π 1 (l) = π 1 (l). the partition p 2 is clearly independent of p 1 w.r.t. l. ## conclusion and future work in this paper, we proposed a theoretical reformulation for the problem of learning syntactic information from a corpus. current language models have difficulty acquiring syntactically relevant generalizations for diverse reasons. on the one hand, we observe a natural tendency to lean towards shallow contextual generalizations, likely due to the maximum likelihood training objective. on the other hand, a corpus is not representative of human linguistic competence but of performance. it is however possible for linguistic competence -syntax -to emerge from data if we prompt models to establish a distinction between syntactic and contextual (semantic/pragmatic) information. two orientations can be identified for future work. the immediate one is experimentation. the current formulation of our syntax learning scheme needs adjustments in order to be applicable to real natural language corpora. at present, we are working on an incremental construction of the space of categories. the second direction is towards extending the approach to morphologically rich languages. in that case, two types of surface realization need to be considered: word order and morphological markers. an agglutinating morphology probably allows a more straightforward application of the method, by treating affixes as individual elements of the vocabulary. the adaptation to other types of morphological markers will necessitate more elaborate linguistic reflection.