Priorless Recurrent Networks Learn Curiously (2020)

Abstract: Recently, domain-general recurrent neural networks, without explicit linguistic inductive biases, have been shown to successfully reproduce a range of human language behaviours, such as accurately predicting number agreement between nouns and verbs. We show that such networks will also learn number agreement within unnatural sentence structures, i.e. structures that are not found within any natural languages and which humans struggle to process. These results suggest that the models are learning from their input in a manner that is substantially different from human language acquisition, and we undertake an analysis of how the learned knowledge is stored in the weights of the network. We find that while the model has an effective understanding of singular versus plural for individual sentences, there is a lack of a unified concept of number agreement connecting these processes across the full range of inputs. Moreover, the weights handling natural and unnatural structures overlap substantially, in a way that underlines the non-human-like nature of the knowledge learned by the network.

## introduction grammarians have for a long time (e.g.: pān . ini, 6th century bce; aristotle, 350 bce) been interested in characterising what is and is not possible within a given specific language. famously, @xcite used the sentence colorless green ideas sleep furiously and its reversal furiously sleep ideas green colorless to illustrate the difference between grammatical and ungrammatical within english. in addition, modern linguistics has also been concerned more generally with what is and is not possible within the space of all human languages. in this article, we investigate the ability of a neural language model to learn structures on the impossible side of this boundary. that is, we train and test the network on data which has been manipulated to contain unnatural structures. in contrast, the evaluation of such models has usually focused on the other side of the boundary, i.e. on data gathered from real human behaviour within specific natural languages. here, we are interested in the more general question of whether these neural architectures are appropriate models of human language capacities, in terms of what differentiates possible from impossible structures. from a theoretical perspective, a number of grammatical formalisms have been proposed as computational models of the space of possible natural languages: from context-free grammars @xcite to tree-adjoining grammars @xcite and combinatory categorial grammars @xcite . one characteristic that all such grammars share is the organisation of linguistic expressions into hierarchical, as opposed to purely linear, structures. empirical research has also uncovered further design patterns common across many superficially diverse languages. although the concept of language universals is not uncontroversial, most linguist would agree that, at the very least, natural languages display strong shared tendencies and similarities. for example, languages typically have a preferred word order -e.g. subject verb object vs. subject object verb -which is correlated with the ordering of other constructions -e.g. prepositions vs. postpositions. alongside the observation that children learn their native languages efficiently from limited, noisy input, these shared principles have been been used to support the view that the human language faculties are innately adapted to learn and process one specific set of possible languages. or, to put it another way, there are structures that are essentially impossible for these innate faculties to handle. recently, however, domain-general neural sequence models, lacking explicit linguistic inductive biases, have demonstrated impressive performance on a range of practical language processing applications. this, in turn, has led to increasing interest in the question of how effective these architectures are as models of general human linguistic behaviour. whereas our interest here is in their behaviour when trained on unnatural structures, most research has so far been interested in their ability to successfully learn the structure of natural languages. for example, @xcite examined the extent to which a long short term memory @xcite ) network trained on raw text learned about agreement phenomena across a number of languages. they found that, given the right training data, the model was able to predict long-distance agreement on both natural and semantically nonsensical but syntactically valid sentences. similarly, @xcite investigated the ability of such models to track syntactic state within subordinations and garden path sentences. treating their networks as psycholinguistic subjects, they found that given enough data the models responded to subtle cues in a manner comparable to human subjects. these results suggest that although an lstm handles its input in a purely sequential manner, it is able, nonetheless, to learn about phenomena that have traditionally been represented within hierarchical data structures. frank and colleagues @xcite @xcite suggest that, in fact, such sequential models have advantages over hierarchical alternatives as cognitive models of language processing. using evidence from reading time and erp studies, they argue that the ability to accurately model the probabilities of words in context is more important in a psycholinguistic model than the particular architecture employed. given these successes in language learning, @xcite has proposed a complementary role for neural networks alongside generative linguistics. he argues that these architectures may supply the theory of learning that linguistics currently lacks. in response, @xcite invoke the no-free-lunch theorems @xcite and poverty-of-the-stimulus arguments @xcite to question whether neural models actually have the right inductive biases. here, we investigate this question by evaluating a network's ability to learn number agreement within pathological structures unlike any natural language. while failure to learn natural structures would give important insights into a model's limitations, success on unnatural structures is also revealing, suggesting a mismatch between the model's architecture and human linguistic processes. we therefore train and test an lstm on manipulated english sentences and compare this performance to that on the original unmanipulated corpus. based on prior research using unnatural data, we employ three types of modification: reversing word order, inserting a linear sequence of tokens, and shuffling the vocabulary. introducing reversed constructions alongside the original sequences disrupts a language's preference for a consistent word order and so is an effective way of producing unnatural data. moreover, the human experiments described in section 2 suggest that learners struggle to acquire artificial languages constructed in this way. learners also struggle with constructions based on sequential counting, e.g. inserting a word in the nth position regardless of the syntactic context. human languages universally base their syntax around hierarchical structures, and so constructions based on counting and linear order are entirely unnatural. while word order preferences and hierarchical structure are language specific characteristics differentiating natural from unnatural data, machine learning theorists have also proposed domain general methods for generating unnatural data. in this regard, shuffling provides a useful means for for analysing the limits of learning algorithms. in particular, fully randomised shuffling of the training labels in a supervised learning task forces an algorithm to resort to memorisation of the training data, supplying insight into its capacity. in our experiments, we employ a limited form of shuffling, where generalisation is still possible, but which increases the computational load on the model substantially. we describe these experiments and their results in section 3. following that we analyse in more detail how knowledge about number agreement is encoded in the weights of the network in section 4. first, however, we describe some of the prior work on unnatural data in section 2. ## learning and unnatural data performance on unnatural data has been discussed by a number of researchers interested in understanding the characteristics of learning processes in humans and machines. such a perspective can provide insight not only into the limits of human learning, but also into the lack of such limits in machines. for example, @xcite critique a connectionist model of past tense formation in terms of its handling of unnatural structures. they argue that the neural architecture proposed by @xcite cannot be an adequate model of learning because it would permit string reversal as a means of forming the past tense, which is never attested in natural languages. @xcite , in contrast, studied the learning of unnatural linguistic structures by humans rather than machines. working with a polyglot savant, christopher, and a group of normal adult controls, they investigated second language learning of an artifical language, epun, containing both natural and unnatural grammatical structures. they found that while both christopher and the controls could master the linguistically natural aspects, only the controls could eventually handle the structure dependent unnatural phenomena, such as reversing subject and verb to express negation, and neither could master the structure independent aspects, such as marking emphasis on the third word of a sentence. they argue that christopher's abilities are entirely due to his intact linguistic faculties, but that the controls had recourse to employing other general cognitive abilities for some phenomena. this proposal, that second language learners may employ distinct cognitive resources in handling natural and unnatural structures, is supported by the fmri evidence of @xcite . in that experiment, native speakers of german were taught modified versions of italian and japanese, containing unnatural constructions alongside their real grammar. for example, in both languages, interrogatives were constructed by reversing a declarative sentence and the marker for negation was placed after the third word. while the learners mastered both the natural and unnatural phenomena, reaction times were faster for natural structures. moreover, the brain imaging revealed that activation of broca's area was associated with learning of the natural structures, whereas performance on the unnatural phenomena was negatively correlated with this activation. addressing the question of whether statistical methods can provide cognitively faithful models of human language knowledge, @xcite investigated the performance of a statistical parser @xcite when trained and tested on unnatural structures. they found that the accuracy of the parser remained largely unchanged on a corpus in which the orders of verbs and their complements and/or arguments and adjuncts had been reversed in every second sentence. given this insensitivity to the consistency of word order in its training corpus, @xcite conclude that the parser's performance points to a non-human-like ability. extending this line of enquiry, @xcite evaluated the ability of a parser to handle the impossible interrogatives studied by @xcite . they found that both the stanford unlexicalised parser @xcite ) and collin's parser @xcite were able to learn the reversed questions reasonably successfully, particularly when given enough training examples. again, this success in learning from a corpus without consistent word order preferences was taken as evidence for a failure in modelling human language knowledge. the learning of unnatural structures has also been investigated within neural image recognition. @xcite show that state-of-the-art cnns will easily fit to a random labeling of the training data, even when the true images are replaced with unstructured noise. these experiments were inspired by theoretical connections between generalisation ability and rademacher complexity @xcite , which measures the capacity of a model in terms of its ability to fit randomly switched labels. further investigations @xcite suggest that satisfactory in-domain generalisation can frequently be achieved by pushing the number of parameters well beyond the point at which such rote memorisation becomes possible, and allowing the network to smoothly interpolate between training examples. this does raise questions, however, about the performance of such an interpolation strategy on out-of-domain generalisation in comparison to human judgements, particularly on adversarial and fooling images @xcite . ## number agreement in unnatural language structures we apply the lstm architecture of @xcite to modified versions of their original datasets. based on the prior work described in section 2, three types of modification are used: reversing the order of tokens, inserting a linear sequence of tokens and shuffling the vocabulary. in each case, we train and test the lstm on the modified data, evaluating the ability of the model to handle number agreement within structures that deviate substantially from natural constructions. in the first modification, we choose a random point within the sentence, insert a special marker, and reverse the remainder of the sentence beyond that point. so, for example, the man whose dog barks likes apples becomes the man whose dog marker apples likes barks. reversal has been a common manipulation in the generation of unnatural structures @xcite @xcite because it disrupts the tendency for languages to have a preferred word order. our transformation introduces both a strong contextual influence on word order and also highly tangled dependencies involving constituents that cross the boundary. thus, linear position within the surface form of the sentence in relation to the marker determines, for example, whether adjectives follow nouns or precede them, whether prepositions or postpositions are used, or whether an svo or ovs structure is followed. moreover, constituents that cross the boundary, e.g. whose dog barks, become discontinuous and their dependencies become entangled with those of the intervening material. we evaluate the ability of the model to handle dependencies that cross this boundary, by placing the marker immediately before the target verb at test time. in the second transformation, we again choose a random point in the sentence and insert a marker. in this case however, a random vocabulary item is then selected and repeated 23 times before the sentence resumes. such counting-based linear structures have also been used in prior research @xcite as a contrast to the hierarchical structures usually found in natural languages. the repetition of the same token introduces a construction without any of the internal structure, such as dependencies between heads, modifiers and complements, that characterises the syntactic trees found in natural data, and is instead just a linear sequence of the appropriate length. moreover, at test time we use the incorrect form of the verb as the inserted item, placing it just before the correct form. thus, the dog barks becomes the dog marker bark bark . . . bark barks. correctly predicting number agreement between dog and barks requires counting the intervening bark tokens and accurately anticipating where the sentence will resume. the third transformation leaves the syntactic structure of the sentence unchanged, and instead permutes the vocabulary. this is inspired by the shuffling of image labels used by @xcite . we first take a list of the whole lexicon, randomly shuffle it, and then use this to define a mapping from each vocabulary item to another. a sentence in the training corpus is then transformed by selecting a random point, placing a marker, and replacing all the tokens in the remainder of the sentence with their shuffled counterparts. thus, the man whose dog barks likes apples becomes the man whose dog marker and copy simpson. at test time, we evaluate the ability of the model to predict number agreement across this boundary by placing the marker immediately before the target verb. agreement is then measured in terms of the model's ability to predict the shuffled vocabulary item corresponding to the correct form of the original verb. our experiments are based on the model and datasets of @xcite . we retrain their language model using the original hyperparameters, after applying the transformations described above to their training data, consisting of ∼90m words from english wikipedia, which they filtered to exclude sentences containing more than 5% unknown words outside a 50k vocabulary. the model was a two layer lstm, with 650 units in the input embeddings and each recurrent layer, trained using the standard language modelling objective of predicting the next word. after 40 epochs of training, we applied this model to their number agreement test corpus, and evaluated the model's ability to predict correctly whether a singular or plural verb is appropriate in each sentence. in particular, the lstm is given the initial words preceding the verb, and the logits corresponding to the singular and plural forms for the next word prediction are compared. the model's outputs are evaluated as correct whenever the form attested in the sentence is given the higher value, and we report percentage accuracy averaged over 5 training runs. @xcite extracted these evaluation sentences from the english universal dependency treebank @xcite , identifying constructions in which a verb target, morphologically marked for number, is preceded by its noun subject cue, which also agrees in number. ## analysis of internal model structure in this article, we are instead interested in the relation between natural and unnatural sentences, and the way that the model learns their structures. as a consequence, we take a slightly different approach here, based on comparing which weights are important to predicting number agreement correctly on pairs of sentences. the weights, unlike the activities, remain fixed throughout the processing of a sentence, and are also the locus of learning and generalisation. thus, this approach allows us to construct a single representation for a given number agreement prediction that summarises which parts of the network were critical in processing the entire preceding sentence context. across the majority of neural architectures, backpropagation is the standard technique of credit assignment for deciding which weights are contributing most to the current objective or loss, and as a consequence how they need to be changed. in our case, backpropagation through time allows us to compute the gradients in weight space of any objective of interest, defined in terms of the lstm's outputs, given a sentence prefix input. such gradients essentially tell us which weights were important in producing that value, in the sense of changes in more important weights producing larger changes in the objective. inputs with highly similar gradient vectors therefore share a common set of important weights, or, in other words, are the product of overlapping parts of the architecture. in contrast, two inputs with orthogonal gradient vectors rely on disparate sets of important weights and are processed in distinct parts of the network. furthermore, there is an obvious connection between these weight gradient representations and the processes of learning and generalisation. the gradient vector tells us how the objective would change in response to changes in the weights and also the direction in which the weights would be updated if a gradient descent step is taken. thus, two data points with orthogonal gradient vectors will tend not to generalise to each other: the updates made in training on one point will not affect the other, as they involve distinct sets of weights. equally, data points with highly aligned gradient vectors will generalise to each other, due to the shared weights. with these considerations in mind we define our weight-based representations as follows. given that number agreement is the phenomena we are interested in, we take the objective to be the logit for the singular form of the verb minus the logit for the plural form of the verb. since our evaluation of models is based on comparing which of these logits is bigger, their difference is a natural, and differentiable, objective to focus on. furthermore, were we to train on the number agreement task, this or a similar objective would be a reasonable choice. in particular, if f (v, p) represents the function computed by the network in producing the logit associated with a verb, v, that follows a sentence prefix, p, then the value we are interested, given the sentence the dog barks, is the difference, d, between the logits for barks and bark. @xmath0, the dog) -f (bark, the dog) we then take the gradient, g, of this value with respect to the weights, θ, in both layers of the lstm, but not the input embeddings or fully connected output layer. these latter parameters are too lexically specific, relating to the particular words in the sentence, while the former make up the general structure processing machinery of the network. given two number agreement instances with gradient vectors g 1 and g 2 , a small gradient directed update based on the first instance produces a change in the weights of g 1 which results in a first order change in singular-plural difference on the second instance of ( g 1 • g 2 ). thus, the dot product of these vectors is closely related to how learning generalises from one instance to another. however, here we use the cosine as a measure of similarity between the gradients, because it normalises the dot product in terms of the lengths of the vectors, putting all values on a comparable scale, which relates just to the angular alignment of the representations. fort et al. (2020) call such a measure stiffness, as it relates to how rigidly changes in the behaviour at one data point are transmitted to another. he and su (2020) use the term local elasticity for a related measure, emphasising instead the flexibility of a model in which the behaviour on a pair of instances is largely independent. this approach produces a local measure of the which weights are most important in producing a particular number agreement prediction, in the context of small perturbations around the specific values assigned to the weights in the trained model. in contrast, @xcite take a more global viewpoint by ablating whole units, and their weights, which can be thought of as comparing performance with the learned weight values to performance when they are set to zero, rather than infinitesimally perturbed. although this latter approach has the benefit of simplicity, it is a blunter measure and less directly related to learning and generalisation. nonetheless, the two approaches are related, in the sense that the effect of setting a weight to zero can be expressed in terms of an integral of the gradient along a path that takes the weight to zero. here, we use this gradient based similarity measure to compare our model's processing of number agreement across pairs of evaluation sentences. roughly, positive similarity indicates a pair of sentences that utilise overlapping structures in the model, and across which generalisation is possible. zero similarity corresponds to orthogonal vectors, indicating a pair without such commonalities, and which are therefore free to behave independently. negative values are also possible when the shared components end up pushing the pair of outputs in opposite directions, i.e. increasing the output for one data point tends to decrease the output for the other. in our number agreement situation, this would mean something like increasing the probability of a singular verb in one sentence leads to a plural verb becoming more probable in the other. in the next section, we use this approach to compare pairs of number agreement predictions made by the model to understand when these predictions rely on shared architectural resources and when they do 0.0 0.2 0.4 0.6 0.8 (a) similarities between different target verbs on the same prefix without reversal. q q q 0.3 0.4 0.5 0.6 0.7 0.8 0.9 (c) similarities on the same target verb between modified and unmodified versions of the same sentences. not. we begin by examining the original model, comparing both instances that share a common prefix and also instances that are wholly distinct. we then investigate a model trained on modified data, first replicating the results from the original model and finally analysing the relation between natural and unnatural structures. ## conclusions the lstm architecture of @xcite demonstrated its capacity in our experiments to handle structures that go beyond those seen in natural languages. seen as a test of broad coverage sequence learning, these results are evidence of its flexibility and generality. however, seen from the perspective of modelling the shared design principles of human languages, these characteristics begin to look like weaknesses. they suggest that the network lacks the necessary inductive biases to distinguish between natural and unnatural structures. many linguists interpret these shared principles in terms of innate cognitive abilities @xcite , with additional evidence coming from human experiments with artificial languages, such as those described in section 2. furthermore, the behaviour of the neural models on out-of-domain examples also suggests stronger innate biases or constraints may be useful. although neural nlp systems often achieve impressive in-domain performance, there is still a problem with the robustness of these abilities @xcite @xcite . making the internal mechanisms of our architectures more human-like is a plausible direction from which to address these problems. moreover, improving the specificity of these models for natural languages, as opposed to arbitrary sequential structures, should reduce the problem of requiring huge datasets to train huge numbers of weights. our analysis of how knowledge is encoded in the weights of the model also underscores the nonhuman-like form of learning embodied in this architecture. the network appears to lack a shared set of weights that could instantiate a unifying concept of number agreement to tie together the disparate predictions of singular versus plural it makes for particular constructions. similarly, @xcite found that long and short range dependencies were handled by independent units. at the same time, the model also appears to process natural and unnatural structures using common resources. future work will examine how to remedy these shortcomings, by examining how inductive biases change the way in which information is encoded in the weights during learning. for example, mitchell and bowers (2020) explore how a stack-like structure can be imposed on the memory cells of an lstm that shares weights between long and short range dependencies, obtaining improved generalisation on a context free language learning task.