Recurrent neural networks ( RNNs ) have achieved impressive results in a variety of linguistic processing tasks , suggesting that they can induce non-trivial properties of language . We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure . We test whether RNNs trained with a generic language modeling objective in four languages ( Italian , English , Hebrew , Russian ) can predict long-distance number agreement in various constructions . We include in our evaluation nonsensical sentences where RNNs can not rely on semantic or lexical cues ( " The colorless green ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas ideas I ate with the chair sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep sleep furiously " ) , and , for Italian , we compare model performance to human intuitions . Our language-model-trained RNNs make reliable predictions about long-distance agreement , and do not lag much behind human performance . We thus bring support to the hypothesis that RNNs are not just shallowpattern extractors , but they also acquire deeper grammatical competence . Recurrent neural networks ( RNNs ; Elman , 1990 ) are general sequence processing devices that do not explicitly encode the hierarchical structure that is thought to be essential to natural language ( Everaert et al . , 2015 ) . Early work using artificial languages showed that they may nevertheless be able to approximate context-free languages ( Elman , 1991 ) . More recently , RNNs have achieved impressive results in large-scale tasks such as language modeling for speech recognition and machine translation , and are by now standard tools for sequential natural language tasks ( e.g. , Mikolov et al . , 2010 ; Graves , 2012 ; Wu et al . , 2016 ) . This suggests that RNNs may learn to track grammatical structure even when trained on noisier natural data . The conjecture is supported by the success of RNNs as feature extractors for syntactic parsing ( e.g. , Cross and Huang , 2016 ; Kiperwasser and Goldberg , 2016 ; Zhang et al . , 2017 ) . Linzen et al . ( 2016 ) directly evaluated the extent to which RNNs can approximate hierarchical structure in corpus-extracted natural language data . They tested whether RNNs can learn to predict English subject-verb agreement , a task thought to require hierarchical structure in the general case ( " the girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl the boys like . Their experiments confirmed that RNNs can , in principle , handle such constructions . However , in their study RNNs could only succeed when provided with explicit supervision on the target task . Linzen and colleagues argued that the unsupervised language modeling objective is not sufficient for RNNs to induce the syntactic knowledge necessary to cope with long-distance agreement . . . is is is is is is is is is is is is is is is is is The current paper reevaluates these conclusions . We strengthen the evaluation paradigm of Linzen and colleagues in several ways . Most importantly , their analysis did not rule out the possibility that RNNs might be relying on semantic or collocational / frequency-based information , rather than purely on syntactic structure . In " dogs dogs dogs dogs dogs about what typically barks ( dogs , not neighbourhoods ) , without relying on more abstract structural cues . In a follow-up study to Linzen and colleagues ' , Bernardy and Lappin ( 2017 ) observed that RNNs are better at long-distance agreement when they construct rich lexical representations of words , which suggests effects of this sort might indeed be at play . We introduce a method to probe the syntactic abilities of RNNs that abstracts away from potential lexical , semantic and frequency-based confounds . Inspired by Chomsky 's ( 1957 ) We extend the previous work in three additional ways . First , alongside English , which has few morphological cues to agreement , we examine Italian , Hebrew and Russian , which have richer morphological systems . Second , we go beyond subject-verb agreement and develop an automated method to harvest a variety of long-distance number agreement constructions from treebanks . Finally , for Italian , we collect human judgments for the tested sentences , providing an important comparison point for RNN performance . 1We focus on the more interesting unsupervised setup , where RNNs are trained to perform generic , large-scale language modeling ( LM ): they are not given explicit evidence , at training time , that they must focus on long-distance agreement , but they are rather required to track a multitude of cues that might help with word prediction in general . Our results are encouraging . RNNs trained with a LM objective solve the long-distance agreement problem well , even on nonce sentences . The pattern is consistent across languages , and , crucially , not far from human performance in Italian . Moreover , RNN performance on language modeling ( measured in terms of perplexity ) is a good predictor of long-distance agreement accuracy . This suggests that the ability to capture structural generalizations is an important aspect of what makes the best RNN architectures so good at language modeling . Since our positive results contradict , to some extent , those of Linzen et al . ( 2016 ) , we also replicate their relevant experiment using our best RNN ( an LSTM ) . We outperform their models , suggesting that a careful architecture / hyperparameter search is crucial to obtain RNNs that are not only good at language modeling , but able to extract syntactic generalizations . 2 Constructing a long-distance agreement benchmark Overview . We construct our number agreement test sets as follows . Original sentences are automatically extracted from a dependency treebank . They are then converted into nonce sentences by substituting all content words with random words with the same morphology , resulting in grammatical but nonsensical sequences . An LM is evaluated on its predictions for the target ( second ) word in the dependency , in both the original and nonce sentences . Long-distance agreement constructions . Agreement relations , such as subject-verb agreement in English , are an ideal test bed for the syntactic abilities of LMs , because the form of the second item ( the target ) is predictable from the first item ( the cue ) . Crucially , the cue and the target are linked by a structural relation , where linear order in the word sequence does not matter ( Everaert et al . , 2015 ) In all these cases , the number of the main verb " thinks " is determined by its subject ( " girl " ) , and this relation depends on the syntactic structure of the sentence , not on the linear sequence of words . As the last sentence shows , the word directly preceding the verb can even be a noun with the opposite number ( " friends " ) , but this does not influence the structurally-determined form of the verb . When the cue and the target are adjacent ( " the girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl girl thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks thinks . . . " ) , an LM can predict the target without access to syntactic structure : it can simply extract the relevant morphosyntactic features of words ( e.g. , number ) and record the co-occurrence frequencies of patterns such as N P lur V P lur ( Mikolov et al . , 2013 ) . Thus , we focus here on long-distance agreement , where an arbitrary num- ber of words can occur between the elements of the agreement relation . We limit ourselves to number agreement ( plural or singular ) , as it is the only overt agreement feature shared by all of the languages we study . Identifying candidate constructions . We started by collecting pairs of part-of-speech ( POS ) tags connected by a dependency arc . Independently of which element is the head of the relation , we refer to the first item as the cue and to the second as the target . We additionally refer to the POS sequence characterizing the entire pattern as a construction , and to the elements in the middle as context . For each candidate construction , we collected all of the contexts in the corpus that intervene between the cue and the target ( we define contexts as the sequence of POS tags of the top-level nodes in the dependency subtrees ) . For example , for the English subject-verb agreement construction shown in Fig . 1a , the context is defined by VERB ( head of the relative clause ) and ADV ( adverbial modifier of the target verb ) , which together dominate the sequence " the boys like often " . For the Russian adjective-noun agreement construction in Fig . 1b , the context is NOUN , because in the dependency grammar we use the noun " moment " is the head of the prepositional phrase " at that moment " , which modifies the adjective " deep " . The candidate agreement pair and the context form a construction , which is characterized by a sequence of POS tags , e.g. , NOUN VERB ADV VERB or VERB NOUN CCONJ VERB ( Fig . 1c ) . is is is is is is is is is is is is is is is is is " and " girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls girls who stayed at home were were were were were were were were were were were were were were were were were " . Conversely , standard syntactic structures might be split between different constructions , e.g. , relative clause contexts occur in both NOUN VERB VERB and NOUN VERB ADV VERB constructions ( the latter is illustrated by the English example in Fig . 1a ) . Construction contexts can contain a variable numbers of words . Since we are interested in challenging cases , we only considered cases in which at least three tokens intervened between the cue and the target . Excluding non-agreement constructions . In the next step , we excluded constructions in which the candidate cue and target did not agree in number in all of the instances of the construction in the treebank ( if both the cue and the target were morphologically annotated for number ) . This step retained English subject-verb constructions , for example , but excluded verb-object constructions , since any form of a verb can appear both with singular and plural objects . To focus on robust agreement patterns , we only kept constructions with at least 10 instances of both plural and singular agreement . When applied to the treebanks we used ( see Section 3 ) , this step resulted in between two ( English ) and 21 ( Russian ) constructions per lan-guage . English has the poorest morphology and consequently the lowest number of patterns with identifiable morphological agreement . Only the VP-conjunction construction ( Fig . 1c ) was identified in all four languages . Subject-verb agreement constructions were extracted in all languages but Russian ; Russian has relatively flexible word order and a noun dependent preceding a head verb is not necessarily its subject . The full list of extracted constructions in English and Italian is given in Tables 2 and 3 , respectively . For the other languages , see the Supplementary Material ( SM ) . 2 Original sentence test set . Our " original " sentence test set included all sentences from each construction where all words from the cue and up to and including the target occurred in the LM vocabulary ( Section 3 ) , and where the singular / plural counterpart of the target occurred in the treebank and in the language model vocabulary ( this is required by the evaluation procedure outlined below ) . The total counts of constructions and original sentences in our test sets are provided in Table 1 . The average number of context words separating the cue and the target ranged from 3.6 ( Hebrew ) to 4.5 ( Italian ) . Generating nonce sentences . We generated nine nonce variants of each original sentence as follows . Each content word ( noun , verb , adjective , proper noun , numeral , adverb ) in the sentence was substituted by another random content word from the treebank with matching POS and morphological features . To avoid forms that are ambiguous between several POS , which are particularly frequent in English ( e.g. , plural noun and singular verb forms ) , we excluded the forms that appeared with a different POS more than 10 % of the time in the treebank . Function words ( determiners , pronouns , adpositions , particles ) and punctuation were left intact . For example , we generated the nonce ( 1b ) from the original sentence ( 1a ): ( ( e.g. , " it stays the shuttle " in ( 1b ) ) . Evaluation procedure . For each sentence in our test set , we retrieved from our treebank the form that is identical to the agreement target in all morphological features except number ( e.g. , " finds " instead of " find " in ( 1b ) ) . Given a sentence with prefix p up to and excluding the target , we then compute the probabilities P ( t 1 |p ) and P ( t 2 |p ) for the singular and plural variants of the target , t 1 and t 2 , based on the language model . Following Linzen et al . ( 2016 ) , we say that the model identified the correct target if it assigned a higher probability to the form with the correct number . In ( 1b ) , for example , the model should assign a higher probability to " finds " than " find".3 3 Experimental setup Treebanks . We extracted our test sets from the Italian , English , Hebrew and Russian Universal Dependency treebanks ( UD , v2.0 , Nivre et al . , 2016 ) . The English and Hebrew treebanks were post-processed to obtain a richer morphological annotation at the word level ( see SM for details ) . We ran an extensive analysis of the abilities of RNNs trained on a generic language-modeling task to predict long-distance number agreement . Results were consistent across four languages and a number of constructions . They were above strong baselines even in the challenging case of nonsense sentences , and not far from human performance . We are not aware of other collections of human long-distance agreement judgments on nonsensical sentences , and we thus consider our publicly available data set an important contribution of our work , of interest to students of human language processing in general . The constructions we considered are quite infrequent ( according to a rough estimate based on the treebanks , the language in which they are most common is Hebrew , and even there they occur with average 0.8 % sentence frequency ) . Moreover , they vary in the contexts that separate the cue and the target . So , RNNs are not simply memorizing frequent morphosyntactic sequences ( which would already be impressive , for systems learning from raw text ) . We tentatively conclude that LMtrained RNNs can construct abstract grammatical representations of their input . This , in turn , suggests that the input itself contains enough information to trigger some form of syntactic learning in a system , such as an RNN , that does not contain an explicit prior bias in favour of syntactic structures . In future work , we would like to better understand what kind of syntactic information RNNs are encoding , and how . On the one hand , we plan to adapt methods to inspect information flow across RNN states ( e.g. , Hupkes et al . , 2017 ) . On the other , we would like to expand our empirical investigation by focusing on other long-distance phenomena , such as overt case assignment ( Blake , 2001 ) or parasitic gap licensing ( Culicover and Postal , 2001 ) . While it is more challenging to extract reliable examples of such phenomena from corpora , their study would probe more sophisticated syntactic capabilities , possibly even shedding light on the theoretical analysis of the underlying linguistic structures . Finally , it may be useful to complement the corpus-driven approach used in the current paper with constructed evaluation sentences that isolate particular syntactic phenomena , independent of their frequency in a natural corpus , as is common in psycholinguistics ( Enguehard et al . , 2017 ) .

Challenge: Previous work only shows that RNNs can handle constructions that require hierarchical structure when explicit supervision on the target task is given.
Approach: They introduce a probing method for syntactic abilities to evaluate long-distance agreement on standard and nonsensical sentences in multiple languages with different morphological systems.
Outcome: The RNNs trained on an LM objective can solve long-distance agreement problems well even on nonsensical sentences consistently across languages indicating their deeper grammatical competence.