How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? (2020)

Abstract: Long short-term memory (LSTM) networks and their variants are capable of encapsulating long-range dependencies, which is evident from their performance on a variety of linguistic tasks. On the other hand, simple recurrent networks (SRNs), which appear more biologically grounded in terms of synaptic connections, have generally been less successful at capturing long-range dependencies as well as the loci of grammatical errors in an unsupervised setting. In this paper, we seek to develop models that bridge the gap between biological plausibility and linguistic competence. We propose a new architecture, the Decay RNN, which incorporates the decaying nature of neuronal activations and models the excitatory and inhibitory connections in a population of neurons. Besides its biological inspiration, our model also shows competitive performance relative to LSTMs on subject-verb agreement, sentence grammaticality, and language modeling tasks. These results provide some pointers towards probing the nature of the inductive biases required for RNN architectures to model linguistic phenomena successfully.

## introduction for the last couple of decades, neural networks have been approached primarily from an engineering perspective, with the key motivation being efficiency, consequently moving further away from biological plausibility. recent developments @xcite @xcite have however incorporated explicit constraints in neural networks to model specific parts of the brain and have found a correlation between the learned activation maps and actual neural activity recordings. thus, these trained networks can perhaps act as a proxy for a theoretical investigation into biological circuits. recurrent neural networks (rnns) have been used to analyze the principles and dynamics of neural population responses by performing the same tasks as animals @xcite . however, these networks violate dale's law @xcite , which states that the neurons have either a purely excitatory or inhibitory effect on other neurons in the mammalian brain. the decaying nature of the potential in the neuron membrane after receiving signals (excitatory or inhibitory) from the surrounding neurons is also well-studied @xcite . the goal of our work is to incorporate these biological features into the rnn structure, which gives rise to a neuro-inspired and computationally inexpensive recurrent network for language modeling, which we call a decay rnn (section 4). we perform learning using the backpropagation algorithm. despite its differences with the way learning is believed to happen in the brain, it has been argued that the brain can implement its core principles @xcite . we assess our model's ability to capture syntax-sensitive dependencies via multiple linguistic tasks (section 6): number prediction, grammaticality judgement @xcite which entails subject-verb agreement, and a more complex language modeling task @xcite . subject-verb agreement, where the main noun and the associated verb must agree in number, is considered as evidence of hierarchical structure in english. this is exemplified using a sentence taken from the dataset made available by @xcite al. (2016): 1. *all trips on the expressway requires a toll. ## related work there has been prior work on using lstms @xcite for language 1 main noun and verb are highlighted in bold. intervening nouns are underlined. asterisks mark unacceptable sentences. from the biological point of view, @xcite used a hard reset of the membrane potential in contrast to a soft decay observed in a neuronal membrane. at the same time, their learning paradigm is similar to the hebbian learning scheme @xcite , which does not involve error backpropagation @xcite . our work is closely related to the idea of modeling the population of neurons as a dynamical system (eirnn) proposed by @xcite . however, their time constant parameter was based on the concepts described in @xcite while the sampling rate was arbitrarily chosen. given that the chosen values only considered a certain class of neurons @xcite , we believe that it is not necessary to have the same values of the parameters for each cognitive task. thus, we build on their formulation by making the sampling rate and time constant learnable as manifested by our decay parameter, described in the next section. ## biological preliminaries according to dale's principle, a neuron is either excitatory or inhibitory @xcite . if a neuron output produces a negative (positive) change in the membrane potential of all the connected neurons via its synapse, then it is said to be an inhibitory (excitatory) neuron. in a set of n neurons, if w is the synaptic connection matrix, then the connection from the neuron j to neuron i is 'excitatory' if w @xmath0, and 'inhibitory' if w @xmath1. @xcite have argued that a balance between structural and response variability (entropy), and excitability (synaptic strength) of a network maximizes the overall learning. this balance is governed by the ratio of inhibitory and excitatory neurons. they have further shown that this balance also maximizes the overall performance in multitask learning. catsigeras (2013) mathematically prove that dale's principle is necessary for an optimal foot_0 neuronal network's dynamics. in the postsynaptic neuron, the integration of synaptic potentials is realized by the addition of excitatory (+ve) and inhibitory (-ve) postsynaptic potentials (psps). psps are electronic voltages, that decay as a function of time due to spontaneous reclosure of the synaptic channels. the decay of the psps is controlled by the membrane constant τ , i.e., the time required by the psp to decay to 37% of its peak value @xcite . ## decay rnn here we present our proposed architecture, which we call the decay rnn (drnn). our architecture aims to model the decaying nature of the voltage in a neuron membrane after receiving impulses from the surrounding neurons. at the same time, we incorporate dale's principle in our architecture. thus, our model captures both the microscopic and macroscopic properties of a group of neurons. adhering to the stated phenomena, we define our model with the following update equations for given input x (t) at time t: here f is a nonlinear activation function, w and u are weight matrices, b is the bias and h (t) represents the hidden state (analogous to voltage). we define α ∈ (0,1) as a learnable parameter to incorporate a decay effect in the hidden state (analogous to the decay in the membrane potential). here α acts as a balancing factor between the hidden state h (t-1) and c (t) . foot_1 w dale is a diagonal matrix, and based on the empirical results on the mammalian brain (hendry and jones, 1981), we set the last 20% of entries to -1, representing the inhibitory connections, and the rest to 1 (see appendix a.3). 5 unlike @xcite , we keep self-connections in the network. besides biological inspiration, our model also has the following salient features. first, the presence of α acts as a coupled gating mechanism to the flow of information (figure 1 ), at the same time maintaining an exponential moving average of the hidden state. thus, α values close to 1 correspond to memories of the distant past. it is worth mentioning that oliva et al. ( 2017 ) have considered the exponential moving average in the context of rnns. however, their approach manually selected a set of scaling parameters, whereas we have a systematic way of arriving at the values of those parameters by making them learnable for the task at hand. second, our model also has an intrinsic skip connection deriving out of its formulation. @xcite has shown that the architectures with skip connections provide an alternate path for the flow of gradients during the error backpropagation. at the same time presence of coupled gates slows down the vanishing of gradient @xcite . thus, despite of its simple un-gated structure, the features discussed above provide safeguards against vanishing gradient. to examine the importance of dale's principle in the learning process, we made a variant of our decay rnn without dale's principle, which we call the slacked decay rnn (sdrnn), with updates to c (t) made as follows: to understand the role of the correlation between the hidden states in the decay rnn formulation, we devised an ablated version of our architecture, which we refer to as the ab-drnn. with the following update equation, we remove the mathematical factor (wh (t-1) ) that gives rise to a correlation between hidden states: ## datasets for the number prediction (section 6.1) and grammaticality judgment (section 6.3) tasks, we used a corpus of 1.57 million sentences from wikipedia @xcite , of which 10% were used for training, 0.4% for validation, and the remaining were reserved for testing. on the other hand, for the language modeling task (section 6.4), the model was trained on a 90 million word subset of wikipedia comprising of 3 million training and 0.3 million validation sentences @xcite . despite having a large number of training points, these datasets have certain drawbacks, including the lack of a sufficient number of syntactically challenging examples leading to poor generalization over the sentences out of the training data distribution. therefore, we construct a generalization set as described in @xcite , where we generate the sentences out of templates that can be described using a non-recursive context-free grammar. the use of the generalization set allows us to test on a much broader range of linguistic phenomena. we will use this dataset for the targeted syntactic evaluation of our trained models. ## experiments here we will describe our experiments foot_3 to assess the models' ability to capture syntax-sensitive dependencies. details regarding the training settings are available in appendix a.4. ## conclusion in this paper, we proposed the decay rnn, a bioinspired recurrent network that emulates the decaying nature of neuronal activations after receiving excitatory and inhibitory impulses from upstream neurons. we have found that the balance between the free term (h (t) ) and the coupled term (wh (t) ) enabled the model to capture syntax-level dependencies. as shown by @xcite ; @xcite , explicitly modeling hierarchical structure helps to discover non-local structural dependencies. the contrast in the performance of the language models encourages us to look at the inductive biases, which might have led to better syntactic generalization in certain cases. recently, @xcite showed the existence of a line attractor in the dynamics of the hidden states for sentiment classification. thus, similar dynamical-system-based analysis can be extended to our settings to further understand the working of the decay rnn. from the cognitive neuroscience perspective, it would be interesting to investigate if the proposed decay rnn can capture some aspects of actual neuronal behaviour and language cognition. our results here do at least indicate that the complex gating mechanisms of lstms (whose cognitive plausibility has not been established) may not be essential to their performance on many linguistic tasks, and that simpler and perhaps more cognitively plausible rnn architectures are worth exploring further as psycholinguistic models.