Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020)

Abstract: LSTM-based recurrent neural networks are the state-of-the-art for many natural language processing (NLP) tasks. Despite their performance, it is unclear whether, or how, LSTMs learn structural features of natural languages such as subject-verb number agreement in English. Lacking this understanding, the generality of LSTM performance on this task and their suitability for related tasks remains uncertain. Further, errors cannot be properly attributed to a lack of structural capability, training data omissions, or other exceptional faults. We introduce *influence paths*, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network. The approach refines the notion of influence (the subject’s grammatical number has influence on the grammatical number of the subsequent verb) into a set of gate or neuron-level paths. The set localizes and segments the concept (e.g., subject-verb agreement), its constituent elements (e.g., the subject), and related or interfering elements (e.g., attractors). We exemplify the methodology on a widely-studied multi-layer LSTM language model, demonstrating its accounting for subject-verb number agreement. The results offer both a finer and a more complete view of an LSTM’s handling of this structural aspect of the English language than prior results based on diagnostic classifiers and ablation.

## introduction traditional rule-based nlp techniques can capture syntactic structures, while statistical nlp techniques, such as n-gram models, can heuristically integrate semantics of a natural language. modern rnn-based models such as long short-term memory (lstm) models are tasked with incorporating both semantic features from the statistical associations in their training corpus, and structural features generalized from the same. despite evidence that lstms can capture syntactic rules in artificial languages @xcite , it is unclear whether they are as capable in natural languages @xcite in the context of rules such as subject-verb number agreement, especially when not supervised for the particular feature. the incongruence derives from this central question: does an lstm language model's apparent performance in subject-verb number agreement derive from statistical heuristics (like n-gram models) or from generalized knowledge (like rule-based models)? recent work has begun addressing this question @xcite in the context of language models: models tasked with modeling the likelihood of the next word following a sequence of words as expected in a natural language (see @xcite . subject-verb number agreement dictates that the verb associated with a given subject should match its number (e.g., in figure 1 , the verb "run" should match with the subject "boys"). @xcite showed that the subject grammatical number is associated with various gates in an lstm, and @xcite showed that ablation (disabling activation) of an lstm model at certain locations can reduce its accuracy at scoring verbs of the correct grammatical number. influence offers an alternate means of exploring properties like number agreement. we say an input is influential on an outcome when changing just the input and nothing else induces a change on the outcome. in english grammar, the number of a subject is influential on the number of its verb, in that changing the number of that subject while keeping all other elements of a sentence fixed would necessitate a change in the number of the verb. algorithmic transparency literature offers formal definitions for empirically quantifying notions of influence for systems in general @xcite and for deep neural networks specifically @xcite . the mere fact that subject number is influential on verb number as output by an lstm model is sufficient to conclude that it incorporates the agreement concept in some way but does not indicate whether it operates as a statistical heuristic or as a generalized rule. we address this question with influence paths, which decompose influence into a set of paths across the gates and neurons of an lstm model. the approach has several elements: 1. define an input parameter to vary the conceptspecific quantity under study (e.g., the grammatical number of a particular noun, bottomleft node in figure 1 ) and a concept-specific output feature to measure the parameter's effect on (e.g, number agreement with the parameterized noun, bottom-right node in figure 1 ). the paths demonstrate where relevant state information necessitated by the concept is kept, how it gets there, how it ends up being used to affect the model's output, and how and where related concepts interfere. our approach is state-agnostic in that it does not require a priori an assumption about how or if the concept will be implemented by the lstm. this differs from works on diagnostic classifiers where a representation of the concept is assumed to exist in the network's latent space. the approach is also time-aware in that paths travel through cells/gates/neurons at different stages of an rnn evaluation. this differs from previous ablationbased techniques, which localize the number by clearing neurons at some position in an rnn for all time steps. our contributions are as follows: our results point to generalized knowledge as the answer to the central question. the number agreement concept is heavily centralized to the primary path despite the varieties of contexts. further, the primary path's contribution is undiminished even amongst interfering contexts; number errors are not attributable to lack of the general number concept but rather to sufficiently influential contexts pushing the result in the opposite direction. ## methods our method for computing influence paths begins with modeling a relevant concept, such as grammatical number, in the influence framework of leino et al. (definition 1) by defining a quantity of interest that corresponds to the grammatical number of the verb, and defining a component of the input embedding that isolates the subject's grammatical number (section 3.1). we then decompose the influence measure along the relevant structures of lstm (gates or neurons) as per standard calculus identities to obtain a definition for influence paths (section 3.2). ## evaluation in this section we apply influence path decomposition to the na task. we investigate major gatelevel paths and their influence concentrations in section 4.2. we further show the relations between these paths and the paths carrying grammatical number from intervening nouns (i.e. attractors & helpful nouns) in section 4.3. in both we also investigate high-attribution neurons along primary paths allowing us to compare our results to prior work. ## conclusions the combination of finely-tuned attribution and gradient decomposition lets us investigate the handling of the grammatical number agreement concept attributed to paths across lstm components. the concentration of attribution to a primary path and two primary cell state neurons and its persistence in a variety of short-term and long-term contexts, even with confounding attractors, demonstrates that the concept's handling is, to a large degree, general and localized. though the heuristic decisioning aspect of an lstm is present in the large quantities of paths with non-zero influence, their overall contribution to the concept is insignificant as compared to the primary path. node-based compression results further corroborate these conclusions. we note, however, that our results are based on datasets exercising the agreement concept in contexts of a limited size. we speculate that the primary path's attribution diminishes with the length of the context, which would suggest that at some context size, the handling of number will devolve to be mostly heuristic-like with no significant primary paths. though our present datasets do not pose computational problems, the number of paths, at both the neuron and the gate level, is exponential with respect to context size. investigating longer contexts, the diminishing dominance of the primary path, and the requisite algorithmic scalability requirements are elements of our ongoing work. we also note that our method can be expanded to explore number agreement in more complicated sentences with clausal structures, or other syntactic/semantic signals such as coreference or gender agreement. acknowledgement this work was developed with the support of nsf grant cns-1704845 as well as by darpa and the air force research laboratory under agreement number fa8750-15-2-0277. the u.s. government is authorized to reproduce and distribute reprints for governmental purposes not withstanding any copyright notation thereon. the views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of darpa, the air force research laboratory, the national science foundation, or the u.s. government. we gratefully acknowledge the support of nvidia corporation with the donation of the titan v gpu used for this work.