Inducing Grammar from Long Short-Term Memory Networks by Shapley Decomposition (2020)

Abstract: The principle of compositionality has deep roots in linguistics: the meaning of an expression is determined by its structure and the meanings of its constituents. However, modern neural network models such as long short-term memory network process expressions in a linear fashion and do not seem to incorporate more complex compositional patterns. In this work, we show that we can explicitly induce grammar by tracing the computational process of a long short-term memory network. We show: (i) the multiplicative nature of long short-term memory network allows complex interaction beyond sequential linear combination; (ii) we can generate compositional trees from the network without external linguistic knowledge; (iii) we evaluate the syntactic difference between the generated trees, randomly generated trees and gold reference trees produced by constituency parsers; (iv) we evaluate whether the generated trees contain the rich semantic information.

## introduction recurrent neural networks have demonstrated surprising performance on processing natural language data, surpassing traditional n-gram or handengineered features on a variety of tasks. naturally, curiosity about whether these models capture aspects of linguistic knowledge increases. recent works proposed different probing tests on whether a model learns a set of linguistic properties @xcite such as subject-verb agreement @xcite , syntax-sensitive dependencies @xcite , whether a neuron learns to recognize a group of words with special properties (such as date) @xcite , or by dropping the word in the context far away vs nearby and trace perplexity to see how neural networks leverage context @xcite . however, there are two major limitations of the probing tests: i) probing tests are limited in the scope of their claim; ii) probing tests often treat model as a blackbox, reaching conclusions by directly altering the testing stimuli and observing the change in the outcome. this type of research often does not yield satisfactory conclusion about the underlying complex mechanism of the blackbox model @xcite . more holistic approach has been explored to study whether modern neural networks understand sentences by implicitly inducing recursive structures that match the semantics and syntactic theories in linguistics @xcite . however, @xcite studied a specific type of models that explicitly build tree representations of each sentence, which are far from common text processing models such as long short-term memory (lstm) networks @xcite . in the end, the question of whether common text processing model assumes implicit linguistic structures is left unanswered. in this work, we draw inspirations from the field of deep learning model interpretations to provide a glimpse into how lstm networks process a sentence, and extract a tree structure that lstm networks implicitly create. using the techniques from contextual decomposition @xcite , we propose a tree building algorithm that mimics construction grammar in that the grammar we induce is conditionally dependent on the task and the sentence. we extend @xcite 's analysis on the trees generated from the lstm networks. we evaluate whether the induced tree structures syntactically resemble constituency grammar, and we evaluate whether training a recursive neural network on the induced structure will provide performance gain over recursive neural network on the constituency grammar. similar to @xcite 's conclusion on models that explicitly build tree representation, we conclude that induced trees from lstm networks also do not resemble semantic or syntactic formalism created by human. we hope our work can encourage future work about interpretation-based methods and their connections with semantic and syntactic theory in linguistics. ## discussion and conclusion in this work, we extract trees from lstm by an interpretation algorithm -agglomerative contextual decomposition (acd). we show empirically that the generated trees are not similar to the trees produced from formal syntactic theory. the generated trees also do not seem to provide more computational improvement when we train a recursive neural network leveraging the structure to predict the final label. these negative observations can result from several possible reasons. firstly, as discussed in sec 2.6, the connection between the structure induction through machine interpretation algorithm and construction grammar remains a questionwhether what is semantically important for sentiment analysis is necessarily reflected in the syntax and the way the syntactic constituents are formed in the language? besides, while sentiment analysis requires the understandings of compositionality, models trained on linguistic tasks may better capture syntactic information. for future work, we consider conducting the same experiments on cola, a dataset for judging the grammatical acceptability of a sentence @xcite . moreover, it is unclear whether models truly learned compositionality or just overfit to some spurious patterns of the dataset, as recent works have demonstrated that a well-performing natural language inference model completely fails on challenging cases generated by syntactic transformations @xcite . nonetheless, we conclude with encouragement for the community to look deeper into interpretation-based methods and their connections with semantic and syntactic theory in linguistics.