The Unstoppable Rise of Computational Linguistics in Deep Learning (2020)

Abstract: In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the challenges facing research in deep learning architectures for natural language understanding.

## introduction when neural networks first started being applied to natural language in the 1980s and 90s, they represented a radical departure from standard practice in computational linguistics. connectionists had vector representations and learning algorithms, and they didn't see any need for anything else. everything was a point in a vector space, and everything about the nature of language could be learned from data. on the other hand, most computational linguists had linguistic theories and the poverty-of-thestimulus argument. obviously some things were learned from data, but all the interesting things about the nature of language had to be innate. a quarter century later, we can say two things with certainty: they were both wrong. vector-space representations and machine learning algorithms are much more powerful than was thought. much of the linguistic knowledge which computational linguists assumed needed to be innate can in fact be learned from data. but the unbounded discrete structured representations they used have not been replaced by vector-space representations. instead, the successful uses of neural networks in computational linguistics have replaced specific pieces of computational-linguistic models with new neural network architectures which bring together continuous vector spaces with structured representations in ways which are novel for both machine learning and computational linguistics. thus, the great progress which we have made through the application of neural networks to natural language processing should not be viewed as a conquest, but as a compromise. as well as the unquestionable impact of machine learning research on nlp, the nature of language has had a profound impact on progress in machine learning. in this paper we trace this impact, and speculate on future progress and its limits. we start with a sketch of the insights from grammar formalisms about the nature of language, with their multiple levels, structured representations and rules. the rules were soon learned with statistical methods, followed by the use of neural networks to replace symbols with induced vectors, but the most effective models still kept structured representations, such as syntactic trees. more recently, attention-based models have replaced hand-coded structures with induced structures. the resulting models represent language with multiple levels of structured representations, much as has always been done. given this perspective, we identify remaining challenges in learning language from data, and its possible limitations. ## inducing features of entities early work on neural networks for natural language recognised the potential of neural networks for learning the features as well, replacing feature engineering. but empirically successful neural network models for nlp were only achieved with approaches where the neural network was used to model one component within an otherwise traditional symbolic nlp model. the first work to achieve empirical success in comparison to non-neural statistical models was work on language modelling. @xcite used an mlp to estimate the parameters of an n-gram language model, and showed improvements when interpolated with a statistical n-gram language model. a crucial innovation of this model was the introduction of word embeddings. the idea that the properties of a word could be represented by a vector reflecting the distribution of the word in text was introduced earlier in non-neural statistical models (e.g. @xcite @xcite @xcite ). this work showed that similarity in the ptb constituents model lp lr f1 @xcite pos 57.8 64.9 61.1 @xcite pos 83.3 84.3 83.8 @xcite 88.8 89.5 89.1 @xcite 89.8 90.4 90.1 vinyals et al. (2015 ) @xmath0. (2015) attn 88.3 vinyals et al. (2015) seq2seq semisup 90.5 conll09 dependencies model (transition-based) uas las titov and henderson (2007a)* 91.44 88.65 chen and manning (2014)* 89.17 86.49 yazdani and henderson (2015) 90.75 88.14 stanford dependencies model (transition-based) more recent work has used neural network language models to learn context-dependent embeddings of words. we will refer to such contextdependent embeddings as token embeddings. for example, @xcite train a stacked bil-stm language model, and these token embeddings have proved effective in many tasks. more such models will be discussed below. for syntactic parsing, early connectionist approaches @xcite @xcite had limited success. the first neural network models to achieve empirical success used a recurrent neural network to model the derivation structure of a traditional syntactic constituency parser @xcite . the recurrent neural network learns to model the sequence of parser actions, estimating the probability of the next parser action given the history of previous parser actions. this allows the decoding algorithm from the traditional parsing model to be used to efficiently search the space of possi-ble parses. these models have also been applied to syntactic dependency parsing @xcite and joint syntactic-semantic dependency parsing @xcite . crucially, these neural networks do not model the sequence of parser decisions as a flat sequence, but instead model the derivation structure it specifies. a derivation structure includes relationships for the inter-dependencies between nodes in the parse tree. the pattern of interconnections between hidden layers of the recurrent neural network (henceforth referred to as the model structure) is designed to follow locality in this derivation structure, thereby giving the neural network a linguistically appropriate inductive bias. more recently, @xcite provide a more direct relationship between the derivation structure and the model structure with their stacklstm parsing model. in all these models, the use of recurrent neural networks allows arbitrarily large parse structures to be modelled without making any hard independence assumptions, in contrast to non-neural statistical models. feed-forward neural networks have also been applied to modelling the derivation structure @xcite , but the accuracy is worse than using recurrent models (see representing the parse tree as a derivation sequence, rather than a derivation structure, makes it possible to define syntactic parsing as a sequenceto-sequence problem, mapping the sentence to its parse sequence. if a neural network architecture for modelling sequences (called seq2seq models) can perform well at this task, then maybe the structured linguistic representations of natural language are not necessary (contrary to @xcite ), not even to predict those structures. @xcite report very poor results for seq2seq models when trained on the standard dataset, but good results when trained on very large automatically-parsed corpora (see all these results demonstrate the incredible effectiveness of inducing vector-space representations with neural networks, relieving us from the need to do feature engineering. but neural networks do not relieve us of the need to understand the nature of language when designing our models. instead of feature engineering, these results show that the best accuracy is achieved by engineering the inductive bias of deep learning models through their model structure. by designing a hand-coded model structure which reflects the linguistic structure, locality in the model structure can reflect locality in the linguistic structure. the neural network then induces features of the entities in this model structure. ## inducing relations between entities with the introduction of attention-based models, the model structure can now be learned. by choosing the nodes to be linguistically-motivated entities, learning the model structure in effect learns the statistical inter-dependencies between entities, which is what we have been referring to as relations. ## conclusions we conclude that the nature of language has influenced the design of deep learning architectures in fundamental ways. vector space representations (as in mlps) are not adequate, nor are vector spaces which evolve over time (as in lstms). attentionbased models are fundamentally different because they use bag-of-vector representations. bov representations are nonparametric representations, in that the number of vectors in the bag can grow arbitrarily large, and these vectors are exchangeable. with bov representations, attention-based neural network models like transformer can model the kinds of unbounded structured representations that computational linguists have found to be necessary to capture the generalisations in natural language. and deep learning allows many aspects of these structured representations to be learned from data. however, successful deep learning architectures for natural language currently still have many handcoded aspects. the levels of representation are hand-coded, based on linguistic theory or available resources. often deep learning models only address one level at a time, whereas a full model would involve levels ranging from the perceptual input to logical reasoning. even within a given level, the set of entities is a pre-defined function of the text. this analysis suggests that an important next step in deep learning architectures for natural language understanding will be the induction of entities. it is not clear what advances in deep learning methods will be necessary to improve over our current fixed entity definitions, nor whether the resulting entities will be any different from the ones postulated by linguistic theory. if we can induce the entities at a given level, a more challenging task will be the induction of the levels themselves. the presumably-innate nature of linguistic levels suggests that this might not even be possible. but of one thing we can be certain: the immense success of adapting deep learning architectures to fit with our computational-linguistic understanding of the nature of language will doubtless continue, with greater insights for both natural language processing and machine learning.