Controlled tasks for model analysis: Retrieving discrete information from sequences (2021)

Abstract: In recent years, the NLP community has shown increasing interest in analysing how deep learning models work. Given that large models trained on complex tasks are difficult to inspect, some of this work has focused on controlled tasks that emulate specific aspects of language. We propose a new set of such controlled tasks to explore a crucial aspect of natural language processing that has not received enough attention: the need to retrieve discrete information from sequences. We also study model behavior on the tasks with simple instantiations of Transformers and LSTMs. Our results highlight the beneficial role of decoder attention and its sometimes unexpected interaction with other components. Moreover, we show that, for most of the tasks, these simple models still show significant difficulties. We hope that the community will take up the analysis possibilities that our tasks afford, and that a clearer understanding of model behavior on the tasks will lead to better and more transparent models.

## introduction there has been a continuous increase in performance in computational linguistics in recent years. this development correlates with larger and more complex models which are trained on ever bigger datasets. these new characteristics of modelling made it harder to understand which model components are relevant and how they work. this has motivated the nlp community to develop new methods for model analysis. given the difficulties analyzing large models, some of this work @xcite @xcite @xcite has focused on controlled tasks that emulate specific aspects of language. we propose a new set of such controlled tasks to explore a crucial aspect of natural language processing that has not received enough attention: the need to retrieve discrete information from sequences. 1 retrieving discrete information from sequences is necessary for natural language processing for instance to track agreement between different constituents of a sentence and to establish coreference relationships (among many other examples). the controlled tasks are built using binary vectors as input in order to exclude the possible inferences of linguistic patterns that are not the focus of our analysis. in particular, we design the tasks such that the models need to emulate four abilities that are crucial for natural language: incremental processing, indirect mappings, contextualization, and order tracking. we study model behavior on the proposed tasks with simple instantiations of transformers @xcite and lstms @xcite , and show that, for most of the tasks, these models show significant difficulties. transformers are the state of the art in deep learning for nlp, and lstms are a classical architecture that was designed for sequence processing. in the analysis, we aim at understanding the role of the different components of the models, focusing on self-attention and positional encoding for transformers and decoder attention for lstms. in the transformer, self-attention turns out to only have a clear beneficial role when the task requires taking token order into account, with decoder attention performing most of the work in other cases. positional encoding plays a counterintuitive role, helping in tasks that do not involve order by providing beneficial noise, and harming performance when it is redundant with self-attention. the lstm only compares favorably to the transformer in short sequence lengths for simple tasks, and decoder attention is highly beneficial for this model. ## related work the current study belongs in the newly developed area of interpretability and explainability of computational linguistics, which seeks to understand how models capture natural language phenomena @xcite @xcite . much of this literature has focused on model behaviour in complex nlp tasks @xcite @xcite @xcite . some other work has used controlled tasks to analyze neural networks regarding specific aspects, such as reasoning skills @xcite , compositionality @xcite , pos tagging @xcite , inductive biases (white and cotterell, 2021) or hierarchical structure @xcite . we follow this latter methodology, and design tasks that highlight one of the core abilities that a model needs to possess in order to process natural language, namely detecting one or more features in a sequence of tokens, possibly in a context-and/or order-sensitive way. ## models the general structure of the models is presented in figure 2 . model equations and training details are provided in the appendix. 2 transformer. the transformer architecture consists of a sequence encoder and a decoder adapted to our tasks. we choose the most basic architecture for interpretability: a single-head, single-layer architecture. to enable incremental processing, the encoder self-attention only looks into the past, as in standard "causal" architectures such as the one of @xcite . the input vectors are mapped to a 100dimensional space and combined with sine-based positional embeddings, as in @xcite . each input token vector then goes through a transformer encoder cell with one attention head and one layer. the query vector is also mapped onto a 100dimensional space, through a separate embedding matrix. decoding works as a dot-product-based attention between the query embedding and each token representation @xcite . the concatenation of the query embedding and the decoded sequence representation is then passed through a multi-layer perceptron (mlp) to generate the answer, a number in the range of [0,1] obtained via the sigmoid function. the model is optimized with a binary cross-entropy loss. at test time, a result larger than 0.5 is considered a positive answer, as is standard. we ablate the full transformer architecture by removing self-attention and skipping positional encoding when embedding the input. the variants solely based on decoder attention are akin to memory networks @xcite . ## conclusion our tasks shed light on how the main model components act and interact regarding the retrieval of discrete information from sequences, uncovering behaviours that would be difficult to detect when the architectures are applied to complex nlp tasks. a take-home message from our experiments is that the presence of a component in an architecture does not imply that the model will learn to use it as expected in a task. in particular, we found that only the need to track ordered information led the architecture to use self-attention in the predicted way, and that decoder attention can be difficult for lstms to use correctly. on the other hand, the components can assume unexpected functions. self-attention in transformers can simply serve to blindly propagate information across time, leading to more robust representations of features contained in earlier tokens, that are copied over and over. also, surprisingly, positional embeddings provide a small but consistent benefit in tasks that do not require order tracking. this suggests that they might have a serendipitous function, possibly adding helpful noise to the representations. moreover, as both self-attention and positional embeddings can learn to keep track of order, the two mechanisms get in the way of each other, making it harder, when combined, for the transformer models to converge on a single strategy to track order. regarding lstm, the sequence length is a very big factor. the model has a steep loss in performance when the sequence gets longer. on the simplest task, the model enhanced with decoder attention develops the ability to switch between strategies, but this doesn't generalize to longer sequences. to conclude, we hope to have shown that the proposed tasks constitute a useful probing mechanism for the ability of models to detect discrete information in sequences, testing in particular four key abilities: incremental processing (in the sense that the query is not known at input processing time), indirect mappings, context-dependence and order tracking. we have shown that most of the tasks are difficult for models even for short sequences; and they can be easily extended to more dimensions and even longer sequences. future work should go in two different directions: 1) examining how behavior changes when the probed models are scaled by increasing the number of layers and attention heads, and 2) modify the models to solve some of the problems that we encounter in the current experiments. we hope that the community will take up the analysis possibilities that our tasks afford, and that a clearer understanding of model behavior on the tasks will lead to better and more transparent models.