Seeing Both the Forest and the Trees: Multi-head Attention for Joint Classification on Different Compositional Levels (2020)

Abstract: In natural languages, words are used in association to construct sentences. It is not words in isolation, but the appropriate use of hierarchical structures that conveys the meaning of the whole sentence. Neural networks have the ability to capture expressive language features; however, insights into the link between words and sentences are difficult to acquire automatically. In this work, we design a deep neural network architecture that explicitly wires lower and higher linguistic components; we then evaluate its ability to perform the same task at different hierarchical levels. Settling on broad text classification tasks, we show that our model, MHAL, learns to simultaneously solve them at different levels of granularity by fluidly transferring knowledge between hierarchies. Using a multi-head attention mechanism to tie the representations between single words and full sentences, MHAL systematically outperforms equivalent models that are not incentivized towards developing compositional representations. Moreover, we demonstrate that, with the proposed architecture, the sentence information flows naturally to individual words, allowing the model to behave like a sequence labeler (which is a lower, word-level task) even without any word supervision, in a zero-shot fashion.

## introduction compositional reasoning is fundamental in human cognition: we use it to interact with objects, take actions, reason about numbers, and move in space @xcite . this is also reflected in some aspects of the human language @xcite @xcite since we use words and phrases in association to construct sentences. consequently, there are lower linguistic components that act as building blocks for higher levels. in this work, we focus on two levels of the compositional hierarchy -words and sentences -and ask the following question: are deep neural networks (dnns) trained for a higher-level task (i.e., at the sentence level) able to pick up the features of the compounds needed to solve the same task but at a lower level, such as at the word level? moreover, how are the different hierarchical levels interacting under varying supervision signals? to the best of our knowledge, very few studies have investigated the transferability of a task solution between compositional levels using dnns in controlled experiments. it has been shown that neural networks are universal function approximators @xcite ; they can perform arbitrary function combinations to learn expressive features. dnns trained for language tasks are not an exception to this rule, and recent studies have shown their power in extracting linguistically-rich representations @xcite . however, when trained end-to-end, learning the connection between the different compositional levels is not trivial for these models. this is in part due to the vast syntactic and semantic complexity of natural language. there are also data limitations on most tasks, resulting in networks picking up the noise and biases of the datasets. crucially, dnns trained to solve a task at a higher hierarchical level are usually treated as black boxes with respect to the lower levels. we propose a novel dnn design that stimulates the development of hierarchical connections. the architecture is based on a multi-head attention mechanism that ties the representations between single words and full sentences in a way that enables them to reinforce each other. the proposed multi-level architecture can be viewed as a sentence classifier, where each customized attention head is guided to behave like a sequence labeler, detecting one particular label on each token. thus, it can simultaneously solve language tasks that are situated at different levels of granularity. based on experiments, this architecture systematically outperforms equivalent models that focus only on one level. the token-level supervision explicitly teaches the classifier which areas it needs to focus on in each sentence, while the sentence-level objective provides a regularizing effect and encourages the model to return coherent sequence labeling predictions. moreover, we show that the sentence-level information flows naturally to individual words, allowing the model to behave like a sequence labeler even when it does not receive any word-level supervision. our model exhibits strong transfer capabilities, which we validated on three different tasks: sentiment analysis, named entity recognition, and grammatical error detection. ## multi-head attention labeling (mhal) we describe an architecture that directly ties together the sentence and word representations for multiclass classification, incentivizing the model to make better use of the information on each level of granularity. in addition, we present several auxiliary objectives that guide this architecture towards useful hierarchical representations and better performance. ## experiments in this section, our goal is to test whether the proposed joint training architecture is able to 1) transfer knowledge between words and sentences and improve on both text labeling tasks, 2) learn to re-use the supervision signal received on the sentence-level to perform a word-level task, and 3) use the auxiliary objectives and regularization loss to improve its performance. we perform three main experiments under different training regimes: if successful, this model will be able to perform sophisticated word-predictions solely based on the considerably cheaper sentence annotations. ## related work most methods for text classification (hierarchical or not) treat sentence classification and sequence labeling as completely separate tasks @xcite @xcite . more recently, training a model end-to-end for more than one language task has become increasingly popular @xcite , as well as using auxiliary objectives to inject useful inductive biases @xcite @xcite . our work is similar in terms of motivation for the auxiliary objectives and multi-task training procedure. however, instead of learning to perform multiple tasks on the same level, we focus on performing the same task on multiple levels. by carefully designing the network and including specific auxiliary objectives, these levels are able to provide mutually beneficial information to each other. other hierarchical multi-task systems, such as the models proposed by @xcite and @xcite , solve each task at a different dnn layer, but their formulation does not follow a compositional linguistic motivation. our work is most similar to @xcite and @xcite , who described an architecture for supervising attention in a binary text classification setting. @xcite also used a related model to guide the network to focus on similar areas as humans, based on human gaze recordings. we build on these ideas and describe a more general framework, extending it to both multiclass text classification and multiclass sequence labeling. an important part of our new architecture is based on attention mechanisms @xcite , and, in particular, on the properties of multi-head attention @xcite . other regularization techniques that explicitly encourage the learning of more diverse attention functions have been proposed by @xcite , who introduced a disagreement regularization term, and by @xcite and @xcite , who proposed sparse attention for increased interpretability. ## conclusion we investigated a novel neural architecture for natural language representations, which explicitly ties together predictions on multiple levels of granularity. the dynamically calculated weights in a multi-head attention component for composing sentence representations are also connected to token-level predictions, with each attention head focusing on detecting one particular label. this model can then be optimized as either a sentence classifier or a token labeler, or jointly for both tasks, with information being shared between the two levels. supervision on the token labeling task also teaches the model where to assign more attention when composing sentence representations. in return, the sentence-level objective acts as a contextual regularizer for the token labeler and encourages the model to predict token-level tags that cohere with the rest of the sentence. we also introduce several auxiliary objectives that are further incentivizing the architecture to share information between the different levels and help this model get the most benefit from the available training data, increasing its efficiency. we evaluated the proposed architecture on three different text classification tasks: sentiment analysis, named entity recognition, and grammatical error detection. the experiments showed that supervision on both levels of granularity consistently outperformed models that were optimized only on one level. this held true even for cases where the sentence-level labels could be automatically derived from token-level annotation, therefore requiring manually annotated labels only on one level. the auxiliary objectives, designed to connect the predictions between the two tasks more closely, further improved model performance. the semi-supervised experiments showed that this architecture can also be used with partial labeling -with a 50-70% reduction in token-annotated data, the model was able to get comparable results to the baseline architecture using the full dataset. finally, we presented the first experiments for multi-class zero-shot sequence labeling, where the model needs to label tokens while only learning from the sentence-level annotation. as the architecture connects each attention head to a particular label, it was able to learn even in this challenging setting, with the auxiliary objectives being particularly beneficial. the overall multi-level learning approach also has potential future applications in the area of neural network interpretability, as the model can be trained to focus on the same evidence as human users when classifying text and the resulting token-level decisions can be both measured and visualized.