Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization (2021)

Abstract: Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of information scattered across the long document, and (2) composing a cohesive text by reconstructing these salient facts into a shorter summary that faithfully reflects the complex relations connecting these facts. In this paper, we adapt TP-Transformer (Schlag et al., 2019), an architecture that enriches the original Transformer (Vaswani et al., 2017) with the explicitly compositional Tensor Product Representation (TPR), for the task of abstractive summarization. The key feature of our model is a structural bias that we introduce by encoding two separate representations for each token to represent the syntactic structure (with role vectors) and semantic content (with filler vectors) separately. The model then binds the role and filler vectors into the TPR as the layer output. We argue that the structured intermediate representations enable the model to take better control of the contents (salient facts) and structures (the syntax that connects the facts) when generating the summary. Empirically, we show that our TP-Transformer outperforms the Transformer and the original TP-Transformer significantly on several abstractive summarization datasets based on both automatic and human evaluations. On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and the performance gain by information specificity of the role vectors and improved syntactic interpretability in the TPR layer outputs.(Code and models are available at https://github.com/jiangycTarheel/TPT-Summ )

## introduction abstractive summarization is the task of generating a shorter version of a source text without necessarily reusing the sentences from the original source, original text (truncated): authorities said the incident took place on sao joao beach in caparica, south-west of lisbon. the national maritime authority said a middle-aged man and a young girl died after they were unable to avoid the plane. [....] other reports said the victims had been sunbathing when the plane made its emergency landing. @xcite video footage from the scene carried by local broadcasters showed a small recreational plane parked on the sand, apparently intact and surrounded by beachgoers and emergency workers. @xcite reference summary: a man and a child have been killed after a light aircraft made an emergency landing on a beach in portugal. while preserving the meaning of its salient contents. it is a complex task that requires: semantic understanding of the source text and reasoning over its lexical units, making inferences about their relation to extract salient facts which are scattered across the long document, as well as generating a concise and coherent sequence of new sentences that covers the salient facts. while humans are remarkably good at this type of reasoning and abstraction, developing models that are capable of extraction, comprehension, abstraction, and reformulation of salient contents has been an open research question. one prominent aspect of abstractive summarization is that models struggle with combining multiple salient aspects in the source text into a coherent and grammatical set of sentences that preserve the original information in the source document. as shown in fig. 1 , these pieces of salient information ("death", "emergency landing", "beach") are often connected by complex syntactic, causal, and temporal relations and are loosely grouped under the main topic of the source document. the transformer models @xcite encode syntactic and semantic information of the input text into a single representation space with the self-attention, and decode the salient aspects into a short summary with the cross-attention. however, despite the large number of training examples, current state-of-theart transformer based approaches still struggle with systematic generalization of the composition of multiple salient pieces of information. in this paper, we investigate new types of computational primitives for transformers based on tensor product representations (tprs) @xcite ) which are explicitly-compositional vector embeddings of symbolic structures. a tensor product representation encodes a constituent in a symbolic structure as a composite of a role, which encodes the structural information (e.g., the dependency relation with another word), and a filler, which encodes the content of the constituent (e.g., the meaning of a word). analogously, the tp-transformer constructs a pair of representations for every token at every layer: a filler vector returned by attention and a novel role vector. as visualized in fig. 2 , the model then binds the role and filler vectors to produce the output of every token as a tpr. we adapt the tp-transformer @xcite , which was proposed for solving mathematics problems, for the task of abstractive summarization. unlike the original tp-transformer, which directly projects the input representation into a continuous role vector space, our model generates the role vectors by attending to a learned dictionary of role embeddings @xcite . we observe that most learned role attention distributions are approximately one-hot, thus restricting the role vectors to a highly discrete space. this structural inductive bias encourages the tp-transformer to encode the syntactic information in the discrete roles while isolating the semantics in the continuous fillers. to test the ability of our tp-transformer with discrete roles against the standard transformer and the tp-transformer with continuous roles, we build several models from scratch on a number of summarization datasets spanning different degrees of abstractiveness, output summary lengths, and domains. our tp-transformer significantly outperforms the standard transformer and the tp-transformer with continuous roles on the xsum @xcite , wikihow @xcite , and arxiv @xcite datasets and achieves competitive performance on the cnn/daily mail @xcite dataset, measured by automatic metrics including rouge @xcite and meteor @xcite . our human evaluations on xsum and wikihow datasets also correlate with the automatic metrics, demonstrating that summaries generated by our tp-transformer are indeed better than the trans-former's generations. furthermore, to investigate the structural representation that naturally emerges during training and the advantage of having compositional tpr hidden states, we design a suite of decoder probing tasks to explore the information encoded in the role, filler, and tpr space. we adopt the encoder probing task design presented in @xcite and create four decoder probing tasks: part-of-speech tagging (pos), dependency labeling (dep), semantic role labeling (srl), and named entity labeling (nel). our findings collectively show that the decoder's role vectors encode a wealth of syntactic structures, aiding the decoder in deducing the syntactic features (e.g., being a proper noun, being the object of the root predicate) of the next token to be generated. the decoder's filler vectors on the other hand encode more semantic information (e.g., being a person's name). furthermore, we observe that having the compositional tpr results in a more interpretable final representation than the original transformer has at every layer, regarding the syntactic features of the next word to be generated. our results support our hypothesis that by disentangling semantics and syntax, such structured intermediate representations enable the model to better control both the content to be conveyed and the syntactic structure needed to express it, ultimately improving the factuality and grammaticality of the generated summaries. our overall contributions are as follows: (1) we present a novel adaptation of the original transformer architecture that incorporates a dictionary of role embeddings at every layer and generates tensor product representation by binding the role vectors with attention outputs (filler vectors); (2) show that our tp-transformer outperforms the transformer as well as the original tp-transformer @xcite on several abstractive summarization datasets; and (3) demonstrate the emergent structures in representations by revealing the disentangled syntactic and semantic information encoded in the role and filler spaces. ## the tp-transformer we build our tp-transformer based on the transformer architecture used in @xcite . a tp-transformer encoder applied to a sequence of tokens @xmath0, ..., i can be seen as a 2-dimensional lattice of cells (i, l) where i is the position of the input token and @xmath1, ..., l are the layer indices. all cells in the encoder have the same architecture and the cells at the same layer share the same weights. we introduce the basic components of a tp-transformer cell in sec. 2.2 and its encoder and decoder cells in sec. 2.3. ## probing experiments probing is a method to test whether some particular information is present in the model's encodings. to achieve this, an auxiliary classifier is trained to predict specified linguistic features from the model's internal representations. we probe different components (roles, filler, tprs) in our tp-transformers as well as the attention+residual outputs (equivalent to the filler) of the transformer to assess the naturally emergent structures encoded in the role vectors and the effectiveness of the tpr in the decoding process. by conducting the probing experiments, we aim to (1) provide some insights and evidence of the different information encoded by the role and filler vectors; and (2) explain the rouge advantage of our tp-transformer by showing that its output representation can better encode the linguistic structural information concerning multiple probing tasks. ## conclusion in this work, we enrich the transformer model with the structured tensor product representation for abstractive summarization tasks. we represent every token as a pair of role and filler vectors. we show that our tp-transformer with discrete roles outperforms transformer and tp-transformer with continuous roles on several abstractive summarization datasets, in both metrics scores and human evaluation. we further demonstrate the syntactic structures encoded in the role vectors and show the improved syntactic interpretability in our model's hidden states.