Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyond (2023)

Abstract: Is the output softmax layer, which is adopted by most language models (LMs), always the best way to compute the next word probability? Given so many attention layers in a modern transformer-based LM, are the pointer networks redundant nowadays? In this study, we discover that the answers to both questions are no. This is because the softmax bottleneck sometimes prevents the LMs from predicting the desired distribution and the pointer networks can be used to break the bottleneck efficiently. Based on the finding, we propose several softmax alternatives by simplifying the pointer networks and accelerating the word-by-word rerankers. In GPT-2, our proposals are significantly better and more efficient than mixture of softmax, a state-of-the-art softmax alternative. In summarization experiments, without very significantly decreasing its training/testing speed, our best method based on T5-Small improves factCC score by 2 points in CNN/DM and XSUM dataset, and improves MAUVE scores by 30% in BookSum paragraph-level dataset.

## introduction when recurrent neural networks such as lstm @xcite are the mainstream language model (lm) architecture, pointer networks, or so-called copy mechanisms @xcite , have been shown to improve the state-of-the-art lms for next word prediction @xcite and summarizations @xcite by a large margin. however, after transformer @xcite becomes the dominating lm architectures, the pointer networks are rarely used in the state-of-the-art pretrained lms.one major reason is that the attention mechanism in every transformer layer can learn to copy the words from the context, so it * indicates equal contribution † the work is done while the author was at umass ## background before introducing our method, we would first briefly review the problem we are solving and its state-of-the-art solutions. ## methods to break the softmax bottleneck more efficiently compared to mos, our overall strategy is simple. if we can identify a small partition of words that are very likely to become the next word, we can just compute the dot products between a hidden state and the embeddings of these likely words instead of all the words as in mos. for example, if we can identify king and woman are much more likely to appear than queen and man, we can only compute the dot product between a hidden state and the embeddings of king and woman without being interfered by other words. specifically, when we compute the next word probability in equation 1, the logit of the word x given the context c t to simplify our explanation, we will focus on the decoder-only lm (i.e., gpt-2) first and extend our method to encoder-decoder lm (i.e., t5 and bart). ## experiments the pointer network was a popular technique in language modeling @xcite and summarization @xcite . thus, we also focus on these two fundamental applications. ## related work repetition and hallucination are two common problems in language generation tasks. one common solution for repetition is to avoid outputting the words in the context, which is often called unlikelihood training @xcite to alleviate the hallucination problem or satisfy some constraints, many recent generation models rerank the generated text @xcite @xcite @xcite @xcite @xcite @xcite @xcite . although being effective, the rerankers usually slow down significantly the training and/or inference speed (as our word-by-word reranker baseline) and might occupy extra memory resources. our analyses demonstrate that parts of the hallucination and repetition problem come from the softmax bottleneck. the findings provide an explanation for the effectiveness of prior studies such as the above reranker approaches and pointer networks @xcite @xcite . another example is encouraging the word embeddings to be isotropy @xcite . their improvement might also come from reducing linear dependency of the candidate word embeddings. nevertheless, their side effect of breaking the similarity structure in the word embedding space might hurt the generation quality in some cases. concurrently to our work, @xcite also use the softmax bottleneck theory @xcite to explain the improvement of a pointer network. their empirical results also support our conclusion that softmax bottleneck is a major reason that causes the factuality problem of lms. our work is motivated and inspired by @xcite . in their work, they also propose to use different hidden states for different vocabulary partitions, but their partitioning is global and needs to be combined with the mixture of softmax (mos) approach, which adds a significant overhead compared to the standard softmax layer. our dynamic partitioning methods not only perform better but greatly reduce the overhead by removing the reliance on mos. ## conclusion since the transformer becomes the mainstream encoder and decoder for lms, the output softmax layer seems to be the only reasonable option for computing the word probability distribution. although being simple and efficient, the softmax layer is inherently limited while the existing solutions are relatively slow @xcite . this work proposes a series of softmax alternatives that can improve the text generation models without increasing the computational costs significantly. our experiments suggest that the main improvement of the pointer network on top of a transformer comes from breaking the softmax bottleneck. our results also indicate that the alternatives could alleviate some problems of hallucination, repetition, and too generic generation. furthermore, all of the proposed alternatives can be applied to the lms that have already been pretrained using softmax without requiring retraining from scratch. for the practitioner, we recommend using all the partitioning methods together to get the best performance, or using only the simple context partition to keep the architecture simple while getting the majority of the gain.