Continuous Decomposition of Granularity for Neural Paraphrase Generation (2022)

Abstract: While Transformers have had significant success in paragraph generation, they treat sentences as linear sequences of tokens and often neglect their hierarchical information. Prior work has shown that decomposing the levels of granularity (e.g., word, phrase, or sentence) for input tokens has produced substantial improvements, suggesting the possibility of enhancing Transformers via more fine-grained modeling of granularity. In this work, we present continuous decomposition of granularity for neural paraphrase generation (C-DNPG): an advanced extension of multi-head self-attention with: 1) a granularity head that automatically infers the hierarchical structure of a sentence by neurally estimating the granularity level of each input token; and 2) two novel attention masks, namely, granularity resonance and granularity scope, to efficiently encode granularity into attention. Experiments on two benchmarks, including Quora question pairs and Twitter URLs have shown that C-DNPG outperforms baseline models by a significant margin. Qualitative analysis reveals that C-DNPG indeed captures fine-grained levels of granularity with effectiveness.

## introduction with the continued success in nlp tasks @xcite , transformer has been the mainstream neural architecture for paraphrase generation @xcite @xcite @xcite . the core component of transformer is the self-attention network (san) @xcite which computes sentence representations at each position by baking representations over all other positions in a parallel way. despite their effectiveness, transformer has been shown to be limited in text what is the reason for world war ii ? decomposition 1 what is the reason for world war ii ? decomposition 2 what is the reason for world war ii ? decomposition 3 what is the reason for world war ii ? decomposition 4 what is the reason for world war ii ? decomposition 5 what is the reason for world war ii ? ⇓ levels of granularity (marked as superscripts): what 1 is 1 the 2 reason 3 of 2 world 4 war 4 ii 5 ? one potential route towards addressing this issues is multi-granularity text modeling (illustrated in despite showing promising results, dnpg only captures a discrete (coarser-grained) decomposition of granularity, which restricts the capacity in representing more fine-grained semantic hierarchy. furthermore, dnpg models the different levels of granularity using multiple encoders and decoders. that amounts to training multiple transformers. the computational cost increases greatly as the number of granularity levels grows. in this paper, we present c-dnpg (stands for continuous decomposition of granularity for paraphrase generation), a simple, fine-grained, and seamlessly integrated model for granularity-aware paragraph representation. c-dnpg extends the vanilla attention network with a granularity head, which neurally estimates a continuous level of granularity for each token. in order to efficiently encode granularity into transformers, c-dnpg adjusts the original self-attention weights using two novel attention masks: 1) a granularity-resonance mask which encourages attentions to exist between tokens with similar granularity; and 2) a granularity scope mask which encourages a small attention scope for lower-level (words or phrases) tokens. the granularity-aware attention mechanism provides a continuous modeling of sentence granularity and can be seamlessly integrated into the vanilla transformer as the basic processing cell. we evaluate the proposed c-dnpg on two commonly-used benchmarks, including the quora question pairs and twitter url paraphrasing. experimental results show that c-dnpg remarkably outperforms baseline models on both benchmarks and achieve the state-of-the-art results in many metrics. qualitative study confirms the ability of the proposed approach in modeling fine-grained granularity. our contributions can be summarized as follows: ## background our approach is extended based on the transformer and self-attention networks. we begin by introducing the background of these techniques. ## approach the vanilla transformer processes disperse words in a flat and uniform way, which makes it difficult to represent words in terms of their syntactic guidance @xcite . prior work has shown that decomposing the levels of granularity (phrases or templates) has produced substantial gains in paraphrase generation @xcite , suggesting the possibility of further improvement from finer-grained modeling of granularity @xcite . motivated by the benefit of explicitly denoting word granularity, we propose ga-attention, a new self-attention block which automatically decomposes fine-grained granularity and learns granularity-aware sentence representations. we integrate ga-attention into the vanilla transformer to generate granularity-aware paraphrases. ## experiments we evaluate our approach by experimenting on two widely used datasets, including the quora question pairs and the twitter urls. we will introduce the common experimental setup and the empirical results. implementation details we implemented our approach on top of the huggingface pytorch transformer @xcite . for a fair comparison, we followed the hyperparameter settings in related works @xcite for the transformer. both encoder and decoder consist of 3 transformer layers, have a hidden size of 450, and contain 9 attention heads (@xmath0, @xmath1, @xmath2. following previous work @xcite , we truncate sentences to 20 tokens. we utilize the pretrained tokenizer by huggingface foot_0 (i.e., bert-baseuncased) for tokenization which has been common in nlp. during decoding, we employ beam search with a beam size of 8. all models were optimized with adamw @xcite . the learning rate was varied under a linear schedule with warmup steps of 5,000 and the maximum learning rate of 5e -5 . the model was training for 100,000 batches until achieving the best validation loss. the experiments were repeated for 5 times and were reported with their average results. all models were trained on a machine with nvidia tesla v100 gpu allocated with a batch size of 32 samples. baseline models we compare our approach with popular paraphrase generation methods including: (i) rediduallstm @xcite : an lstm sequence-to-sequence model using residuals between rnn layers; (ii) pointergenerator @xcite : an rnn sequence-to-sequence model using copy mechanism; (iii) transformer @xcite : the vanilla transformer model; (iv) trans-former+copy: an enhanced transformer with copy mechanism @xcite ; and (v) dnpg @xcite : a popular paraphrase generation model based on transformer. dnpg extends transformer by generating paraphrases at multiple levels of granularity such as the phrase level and the sentence level. the model is composed of two encoderdecoder pairs, which correspond to phrase-level and sentence-level paraphrasing, respectively. we use the default settings of the baseline models as reported in their papers. (vi) fset @xcite : the state-of-the-art paraphrase generation model that retrieves a paraphrase pair similar to the input sentence from a pre-defined index, then editing it using the extracted relations between the retrieved pair of sentences. we directly report the performance from their original paper. ## related work this work is closely related to ( 1 ) multi-granularity paraphrase generation and (2) multi-granularity attention. ## conclusion in this paper, we have proposed a novel paraphrase generation model named c-dnpg for continuously decomposing sentences at different levels of granularity. c-dnpg extends the multi-head attention with a granularity head which neurally estimates continuous granularity level of each input token. to efficiently incorporate granularity into attentions, we propose two novel attention masks, namely, granularity-resonance mask and granularity-scope mask, to adjust the original attention weights. results on two paraphrase generation benchmarks show that c-dnpg remarkably outperforms baseline models in both quantitative and qualitative studies. in future work, we will investigate the effect of c-dnpg in pre-trained models and other nlp tasks.