Dual-Stage Multi-Task Syntax-Oriented Pre-Training for Syntactically Controlled Paraphrase Generation (2024)

Abstract: Syntactically Controlled Paraphrase Generation (SCPG), which aims at generating sentences having syntactic structures resembling given exemplars, is attracting more research efforts in recent years. We took an empirical survey on previous SCPG datasets and methods and found three tacitly approved while seldom mentioned intrinsic shortcomings/trade-offs in terms of data obtaining, task formulation, and pre-training strategies. As a mitigation to these shortcomings, we proposed a novel Dual-Stage Multi-Task (DSMT) pre-training scheme, involving a series of structure-oriented and syntax-oriented tasks, which, in our opinion, gives sequential text models the ability of com-prehending intrinsically non-sequential structures like Linearized Constituency Trees (LCTs), understanding the underlying syntactics, and even generating them by parsing sentences. We performed further pre-training of the popular T5 model on these novel tasks and fine-tuned the trained model on every possible variant of SCPG task in literature, finding that our models significantly outperformed (up to 10+ BLEU-4) previous state-of-the-art methods. Finally, we carried out ablation studies which demonstrated the effectiveness of our DSMT methods and emphasized on the SCPG performance gains compared to vanilla T5 models, especially on hard samples or under few-shot settings.

## introduction given a source sentence 𝑋, (e.g., 'how exciting that is.'), and a template sentence 𝑌 (e.g., 'that's exactly right'), the goal of syntactically controlled paraphrase generation (scpg) is to produce a paraphrase sentence 𝑍 whose syntactic structure stays as close to the 𝑌 as possible, while retaining 𝑋's original semantics (e.g., 'that's really exciting.'). it is a simple yet crucial task, beneficial to a great many nlp tasks, such as neural machine translation @xcite , abstractive text summarization @xcite , dialogue generation @xcite , as well as data augmentation @xcite and improving model robustness @xcite . in principle, supervised learning for scpg task requires a set of (𝑋, 𝑌, 𝑍) triplets. practically, it's possible to obtain large-scale (𝑋, 𝑍) pairwise datasets by means of neural machine translation inference @xcite , or various data-mining & web-crawling techniques @xcite @xcite . however, obtaining (𝑋, 𝑌, 𝑍) triplets constitutes a more delicate situation, usually involving heavy human workload or intolerable computational costs @xcite . as a compromise to this problem, preliminary scpg datasets @xcite use a great quantity of easy-to-gain (𝑋, 𝑍) pairs as training sets, together with a small portion of them delicately annotated and edited to form (𝑋, 𝑌, 𝑍) triplets, constituting the validation/test set. ## related work prior works about scpg can be categorized according to different perspectives. in terms of model structures, early practices of scpg such as @xcite utilize rnn (lstm, gru) or rnn-based vae models @xcite @xcite , while recent works mainly focus on randomly-initialized or pre-trained encoder-decoder transformers @xcite @xcite @xcite with reasonable parameter magnitudes. in terms of syntactic control, some of these vae approaches @xcite exploit syntactical latent variables while others use explicit constituency tree inputs. among those who feed models explicit constituency trees, the way that they input the trees also differs, with some of them linearize @xcite the constituency trees and directly feed them into seq2seq models just like sentences, while others design dedicated model structures such as tree-lstm (kumar et al., 2020) or tree-transformer @xcite for intensive tree modeling. this essentially constitutes the second issue as mentioned in section 1, in our opinion. moreover, apart from supervised scpg methods trained on parallel corpus, unsupervised methods on standalone sentences are also becoming a growing strand, which mainly achieve unsupervised training by separately encoding semantics and syntactics utilizing corresponding encoders and learn to reconstruct the sentences @xcite . these methods can adapt to scpg tasks directly by mixing source sentence's semantics and template sentence's syntactics. these methods are spiritly similar to early vae practices like @xcite which, in our opinion, lacks variety in terms of tasks and model structures. ## dual-stage multi-task syntax-oriented pre-training: overview the framework of our methods is shown in figure 3 . we base our work on t5 model @xcite and paranmt-50m @xcite dataset. our dsmt pre-training involves structureaware pre-training and syntax-aware pre-training. the former one focus on the structural aspects of understanding lcts, like understanding the topologies which bracketing sequences (as in figure 2 ) stand for, or inferring useful information such as heights or parent/sibling relationships from these sequences. the latter one focus on learning the meanings of different syntax tree constituents and understanding the co-relationship between constituency trees and sentences. like t5 pre-training, we also cast all tasks to text2text format by introducing task-specific prefixes. upon pre-training, we finevanilla t5 treeposition indexing position: 0, 1 tree: (<@xmath0: i am a student. (root (s (np(prp d) (vp (vbp am)(np (dt a)(nn student)))(..))) scpg source sentence: how exciting that is . template sentence: that is exactly right how exciting that is . <@xmath1. .))) <@xmath2... that's really exciting. <@xmath3. .))) that's really exciting. dual stage multi-task pre-training scpg structure-aware pre-training (stage 1) syntax-aware pre-training (stage 2) exemplar based scpg target based scpg ## stage 1: structure-aware pre-training in the following section, we'll be discussing the first stage which makes our model understand lct sequences topologically, namely the structureaware pre-training stage. ## stage 2: syntax-aware pre-training this section is about the second stage where our models learn the about lct themselves (like what syntax nodes like np/vp usually stand for), and the relationships between lcts and sentences (like a sentence's corresponding lct). we name it the syntax-aware pre-training stage. ## downstream task: constituency-based and sentence-based scpg a further pre-trained model is gained after dualstage multi-task pre-training. we call the model dual-stage multi-task pre-trained t5 (dsmt-t5). with respect to the two genres of previous supervised scpg methods mentioned in section 1, it's time to answer the following two questions: ## ablation studies in this section, we're dedicated to answer the following questions: for dsmt pre-training: for exemplar and target based scpg: experiment settings. to answer q1, we directly train t5 model on stage 2 syntax-aware tasks using the same multitask training settings as what is described in section 5. to answer q2, we fine-tune vanilla t5 model on stage 1 tasks individually, and fine-tune t5 model, which is already trained on stage 1, on stage 2 tasks individually. we train using the same quantities of data, steps of optimization and hyperparameters as dsmt training for each task for fair comparison. to answer q3, we directly fine-tune t5 model under target-based and scpg/s (exemplar-sentence-based) scpg settings as shown in section 6. we call it vanilla t5. we compare scpg performance of vanilla t5 and our dsmt-t5. ## conclusion and future work we found three issues of previous scpg works. we proposed a novel dual-stage multi-task pretraining framework which offers a possibility to apply pre-trained models on scpg tasks with no external structures or specifically-designed input/output schemes. upon resolving the three issues and proving the effectiveness of our pre-trained models on scpg tasks, we have two critical findings. first, we found that exemplar-based scpg exhibits better performance when trained on triplet dataset, or directly using sentential exemplars. this paved the way for subsequent exemplar-based scpg researchers and suggests them to pay more attention on how to better utilize natural language based syntactical constraints and dive deeper on model's understanding of syntactics of sentences. second, we found that transformer based seq2seq models may also have strong performance on tree comprehending tasks even with constituency trees fully linearized. this, to our best knowledge, is the first solid step on exploring the extent for seq2seq models to understand and generate lcts, appealing for various syntax-intensive downstream applications, and also appealing further constituency parsing works to focus not only on working under cky framework and taking neural models as a neural alternative of pcfg rules, but also focus on the networks themselves and explore various seq2seq solutions.