Learning to Selectively Learn for Weakly Supervised Paraphrase Generation with Model-based Reinforcement Learning (2022)

Abstract: Paraphrase generation is an important language generation task attempting to interpret user intents and systematically generate new phrases of identical meanings to the given ones. However, the effectiveness of paraphrase generation is constrained by the access to the golden labeled data pairs where both the amount and the quality of the training data pairs are restricted. In this paper, we propose a new weakly supervised paraphrase generation approach that extends the success of a recent work that leverages reinforcement learning for effective model training with data selection. While data selection is privileged for the target task which has noisy data, developing a reinforced selective learning regime faces several unresolved challenges. In this paper, we carry on important discussions about the above problem and present a new model that could partially overcome the discussed issues with a model-based planning feature and a reward normalization feature. We perform extensive evaluation on four weakly supervised paraphrase generation tasks where the results show that our method could significantly improve the state-of-the-art performance on the evaluation datasets.

## introduction paraphrase generation is an important natural language generation task which aims to generate a target sentence that encapsulates the meaning of a given source sentence while conforming to the style of some desired exemplar. it plays an essential role in many real-world applications for natural language processing, such as semantic parsing @xcite , machine translation @xcite , recommend system @xcite and question answering @xcite @xcite . different from other controllable text generation tasks where golden labelled data pairs are accessible and often being readily available, for paraphrase generation tasks, large scale of parallel paraphrase samples are often extremely hard to collect because generating them would often consume extensive domain knowledge or the generation could hardly be standardized. therefore, the chance of performing supervised learning in real life scenarios would be considerably limited. to overcome the data unattainable issue, unsupervised and semi-supervised approaches have achieved growing attention in the recent decade. generally, the unsupervised approaches adopt sampling-based or editing-based techniques @xcite to remedy golden standard knowledge but they generally result in less coherent or controllable target phrases due to their lose of supervision. therefore, in our paper, we focus on weakly-supervised paraphrase generation which has demonstrated great effectiveness in many major natural language processing tasks @xcite . although weakly-supervised approaches have successfully pushed forward the state-of-theart performance standard for the language-based tasks, when employed for paraphrase generation, they still face the challenge of how to acquire highquality paired paraphrase data and therefore lead to noisy data pairs which might bring negative effect to the downstream task of training paraphrase generation models. to overcome the aforementioned challenge and effectively train paraphrase generation models from noisy and not equally informative paraphrase pairs, we adopt a learning to selectively learn approach. that is, a meta model is learned to select intuitive paraphrase pairs while eliminating the low quality ones. thus the paraphrase generation model which is jointly learned with the meta data selector model could achieve better performance through the carefully specified selective learning process. nonetheless, it is impossible for learning an effec-tive meta data selection policy to be a supervised learning task due to the missing of optimal target selection policy. to overcome this issue, we adopt a reinforcement learning-based approach to learn effective selection policy without supervised signal. to this end, we extend the success of previous reinforcement learning-based approach for data selection @xcite . however, formulating a markov decision process (mdp) for the paraphrase learning process is a non-trivial task. in previous works, several important parts of their mdp formulation, such as the design of reward signal, are in need of further investigation @xcite and there also lacks in depth discussion on the challenge of solving the reinforcement learning problem. in this paper, we are motivated to extend this important line of using reinforcement learning to perform selective learning in weakly-supervised paraphrase generation problems and thus overcoming the data unattainable issue. overall we present several key insights into formulating the mdp for the selective learning problem as well as developing a model-based reinforcement learning framework to effectively solving the mdp. this paper has three main contributions: selective learning refers to the case of selecting items, e.g., features or data points, to learn from among other items. it motivates many important fields in machine learning, such as active learning @xcite @xcite @xcite and robust machine learning @xcite @xcite . our work is motivated by the existing instance-wise active data/feature acquisition approaches. one typical example is the conventional linear model that poses sparsity inducing prior distribution to the model @xcite and thus actively selects important features to the model. recently, there also emerged approaches that adopt reinforcement learning to actively find optimal feature subsets @xcite @xcite . though such attempts have demonstrated certain efficacy in handling instance-wise feature selection, they only deal with non timeseries data in non nlp domains, while the focus of our work is to deal with noisy labeled pairs in paraphrase generation tasks. our work is mostly related to the instance-level active data acquisition approaches @xcite , which are mostly adopted under the circumstances of data efficient or cost-sensitive learning or when dealing with noisy data. yoon et al. (2020) and ding et al. (2021) are formulation-wise identical while yoon et al. (2020) is among the very first model for active data selective learning, whereas ding et al. (2021) applies it on the task of paraphrase generation. our work extends this important direction to perform selective learning but we formulate a new model-based reinforcement learning method which aims to overcome partial limitations for the existing work and is empirically proven to be more effective than it on all the experimental domains. ## reinforced selective learning for paraphrase generation we present the general formulation for reinforcement learning-based selective weakly-supervised paraphrase generation problems. ## mb-rpg: model-based reinforced paraphrase generation framework in this section, we introduce our proposed modelbased reinforced paraphrase generation framework (mb-rpg). the overview of mb-rpg is shown in figure 1 . the essences of mb-rpg can be summarised as two points. one, mb-rpg adopts model-based planning attempting to perform decision making based on multi-step lookahead. this way, we can address the first limitation, namely, the short-slightness brought by the conventional score-based approach, one-step look-ahead. two, mb-rpg adopts a reward scaling module to normalize the reward as a fixed distribution to overcome the stochasticity of reward distribution. ## experiments in this section, we present extensive empirical evaluation results on comparing our method with its various counterparts on four commonly used paraphrase generation datasets 1 . ## conclusion our work tackles an important problem of leveraging reinforcement learning-based selective learning techniques to effectively deal with the noisy label issue in paraphrase generation tasks. we introduce a model-based framework which performs planning to capture the long-term effects for efficient exploration so as to overcome the noisy short-term reward issue experienced by most of the existing approaches. we also propose an effective reward normalization approach which could normalize the noisy reward signal to a distribution with a fixed zero-mean. we demonstrate that our proposed method could outperform baseline approaches with significant margins on the testified domains. future work includes refining the state and reward terms in the markov decision process for better data valuation or feature selections. one ongoing work is to integrate mb-rpg into our previous feature exploration work in video recommendation @xcite to improve feature quality. another one is to employ mb-rpg in our rl-based coreference resolution to filter out irrelevant features @xcite . in addition, we could consider constructing the policy model upon alternative generator backbone. also, we could consider inferring the reward from different sources, such as incorporating auxiliary language models in the training. tatsunori b. hashimoto, kelvin guu, yonatan oren, and percy liang. 2018. a retrieve-and-edit framework for predicting structured outputs. in advances in neural information processing systems (neurips), pages 10073-10083, montréal, canada. dan hendrycks, mantas mazeika, duncan wilson, and kevin gimpel. 2018. using trusted data to train deep networks on labels corrupted by severe noise. in advances in neural information processing systems (neurips), pages 10477-10486, montréal, canada. matteo hessel, hubert soyer, lasse espeholt, wojciech czarnecki, simon schmitt, and hado van hasselt. 2019. multi-task deep reinforcement learning with popart. in proceedings of the thirty-third aaai conference on artificial intelligence (aaai), pages 3796-3803, honolulu, hi.