DORB : Dynamically Optimizing Multiple Rewards with Bandits (2020)

Abstract: Policy gradients-based reinforcement learning has proven to be a promising approach for directly optimizing non-differentiable evaluation metrics for language generation tasks. However, optimizing for a specific metric reward leads to improvements in mostly that metric only, suggesting that the model is gaming the formulation of that metric in a particular way without often achieving real qualitative improvements. Hence, it is more beneficial to make the model optimize multiple diverse metric rewards jointly. While appealing, this is challenging because one needs to manually decide the importance and scaling weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit approach (DORB), where at each round, the bandit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierarchical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our approaches via various automatic metrics and human evaluation on two important NLG tasks: question generation and data-to-text generation. Finally, we present interpretable analyses of the learned bandit curriculum over the optimized rewards.

## introduction recent advancements in end-to-end neural networks-based approaches have shown wide success in various sequence generation tasks: machine translation @xcite , dialogue systems @xcite , textual summarization @xcite @xcite , image/video captioning @xcite @xcite , question generation @xcite @xcite , etc. in all of these tasks, cross-entropy loss optimization has been widely used as a standard optimization approach @xcite , but this approach suffers from exposure-bias issue @xcite and does not optimize for the non-differentiable automatic evaluation metrics that measure the quality of the generated sequence. recent introduction of policy gradient-based reinforcement learning approaches address these issues for sequence generation tasks by directly optimizing the non-differentiable evaluation metrics @xcite @xcite . however, optimizing for a particular metric/reward via policy gradient-based approaches often leads to improvement in mostly that specific metric, suggesting that this approach is gaming the metrics @xcite . the weighted average of multiple metrics or surrogate rewards have been explored @xcite , but these approaches have to deal with finding the optimal scale balance across different metrics. one can alternatively optimize multiple metrics via a mixing ratio @xcite , but this still needs careful tuning of the mixing ratio. moreover, all these reward approaches are fixed and do not change over training, and all the metrics may not be important over every stage of the training. thus, it might be useful to consider using a dynamic combination of metrics, which rewards to use early vs. later, or which rewards might be useful to come back later in training, and consider the context of the full history of rewards, as well as the models current state and the nature of the metric. to this end, we present a multi-armed bandit approach (which we name the dorb framework) where the arms of the bandit are the choices of the metrics that we want to optimize as rewards. at every round, the bandit chooses the next possible metric to optimize based on its previous performance history over these metrics, hence allowing the automatic learning of an optimal curriculum of rewards. we explore this approach in the context of exploration vs. exploitation via exp3 algorithm @xcite with two novel approaches for bandit rewards: (1) single multireward bandit (sm-bandit); (2) hierarchical multireward bandit (hm-bandit). first, we present a reward scaling approach to maintain the metric rewards range in [0, 1]. next, we present our sm-bandit, where at each round, the bandit's reward is based on the performance improvement from multiple sources. here, we use the average of all the scaled metric rewards from multiple sources as the final reward to the bandit. finally, we present our hm-bandit, which consists of a single firstlevel controller, as well as k second-level multiarmed bandits. the first-level controller's goal is to find the under-performing reward metric, while the second-level bandits' goal is to trigger the specific metric optimizer that will lead to a promising improvement in this specific metric. we validate the effectiveness of our approaches on two important generation tasks: question generation and data-to-text generation, via both automatic evaluation metrics and human evaluation. for question generation, we present results on the squad qg dataset @xcite , and for data-to-text nlg, we choose the webnlg dataset @xcite . we show that our bandit-based approaches perform statistically significantly better (based on human evaluation) than strong singlereward based rl models as well as non-bandits based multi-reward methods such as the multi-task approach of @xcite . we further present various interpretable analyses of our bandit progress and learned rewards curriculum over different bandit approaches. ## multi-reward optimization in this section, we first describe the policy gradients-based reinforcement learning (rl) approach for text generation tasks, and then discuss the need for a better multi-reward optimization approach for rl in the context of generation tasks. lastly, we introduce our novel methods for multireward optimization via multi-armed bandits. glossary: agent: rl policy gradients; bandit: multi-armed bandit; controller: controller in hm-bandit (see fig. 2 ). policy gradient background. cross-entropy loss based optimization is traditionally used for the sequence generation tasks. however, recent policy gradient-based reinforcement learning approach has shown two advantages over the crossentropy loss optimization approach: (1) avoiding exposure bias issue which is about the mismatch in the output distributions created by different train and test time decoding approaches in cross-entropy loss optimization; (2) able to directly optimize the non-differentiable evaluation metrics. to this end, reinforce algorithm @xcite is used to learn a policy p θ defined by the model parameters θ to predict the next action (tokens in our setup). specifically, instead of minimizing the negative log-likelihood, we minimize the following loss: there are several ways to calculate the baseline estimator, and in this work we use the scst mechanism @xcite . need for a better multi-reward optimization. often, an rl agent can improve the policy p θ via multiple reward sources. however, efficient ways of optimizing multiple rewards in a policy gradient-based reinforcement learning setup have been less explored. previous works have either explored using a weighted combination of multiple rewards @xcite or alternate fashion of optimizing multiple rewards inspired via multi-task learning setup @xcite . however, these approaches have a disadvantage of tuning the weights of the rewards combination or using a static tunable mixing ratio while optimizing in an alternate fashion. to this end, we explore multi-reward optimization via a multi-armed bandit approach @xcite @xcite . during the training, the bandit explores/exploits the choice of reward functions in order to improve the overall performance of the model. in the remaining part of this section, we discuss various multi-armed bandit-based models for multi-reward optimization (sec. 3.1), and reward settings (sec. 3.2). then, we present metric 3 metric 2 metric 1 multi armed bandit encoder decoder attention rl optimization validation performance pull arm (metrics) optimize the selected metric parameters update model reward (r1) (the selected metric) bandit reward (r2 @xcite ...during the age of enlightenment, philosophers such as john locke advocated the principle in their writings, whereas others, such as thomas hobbes ... who was an advocate of separation of powers? the two novel approaches, namely single multireward bandit (sm-bandit, sec. 3.3) and hierarchical multi-reward bandit (hm-bandit, sec. 3.4). ## tasks and setup we use question generation and data-to-text generation tasks in our experiments. in this section, we discuss the details on these two tasks along with the experimental setup. ## results and analysis in this section, we present the performance of previous work, our cross-entropy baselines, our rlbased baselines, and finally our multi-arm banditbased models. we start with results on automatic evaluation (sec. 5.1-5.2). next, we present results on human evaluation (sec. 5.3). finally, we present an interpretable analysis on the bandits (sec. 5.4). ## conclusion we presented novel approaches for dynamically optimizing multiple reward metrics simultaneously via multi-armed bandit approach in the context of language generation. we described two such mechanisms, namely single bandit and hierarchical bandit with multiple rewards. we conducted experiments on two challenging language generation tasks: question generation and data-to-text generation, and our method achieved strong improvements based on human evaluation over previous approaches. we further presented interpretable analysis on our bandit methods.