Improving the Naturalness and Diversity of Referring Expression Generation models using Minimum Risk Training (2020)

Abstract: In this paper we consider the problem of optimizing neural Referring Expression Generation (REG) models with sequence level objectives. Recently reinforcement learning (RL) techniques have been adopted to train deep end-to-end systems to directly optimize sequence-level objectives. However, there are two issues associated with RL training: (1) effectively applying RL is challenging, and (2) the generated sentences lack in diversity and naturalness due to deficiencies in the generated word distribution, smaller vocabulary size, and repetitiveness of frequent words and phrases. To alleviate these issues, we propose a novel strategy for training REG models, using minimum risk training (MRT) with maximum likelihood estimation (MLE) and we show that our approach outperforms RL w.r.t naturalness and diversity of the output. Specifically, our approach achieves an increase in CIDEr scores between 23%-57% in two datasets. We further demonstrate the robustness of the proposed method through a detailed comparison with different REG models.

## introduction referring expression generation (reg) aims at generating utterances that help anchoring an object within an image. such descriptions are called referring expressions (res) @xcite . early work focused on datasets with relatively simple visual stimuli @xcite @xcite utilizing synthesized images of objects in artificial scenes. the recently released datasets ref-clef, refcoco, refcoco+ and refcocog @xcite @xcite which contain natural images of cluttered scenes, led to a surge of interest in using deep neural networks for reg. such approaches utilize the encoder-decoder paradigm originally proposed for machine translation @xcite and since have been widely used to various other nlg sub-fields @xcite @xcite @xcite . the encoderdecoder model consists of a deep convolutional neural network (cnn) @xcite to encode the visual features into a fixed-size latent representation, and a variation of recurrent neural network (rnn) @xcite , e.g. a long short term memory (lstm) @xcite network to generate the output. the encoder-decoder model is typically trained to maximize the likelihood of a word given the history of generated words so far. this training approach is referred to as "teacher-forcing" @xcite . although intuitive to train a model on token-level, during generation a model is evaluated based on its ability to optimize towards sequence level metrics resulting in a discrepancy between training and testing objectives. furthermore, a second problem that stems from "teacher-forcing" is that during training, the model uses the groundtruth words to predict the next one, while during testing uses its own predictions. this missmatch, coined as exposure bias @xcite , results in error accumulation during generation. recently reinforcement learning (rl) @xcite techniques have been adopted to alleviate the exposure bias problem and directly optimize the non-differentiable task specific metrics. for instance, @xcite propose a method that builds upon the reinforce algorithm to directly optimize the non-differential test metrics and reports promising results in machine translation, while @xcite utilizes an actor-critic method, that involves the training of an additional value network to normalize the reward. however, training with rl is a non-trivial task due to a number of limitations: (1) high variance of the gradient @xcite ; (2) lack of per-token advantage, i.e. the reinforce algorithm makes the assumption that every token contributes equally to the whole sequence @xcite ; and (3) reward configuration @xcite . furthermore, effectively applying rl to reg has not been explored, with the exception of @xcite who incorporate an additional module to reward discriminative res by updating the speaker with a policy gradient algorithm. however, little is reported of how the rl was configured. to the best of our knowledge, this is the first work to thoroughly propose how to effectively train reg models with rl. furthermore, beside the aforementioned limitations of rl methods there is another problem that is often overlooked. while directly optimizing the evaluation metrics one can achieve higher scores, the generated text lacks diversity due to repeated ngrams @xcite . our analysis shows that rl trained models are strongly biased towards frequent res leading to smaller vocabulary and deficiencies in the generated word distribution. to address these issues we propose the use of minimum risk training (mrt) @xcite as an alternative way of optimizing reg systems on sequence level. minimum risk training aims at minimizing the expected loss over training data by taking automatic evaluation metrics into consideration. the mrt objective has the following advantages over mle. first, it can directly optimize sequence level objectives that are not necessarily differentiable. second, while mle maximizes the likelihood of the training data, mrt introduces a notion of ranking amongst candidate sequences by discriminating between sequences. thus, by minimizing the risk, we expect to find a distribution that approximates well the ground-truth distribution. furthermore, the mrt objective is similar to the reinforce algorithm in a sense that both maximize an expected reward or cost. however, there are two fundamental advantages of the mrt over rl: (1) the reinforce algorithm typically utilizes one sample in order to approximate the expectation, whereas the mrt objective considers multiple sequences making it sample and data sufficient; and (2) the mrt objective intuitively estimates the expected risk over a set of candidate sequences, whereas the reinforce algorithm typically relies on the baseline reward to determine effectively the sign of the gradient. therefore, our main contributions are as follows: firstly, we conduct an extensive analysis and benchmarking of rl training strategies for reg, by exploring how different aspects such as the reward and the baseline reward configuration affect reg models (section 8.1). our experiments reveal how to best train reg models using reinforcement learning. secondly, we show that models optimised for cider also achieve higher scores in all other metrics (bleu etc.) even when compared to models directly optimised on them. although our rl approach outperforms the state-of-art, rl still suffers from the limitations discussed earlier. therefore, we propose a novel training strategy for reg which combines mrt with mle and we show its effectiveness in comparison to a number of rl training strategies w.r.t naturalness, diversity and informativeness (section 8.2). our approach achieves improvements between 33.5%-38.7% and 23.4%-57.8% in terms of cider on refcoco and refcoco+ respectively compared to previously proposed approaches. finally, a detailed analysis shows that when a reg model is trained with the proposed approach, uses a larger vocabulary, produces longer referring expressions and generates more uni-grams and bi-grams. ## related work early work in referring expression generation can be dated back to the early 1970s @xcite . the traditional view of reg is a two step procedure where the reg model accounts for the content selection and determination of the referential form @xcite . however, the large body of work in reg focuses on the determination of content for definite descriptions @xcite . algorithms such as the full brevity and the incremental algorithm @xcite have as foundation the gricean maxims @xcite , that provide insights of how people behave in different communication scenarios @xcite . recently due to the availability of larger and more complex natural image datasets, such as ref-coco @xcite there is a surge of interest in applying deep learning methods. neural reg approaches rely on incorporating contextual information by using visual features, appearance attributes @xcite , location features @xcite and global image features as target object representation. in their seminal work, @xcite use a convolutional neural network to extract visual features and an lstm to generate the expression trained on maximum mutual information objective. @xcite propose a unified framework where a speaker module generates res, a listener module comprehends res, and a reinforcer module provides guidance towards informative res. lastly, @xcite examine the impact that variations of beam search have in the length of res. although there are not published attempts on optimizing neural reg systems on a sequence level, we will review a number of works from the wider field of natural language generation. ranzato et al. (2016) were the first to adopt the reinforce algorithm in order to optimize the encoder-decoder model. the discovery that baselines can effectively reduce the variance of the gradient estimation led to a significant body of work in nlg. @xcite used fully connected layers to predict the baseline and used monte carlo rollouts to approximate the state-action value. @xcite utilize an actor-critic framework and combine it with temporal difference learning. the state-action value was modelled by a separate rnn. @xcite propose the utilization of the output of the model at the test time to normalize the reward. although mrt has a long history in training linear model for structured predictions, it has only be used in neural machine translation @xcite as an alternative to mle training. in this work, however, we apply mrt to reg as an alternative to rl and we compare the output of those two training strategies in terms of naturalness and diversity. ## reg model as this work focuses primarily on the training objectives for neural reg models, we adopt a standard encoder-decoder architecture language model similar to @xcite . the encoder is a cnn network that extracts the representation of the target object. then this representation is embedded through a linear projection layer w i . the words are represented as one-hot vector, projected to the same space as the visual representation through a linear embedding layer. the start of each sequence is denoted by a special bos token, while the special stop token eos denotes the end of the sequence. the decoder, which is responsible for the generation of res is modeled as an lstm network. the image features are used only as an input to @xmath0. then, at each time step t, its output depends on the previously generated words and on the hidden state, which encodes the knowledge of the observed input up to this time step. the parameters θ of the model are learned by maximizing the likelihood of the observed sequence. specifically, given n training pairs the training objective is defined as: ## training reg with reinforcement learning the generation process can be cast into a reinforcement learning process as first described in @xcite . within the classic reinforcement learning paradigm, an agent performs an action under a specific policy π. the nature of the policy is application dependent. within reg the language model can be seen as an agent that interacts with its environment (i.e. the previously generated words and the visual features at each time step t). the parameters θ of the agent define a policy π θ . the agent selects an action, which is a candidate token from the vocabulary under the policy, until it generates the eos token. once the agent reaches the end of the sequence it observes a terminal reward r, which is the score for generating a re ŷn given an object o and a ground truth referring expression (or a set of referring expressions) y. the reward is a scalar produced by any evaluation metric such as cider. therefore, the training aims at parameterizing the agent in order to maximize the reward as follows: an inherent challenge of the reinforce algorithm is that typically leads to highly unstable training due to the noise in gradient estimation and reward computation @xcite . thus, in the next sections we explore a number of methods that have been proposed in literature that stabilize training. specifically, for reward computation we study: (1) how to sample the candidate samples; (2) which reward function to use; and ## minimum risk training for referring expression generation beside the aforementioned problems, there are two other limitations that are often overlooked. first, while these methods can directly optimize the non-differentiable rewards and improve the performance of evaluation metrics, the generated text suffers from lack of diversity due to repetition of common n-grams. the second limitation is that the approximation of the reward is based on one sample which is data and sample inefficient. to address these limitations we explore a principled alternative to the reinforce algorithm, the minimum risk training @xcite . minimum risk training (mrt) minimizes the value of a given task-specific cost function, i.e. risk, over the training data at sequence level. specifically, let x denote a fixed-size representation of the input, then the set y(x (s) ) denotes the set of all possible referring expressions generated by the model with parameters θ. for a given candidate sequence y and ground truth referring expression y, mrt defines a cost function ∆(y , y) which is the semantic distance between y and the standard y. the cost function can be any function that captures the discrepancy between the model's prediction and the ground truth. formally, the objective function of mrt is the following: the mrt objective minimizes the expected value of a cost function which enables us to optimize reg models with respect to specific evaluation metrics of the task. in this work we explore the use of various reg evaluation metrics such as cider and bleu and combination of those. furthermore, for the construction of the subset of the candidate sequences we consider online setting, specifically we regenerate the candidate set for each training sample. again we consider random sampling and beam search as search strategies (see section 4.1). moreover, we also considered offline generation, that is the candidate sequences are generated before training and never refreshed. however, we found that it leads to inferior performance and thus was not included. ## combined objectives we also experiment with combining the mle training objective either rl or mrt. the motivation of the loss combination is to maintain good tokenlevel accuracy while optimizing on the sequencelevel. in other words, using an evaluation metric as a reward can suppress the probability of the words that do not increase the metric score, and thus concentrate the distribution to a single point. thus, we explore a combined objective in order to scale the peakiness of the output distribution. specifically, the weighted combination of mle (equation 1 ) with rl objective (equation 4 ) is defined as follows: equivalently, combing the mrt objective (equation 6 ) with mle we have: ## conclusion in this work we considered the problem of optimizing referring expression generation models with sequence level objectives. specifically, we firstly provide a comprehensive comparison of different aspects of configuring reg models with rl training. we found that (1) random sampling is a better search strategy than beam search; (2) we showed that using random sampling with self-critical training improves cider scores; (3) incorporating reward shaping improves the performance; (4) we showed that combining rl objectives with mle is beneficial to the training, resulting in higher cider scores and diversity. however, there is a considerable gap between mle and rl methods w.r.t. to diversity. thus, as an alternative to rl we proposed the use of minimum risk training. we showed that mrt combined with mle produces superior results in terms of naturalness and diversity of the referring expressions compared to both mle and rl training. while we have focused on analyzing the performance of the presented models with automated evaluation metrics, we intend to further verify these results in a human evaluation.