SURF : Semantic-level Unsupervised Reward Function for Machine Translation (2022)

Abstract: The performance of Reinforcement Learning (RL) for natural language tasks including Machine Translation (MT) is crucially dependent on the reward formulation. This is due to the intrinsic difficulty of the task in the high-dimensional discrete action space as well as the sparseness of the standard reward functions defined for limited set of ground-truth sequences biased towards singular lexical choices. To address this issue, we formulate SURF, a maximally dense semantic-level unsupervised reward function which mimics human evaluation by considering both sentence fluency and semantic similarity. We demonstrate the strong potential of SURF to leverage a family of Actor-Critic Transformer-based Architectures with synchronous and asynchronous multi-agent variants. To tackle the problem of large action-state spaces, each agent is equipped with unique exploration strategies, promoting diversity during its exploration of the hypothesis space. When BLEU scores are compared, our dense unsupervised reward outperforms the standard sparse reward by 2% on average for in- and out-of-domain settings.

## introduction reinforcement learning (rl) has shown promise in the field of text generation. this is mainly due to the fact that it allows the usage of nondifferentiable evaluation functions fit for the discrete natural language tasks. it also serves as a solution for bridging the gap between training and inference time regimes ("exposure bias") that arises from the fact that the model is never exposed to its own errors as only ground-truth labels are used to condition the generation during training @xcite . one of the essential components of the rl framework is the reward function, which is used to provide agents with indicative signals in terms of the effectiveness of chosen actions. the usage of rl in neural machine translation (nmt) and language generation however has been doubted largely due to the difficulty of exploration in the high-dimensional discrete action space combined with the sparse reward signal. the latter comes from the typical metrics used as rewards (e.g., bleu @xcite ). these rewards evaluate text in a shallow way by measuring the string similarity between generated and ground-truth sequences, making them extremely sparse and biased towards singular lexical choices. as the rl policy is usually initialised with some pre-trained distribution over words, suspicion has been raised that in this situation, those words already most likely gain probability mass regardless of the rewards @xcite . thus, the current sparse rewards are not beneficial for rigorous exploration of different words during training. recent studies suggest that the main benefit for nmt from rl is in performing domain adaptation when using proper hypothesis space exploration along with special emphasis on reward scaling and normalisation @xcite . to address the problem posed by sparse or biased rewards, we propose surf, a formulation of the unsupervised reward function that evaluates machine-generated texts in the semantic space by factoring in different qualitative aspects. furthermore, we introduce an additional scaling and normalisation mechanism which ensures fairness and uniformity of the reward function regardless of the complexity of the natural language task. our main contributions are thus threefold: (a) the proposal of surf, an unsupervised dense reward assessing both sentence fluency and adequacy (section 4). we demonstrate this reward leads to a translation quality favourably comparable to the traditional sparse bleu reward both in automatic and human evaluation; (b) the proposal of an additional normalisation using reward shaping mechanisms for the unsupervised reward; (c) demonstra-tion of the strong potential of the proposed reward to elicit benefits of various rl architectures. we experiment with multi-agent synchronous and asynchronous actor-critic (ac) architectures as applied to the problem of mt (section 3.3). each of the parallel agents in those architectures is trained using different segments of the training dataset which has their own unique exploration strategy. when bleu scores are compared, our dense unsupervised reward outperforms the standard sparse reward by 2% on average for both in-and out-of-domain settings. to the best of our knowledge, this formulation of the unsupervised reward for a range of multi-agent architectures is the first of its kind for mt. our datasets and settings are described in section 5, and results of our experiments are described in section 6. ## related work the following section describes the work related to ours in the subfields of machine translation and rl for language generation. reinforcement learning algorithms for neural machine translation reinforce @xcite algorithm and its variants have so far been the most widely used rl algorithms in mt @xcite @xcite . the fact that reinforce-based approaches suffer from high variance in general and in mt in particular has stimulated attempts to apply actor-critic (ac) models to the task. the first attempt of the kind was the one of @xcite . more advanced ac models with q-learning are rarely applied to language generation problems. however, there are exceptions (e.g., entropy-regularised ac models that promote exploration of actions @xcite ). this could be explained by the difficulty of approximating the q-function for large action space. in this work we explore a series of multi-agent ac architectures which to the best of our knowledge have never been applied to mt before. unsupervised rewards for language generation tasks recent work on unsupervised rewards in nlp has explored both dynamic @xcite and static rewards @xcite . for example, @xcite introduces a dynamic distribution over latent frequency classes as a reward signal. this distribution is shaped to promote more rare words in the policy search space. static rewards are very often designed to assess generated text in terms of its fluency and adequacy. fluency judgment assesses how a hypothesis satisfies the grammatical norms of a language. adequacy judgment assesses how well a hypothesis conveys the meaning of the source sentence. the recent research performs both evaluations as semantic similarity assessments using the pre-trained contextualised embeddings such as bert @xcite . for mt, semantic similarity assessment could be carried out using monolingual pre-trained embeddings against a reference, as in @xcite , or using multilingual pre-trained embeddings in the unsupervised reference-less approach by considering the semantic similarity to source sentences @xcite . we adopt the latter approach to measure adequacy. ## methodology we start by formulating mt using rl, then introduce the actor-critic architectures and the reward functions used. ## semantic-level unsupervised reward function (surf) our semantic-level unsupervised reward, surf, is based on two scores: sentence fluency and sentence-level semantic similarity (slss) @xcite . each score assesses translation quality of generated sequences from different aspects and is computed using a pre-trained model. to prevent reward sparsity, the reward function introduces a score normalisation mechanism which normalises scores of a generated sequence (from all timesteps) with respect to the score of its full target sequence. this subsequently yields an unsupervised reward function that is maximally dense. the sentence fluency score (f (ŷ 1:t )) is defined as the average log-likelihood of the generated sequence tokens (ŷ 1:t ) as defined by a pre-trained large lm. the slss score measures the overall semantic similarity between the entire generated sequence and its source sequence calculated as the cosine similarity between the two sentence cross-lingual embeddings. score normalisation from the rl perspective, the mt task does not define the environment component that the agent operates in. that is, unlike the classical rl setting where the environment is relatively fixed, the 'environment' in the mt task is mostly dependent on the source sequence, in terms of its sophistication, structure, length, etc. as a result, a valid and good translation of a source sequence would receive a relatively high score but is not directly comparable to other sequences due to the difference in source sentence complexity. therefore, it is important to ensure that the reward function is uniform and generalised across all source sentences. in order to do this for each source sequence, the reward function uses the corresponding target sequence as a 'soft' upper bound for what a machine-generated sequence could achieve. that is, for each of the two score metrics outlined above, the scores from all timesteps received by a generated sequence is normalised to the range 0 to 1 with respect to the score of the full target sequence. to demonstrate the normalisation method, let us consider the formulation below which uses the sentence fluency score metric as an example. given a pair of source (x) and target (y) sequences and a candidate sequence (ŷ 1:t ), the fluency scores from all timesteps of ŷ1:t would be {f (ŷ 1 ), f (ŷ 1:2 ), ..., f (ŷ 1:t-1 ), f (ŷ 1:t )} while the fluency score for the entire reference target sequence (y) would be f (y). using the fluency scores of the candidate and that of the reference, the normalised candidate scores can be calculated as follows: considering the example formulation above, one can observe that the normalisation with respect to the reference score is considered as a 'soft' upper bound as it allows for candidate scores to be higher than the reference score (i.e., allowing the possi-bility that candidate sequences can be better than their references). ## conclusion we have presented surf, a new unsupervised semantic-level reward function, efficiently addressing the reward sparsity issue and mimicking human evaluation by considering both sentence fluency and semantic similarity. we have explored this reward for a new family of actor-critic transformerbased architectures with synchronous and asynchronous variants that promote the exploration of the search space. we demonstrate that surf shows strong potential to elicit the benefits of various rl architectures. our results show that it outperforms the traditional sparse bleu reward for the same architectures in the automatic, semantic-level and human evaluation. our code is available at https://github.com/atomanu/surf . there are several directions to take our work further: we can investigate the utility of our reward for other architectures and we can also explore different sampling strategies for each of the agents of our multi-agent models. finally, we have investigated only two datasets to ensure comparability to the existing benchmarks. a more thorough investigation with more datasets and language pairs is needed to fully assess the scope of our contribution. ated sequence and its source sequence. following the methodology adopted in the sentsim evaluation metric @xcite , each token in the source and generated sequences is passed to a crosslingual language model to obtain its cross-lingual embedding. then, each token in the source sequence (x i ) is matched to a token in the generated sequence (ŷ j ) with the highest cosine similarity value to compute the recall. similarly, the precision value is computed by matching each token in the generated sequence to a token in the source sequence based on cosine similarity. as a results, the recall and precision of a generated sequence can be expressed as follows: r(ŷ 1:t , x) = 1 |x| x @xmath0:t x nxe i • ŷnxe j (27) p (ŷ 1:t , x) = 1 |ŷ 1:t | ŷj ∈ŷ 1:t max x @xmath1.e., x nxe @xmath2, respectively. using the precision and recall, the final tlss score is defined as the f1 measure. t lss(ŷ 1:t , x) = f (ŷ 1:t , x) the tlss scores are computed using the bertscore tool @xcite .