Rewarding Semantic Similarity under Optimized Alignments for AMR -to-Text Generation (2022)

Abstract: A common way to combat exposure bias is by applying scores from evaluation metrics as rewards in reinforcement learning (RL). Metrics leveraging contextualized embeddings appear more flexible than their n-gram matching counterparts and thus ideal as training rewards. However, metrics such as BERTScore greedily align candidate and reference tokens, which can allow system outputs to receive excess credit relative to a reference. Furthermore, past approaches featuring semantic similarity rewards suffer from repetitive outputs and overfitting. We address these issues by proposing metrics that replace the greedy alignments in BERTScore with optimized ones. We compute them on a model’s trained token embeddings to prevent domain mismatch. Our model optimizing discrete alignment metrics consistently outperforms cross-entropy and BLEU reward baselines on AMR-to-text generation. In addition, we find that this approach enjoys stable training compared to a non-RL setting.

## introduction automatic evaluation metrics often score natural language generation (nlg) system outputs based on how well they lexically align to humanannotated references. in tasks such as machine translation and summarization, these metrics may unfairly penalize outputs that express the correct semantics despite a lower n-gram overlap with reference strings. as a result, models overfitting to certain token-level patterns may dominate those generating more creatively (e.g., through synonyms or varied sentence structure). nlg systems are typically trained to maximize likelihood of a single set of references. conditioning models on gold prefixes shields them from their own predictions during training-an issue known as exposure bias. adding reinforcement learning (rl) objectives @xcite can aid exploration by giving a model feedback on sequences sampled from its own distribution. however, it is common practice to use automatic evaluation scores like bleu @xcite and rouge @xcite as sequence-level rewards. this results in the same lack of semantic signal described earlier. instead of hinging evaluation on hard n-gram overlap, recent metrics @xcite rely on vector similarity between contextualized subword embeddings to make more semantically faithful judgments. bertscore, in particular f bert , computes a token-level f1 score based on greedy alignment of similar embeddings. with their strength in offline evaluation, it is natural to ask how these embeddings-based metrics can help provide more realistic training feedback. past approaches to train models with semantic similarity scores include both non-differentiable and differentiable objectives. @xcite separately train paraphrastic sentence embeddings that provide semantic similarity rewards to a neural machine translation (nmt) system. rewards were included in a mixed minimum risk and maximum likelihood training phase. besides an embedding training overhead, the model needed a length penalty term to limit repetitive outputs. @xcite adopt a similar fine-tuning approach using an rl objective with f bert for abstractive summarization. while their models were less repetitive, their news domain corpora may have been a natural match for bert embeddings. finally, jauregi @xcite also propose to optimize f bert but with fully differentiable training objectives in nmt. yet their models overfit after only a few epochs and scored lower in bleu at the cost of higher f bert . we hypothesize that metrics employing external pretrained vectors may suffer from domain mismatch with downstream data. this can hurt the accuracy of semantic similarity scores computed during training. in this work, we focus on text generation from abstract meaning representations (amrs, @xcite , sentence-level semantic graphs that are rooted, directed, and acyclic. this task's models may especially benefit from an emphasis on semantic rather than lexical similarity. it also provides a challenging setting to evaluate overfitting given the relatively small corpus size. in our analysis of f bert rewards, we note that f bert could worsen repetition and incomplete outputs in nlg systems. due to its greedy token alignment, f bert precision may assign extra credit to a reference token 'retrieved' multiple times. in response, we contribute the following. ## greedy token alignment the main insight behind bertscore and related metrics is to align hypothesis and reference tokens using their pairwise vector similarity scores. these alignments are later used to weight the contribution of token-level similarity scores towards a final sequence-level score. concretely, given (ŷ 1 , . . . , ŷm ) and (y 1 , . . . , y k ) hypothesis and reference token embeddings, precision in f bert is ## optimized token alignment aligning tokens between hypothesis and reference can be seen as an assignment problem, where a token pair (ŷ i , y j ) is highly weighted if it incurs low cost (i.e., distance). here, we describe discrete token matching (oneto-one) and soft alignment (one-to-many). for the latter, we extract alignments from the earth mover's distance @xcite transport matrix. we weight pairwise token similarities as in f bert using each of these two alignments to provide metrics f disc and f cont . ## semantic similarity rewards we propose to fine-tune on our optimized f1 metrics, applying a weighted average of cross-entropy and rl objectives. given source sequence x (e.g., a linearized amr), the former is computed as to encourage close evaluation scores between sampled ȳ and reference y, the rl objective is ## experiments we examine the performance of our proposed metrics as rl rewards on amr-to-text generation. ## conclusion this paper proposes new f1 score metrics based on optimized rather than greedy alignments between predicted and reference tokens. instead of letting hypotheses align to reference tokens without regard to their frequencies (and vice versa), we extract alignments as a constrained optimization problem. in the discrete case, we treat alignment as a matching problem between hypothesis and reference tokens. in the continuous case, we find alignments that minimize earth mover's distance between the two token embedding distributions. we apply new metrics as rewards during rlbased training for amr-to-text generation, with f disc outperforming both a cross-entropy baseline and one optimizing bleu rewards. despite being computed on a downstream model's token embeddings, the metrics still provide informative rewards during training without signs of overfitting.