NUBIA : N e U ral Based Interchangeability Assessor for Text Generation (2020)

Abstract: We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA showing competitive performance with stateof-the art metrics used to evaluate machine translation and state-of-the art results for image captions quality evaluation. In addition to strong performance, NUBIA models have the advantage of being modular and improve in synergy with advances in text generation models.

## introduction evaluation metrics play a central role in the machine learning community. they direct research efforts and define the state of-the-art models. unlike machine learning tasks such as classification and regression, text generation (i.e. machine translation, summarization, image captioning) is a nuanced task where the gold standard for quality evaluation is human assessment. however, this method of evaluation is expensive and time consuming. as a complement, automatic metrics were designed to approximate human judgment of quality. a consequence of this unique setup is that the metrics themselves have to be frequently upgraded to reflect the dynamic progress of the field. however this has not happened and, while the text generation models have dynamically evolved, the metrics most commonly to assess model outputs used have not. the two most common metrics used for evaluating similarity between candidate and reference texts are bleu (bilingual evaluation under-study) @xcite and rouge (recall-oriented understudy for gisting evaluation) @xcite . both approaches rely on counting the matching n-grams in the candidate text to n-grams in the reference text. the former is precision focused while the latter is recall focused. these metrics have posed serious limitations and have already been criticized by the academic community @xcite @xcite . in this work, we propose a methodology to build text generation evaluation metrics using deep learning models as core components. an implementation of this methodology is then presented and tested in the domains of machine translation and image captioning quality estimation. for assessing the metric in the machine translation domain, we use the wmt 2017, 2018 and 2019 dataset. we conduct further experiments showing that, without any additional fine-tuning, the same model used to assess machine translation quality outperforms existing metrics specifically designed to assess image captioning quality. beyond the promise of this methodology in terms of its ability to lead to metrics with high correlation to human judgment, nubia metrics can be constructed with any base architecture, perform well with only thousands of examples as supervision signal and are expected to improve continuously with future nlp advances. ## nubia model our method has three modules: a neural feature extractor, an aggregator and a calibrator. the feature extractor tested in this paper consists of different transformer architectures fine-tuned on relevant tasks of language evaluation such as semantic similarity, logical inference and sentence likelihood. while we use these features and architectures as the main building blocks of nubia, the specific models can change as long as they maintain the necessary performance in terms of correlation with human judgment on the fine-tuning tasks. the aggregator uses the features extracted by the transformers as well as non-neural features such as reference and candidate sentence length and is trained to predict the quality of the hypothesis sentence given the reference sentence. similar to the wmt challenge, we use past years' data to train this aggregator and test it on the test subset. the calibrator is the final module that caps all predictions to be between 0 and 1. ## experiments to assess our proposed implementation, we used both direct assessment and segment-level relative ranking from different wmt metrics shared tasks @xcite as well as tasks from image captioning. we did not conduct experiments in the domain of machine summarization because there are no labeled datasets containing pairs of summaries and their corresponding human evaluations of the summary quality. in the wmt direct assessment task, candidate and reference translations are given for several language pairs and for each candidate translation, 15 human evaluators assign a quality score between 0 and 100. the final human score is taken as the average of the 15 human assessments. the performance of metrics is assessed using pearson correlation with human judgement. for this task, we used the 2017 dataset because, unlike the wmt 2018 and wmt 2019 dataset, each sentence has been scored by at least 15 human evaluators @xcite . for relative ranking, wmt 2018 and wmt 2019 still use direct human assessments but since there is not at least 15 annotators per sentence pairs, the direct assessment correlation task is converted into relative ranking task. more specifically, for a given reference sentences, up to 5 machine translation systems generate candidate translations. these candidate sentences are rated by human annotators on a discrete 0-25-50-75-100 points scale. after averaging the human annotations, if the gap between two candidate translation is higher than 25 points, one translation is considered to be better than the other. when the gap between two candidate sentences is lower than 25 points, the sentence pairs are not included in the segment-level evaluation @xcite . in that setting, metrics are scored on their ability to preserve the human ranking using the kendall's tau correlation coefficient. ## results in in table-3, we report the results for the relative ranking test of wmt 2018. here we see that nu-bia is only outperformed by bleur. in table -4 we report the results of the image captioning experiments in table -1. here we observe that nu-bia outperforms all existing methods and achieves state-of-the-art correlation with human judgment of caption quality. the strong performance maintained across varied tasks is a strong indicator of the robustness of this methodology and shows its promise to generalize well beyond the training set.