MMT ’s Submission for the WMT 2023 Quality Estimation Shared Task (2023)

Abstract: This paper presents our submission to the WMT 2023 Quality Estimation (QE) shared task 1 (sentence-level subtask). We propose a straightforward training data augmentation approach aimed at improving the correlation between QE model predictions and human quality assessments. Utilising eleven data augmentation approaches and six distinct language pairs, we systematically create augmented training sets by individually applying each method to the original training set of each respective language pair. By evaluating the performance gap between the model before and after training on the augmented dataset, as measured on the development set, we assess the effectiveness of each augmentation method. Experimental results reveal that synonym replacement via the Paraphrase Database (PPDB) yields the most substantial performance boost for language pairs English-German, English-Marathi and English-Gujarati, while for the remaining language pairs, methods such as contextual word embeddings-based words insertion, back translation, and direct paraphrasing prove to be more effective. Training the model on a more diverse and larger set of samples does confer further performance improvements for certain language pairs, albeit to a marginal extent, and this phenomenon is not universally applicable. At the time of submission, we select the model trained on the augmented dataset constructed using the respective most effective method to generate predictions for the test set in each language pair, except for the English-German. Despite not being highly competitive, our system consistently surpasses the baseline performance on most language pairs and secures a third-place ranking in the English-Marathi.

## introduction quality estimation (qe) strives to assess the output of machine translation (mt) systems without the availability of a reference translation of known high quality @xcite @xcite . this capability serves as a valuable asset for expediting and cost-effectively facilitating the evaluation phases throughout the development cycle of mt systems. in this paper, we describe our contribution to the qe shared task at the eighth conference on machine translation (wmt23). we participate in the task 1 of the shared task and we specifically focus on the sentence-level subtask, which centers on predicting the quality score of neural mt outputs at the sentence level without access to reference translations. our study encompasses six language pairs: english-german (en-de), english-marathi (en-mr), english-hindi (en-hi), english-tamil (en-ta), english-telegu (en-te), english-gujarati (en-gu), with annotations derived in two different ways: multi-dimensional quality metrics (mqm) @xcite and direct assessments (da) @xcite . participating systems are assigned the task of predicting the quality score (mqm or da) for each source-target sentence pair, and their performance is evaluated using spearman's rank correlation coefficient as the primary metric, supplemented by the kendall and pearson coefficients as secondary metrics for assessment. our approach investigates the potential to enhance the performance of qe models by exposing them to a diverse range of training examples. to this end, we identify eleven different data augmentation methods and apply each of them individually to augment the training set for each language pair. our results reveal that, for most language pairs, these methods result in varying degrees of performance improvement, with the most effective methods being synonym substitution using the ppdb, words insertion guided by contextual word embeddings, back-translation, and direct paraphrasing. we also show that for some language pairs, it is feasible to further enhance the model's performance by training it on an augmented set formed through the amalgamation of part or all of the said augmentation methods; however, the extent of improvement remains constrained. for each language pair except english-german, we generate predictions utilising the model trained on the augmented dataset constructed through the respective most effective method. although our submission may not be considered highly competitive, they consistently achieve significantly improved performance compared to the organisers' baseline for the majority of language pairs. notably, for the english-marathi pair, our submission ranks third place with the spearman score of 0.650. this observation indicates that the training data augmentation approach may hold particular promise and offer advantages when applied to the english-marathi language pair. ## methodology as mentioned above, we identified a total of eleven distinct data augmentation methods, as detailed in ## experiments in this section, we describe our experimental settings, present the results achieved on the develop- ## conclusion in this paper, we proposed a training data augmentation approach to the wmt 2023 sentence-level qe shared task. we systematically identified eleven various data augmentation methods and applied each of them individually on the source-side sentences to generate augmented training samples for the six studied language pairs. the experimental results demonstrated that in most cases, these methods can enhance the correlation between the predictions of the qe model and human-provided quality scores to varying degrees, albeit not to a significant extent. in addition, we show that training the model on the augmented set, generated through the combination of these methods, contributed further to performance enhancement, although this phenomenon was not universally observed and the degree of improvement was at a negligible level. our methodology yielded a third-ranking outcome for english-marathi and a fifth-place ranking for other da annotated language pairs, among the submissions from the six teams. in terms of future work, we intend to explore other more effective augmentation approaches and extend our study to encompass a more diverse array of language pairs and qe models.