S urrey AI 2023 Submission for the Quality Estimation Shared Task (2023)

Abstract: Quality Estimation (QE) systems are important in situations where it is necessary to assess the quality of translations, but there is no reference available. This paper describes the approach adopted by the SurreyAI team for addressing the Sentence-Level Direct Assessment shared task in WMT23. The proposed approach builds upon the TransQuest framework, exploring various autoencoder pre-trained language models within the MonoTransQuest architecture using single and ensemble settings. The autoencoder pre-trained language models employed in the proposed systems are XLMV, InfoXLM-large, and XLMR-large. The evaluation utilizes Spearman and Pearson correlation coefficients, assessing the relationship between machine-predicted quality scores and human judgments for 5 language pairs (English-Gujarati, English-Hindi, English-Marathi, English-Tamil and English-Telugu). The MonoTQ-InfoXLM-large approach emerges as a robust strategy, surpassing all other individual models proposed in this study by significantly improving over the baseline for the majority of the language pairs.

## introduction the primary objective of quality estimation (qe) systems is to assess the quality of a translation without relying on a reference translation. this make qe valuable within translation processes, as it enables the determination of whether an automatically generated translation is sufficiently accurate for a specific purpose. this aids in deciding whether the translation can be used as is, requires human intervention for full translation, or necessitates post-editing by a human translator @xcite . quality estimation can be conducted across various levels: word/phrase level, sentence level and document level. this paper considers only the sentence-level qe and presents our participation in the wmt23 sentencelevel direct assessment (da) shared task. in the context of this task, participating systems are required to predict the da score for a given (source, target) pair. this score serves as a measure of the translation quality. building upon the ideas presented in transquest by @xcite , our investigation explores the use of various pre-trained models within the monotransquest architecture for the sentence-level quality estimation shared task. the architecture employs autoencoder pre-trained language models to fine-tune the qe data to predict a score which indicates the quality of translation. using the monotransquest architecture as the base we employ the pre-trained transformers separately to implement the systems monotq-xlmv, monotq-infoxlm-large and monotq-xlmr-large. in addition, we propose ensembletq which combines the output of monotransquest when using different pre-trained models. all the proposed systems achieve a significantly higher spearman correlation score compared to the baseline. the paper is structured as follows. section 2 briefly presents related work on quality estimation. section 3 provides a concise overview of the dataset used in the sentence-level qe shared task. moving on to section 4, we introduce the autoencoder pre-trained language models and proposed systems and detail the training methodology. section 5 is dedicated to the evaluation and section 6 comprises the result and discussion. the paper concludes by summarizing findings, highlighting conclusions, and suggesting potential avenues for future research in the final section. ## related work quality estimation in machine translation has evolved significantly throughout the years. initially, it relied on feature engineering and conventional machine learning techniques like svm and basic neural networks @xcite . however, neural networks has since become central to quality estimation, where there is no more need of feature engineering, and the models can be trained directly on the data @xcite . recently, transformer-based architectures have arisen as robust solutions for machine translation and quality estimation. notably, there are two widely recognized frameworks that leverage this transformative approach for qe tasks: transquest @xcite and @xcite . ensemble methods have also been explored extensively in quality estimation tasks @xcite @xcite @xcite . the ensemble approach from @xcite using k-folds consistently outperformed the standard method, underscoring the prevalent belief that ensemble strategies enhance performance outcomes. however, some of the research studies @xcite @xcite show that combining multi-lingual models through ensembling yields better results than the traditional k-fold ensemble technique. @xcite suggest an alternative ensemble method that merges the results from models trained using various sentence-level metrics. our study delves into the performance of cuttingedge pre-trained transformer-based approaches when applied to sentence-level quality estimation tasks. ## dataset we focus on sentence-level direct assessment tasks which comprise datasets for 5 language pairs which has english on the source side and indian languages on the target side: english-gujarati (en-gu), english-hindi (en-hi), english-marathi (en-mr), english-tamil (en-ta) and english-telugu (en-te). among these language pairs, en-hi language pair is considered mid-resourced and all the other language pairs are low-resourced. each language pair includes around 7,000 sentence pairs in the training set, as well as around 1,000 sentence pairs in both the development and testing sets. each translation was evaluated by three professional translators who assigned a score between 0 and 100. these direct assessment (da) scores were normalized using the z-score. the final score for the sentence-level task requires predicting the mean da z-scores for the test sentence pairs. more details on this can be found in @xcite . ## methodology this section outlines the approach taken to formulate our quality estimation techniques. we begin by detailing the autoencoder pre-trained language models employed in our architecture. then we explain the architecture and the strategy employed to train these network architectures in detail. ## evaluation in this section, we outline the evaluation outcomes of our models. we assess the performance of the proposed models under two circumstances: single model configuration and ensembletq. the primary evaluation criterion employed was spearman's rank correlation coefficient @xcite , which is a statistical measure used to evaluate the strength and direction of association between two variables. also, we have calculated the pearson correlation coefficient @xcite as a secondary metric for the evaluation. in the context of quality estimation (qe) for machine translation, it is used to evaluate the correlation between the machine-predicted quality scores and the gold standard labels provided by human annotators in the test dataset. spearman's rank correlation coefficient assesses the monotonic relationship between the two variables, unlike the pearson correlation @xcite , which measures the linear relationship between two variables. it is calculated by first ranking the values of both variables in ascending or descending order and then computing the pearson correlation coefficient between the two sets of ranks. spearman's rank correlation coefficient is often preferred because it is less sensitive to outliers and non-linear relationships between the predicted scores and human scores. ## result and discussion the research is divided into two distinct settings, as outlined in sections 5.1 and 5.2. the primary evaluation metric employed in this study is the spearman correlation coefficient. as shown in from our experiment results, as shown in the conducted experiments across midresourced and low-resourced language pairs unravel intricate performance dynamics among various models. ## conclusion this paper comprehensively evaluates the proposed architecture within the context of sentence-level direct quality assessment, employing diverse encoder-based pre-trained models. our investigation notably highlights the enhanced performance attributed to the monotq-infoxlm-large, which surpasses the other configuration approaches, namely monotq-xlmv, ensembletq strategy and monotq-xlmr-large. while our outcomes in the wmt23 sentence-level direct assessment task did not attain peak performance, they nevertheless exhibited a marked improvement over the baseline and showed notable performance scores close to the winning systems. looking ahead, our research trajectory anticipates a continued exploration of quality estimation employing large language models. this involves further experimentation encompassing a broader spectrum of low-resourced language pairs. these forthcoming endeavours aspire to deepen our insights into the intricacies of direct quality assessment and contribute to advancing the frontiers of natural language processing. also, we are focused on continuing the experimentation of pre-trained language models incorporated into different qe frameworks.