Prompt-based Explainable Quality Estimation for E nglish- M alayalam (2025)

Abstract: The aim of this project was to curate data for the English-Malayalam language pair for the tasks of Quality Estimation (QE) and Automatic Post-Editing (APE) of Machine Translation. Whilst the primary aim of the project was to create a dataset for a low-resource language pair, we plan to use this dataset to investigate different zero-shot and few-shot prompting strategies including chain-of-thought, towards a unified explainable QE-APE framework.

## introduction this project is a one-year-long initiative funded by the european association for machine translation (eamt) 1 . the primary focus of our project was to create novel quality estimation (qe) and automatic post-editing (ape) datasets for the english (en) -malayalam (ml) language pair. qe refers to the task of automatically predicting the quality of machine-translated output without reference translations, while ape aims to automatically correct errors in machine translations. the malayalam language is a low-resource language with over 38 million speakers across the world. despite its presence with 86,553 wikipedia articles 2 on the web, there was no available data for evaluating the quality of translation from english to malayalam. for english to low-resource indic language pairs, qe data exists for english to {hindi, marathi, and gujarati} where target languages belong to the indo-aryan language family. however, from the dravidian language family, qe data is only available for english-tamil and english-telugu @xcite . our project addresses this gap by introducing a novel english-malayalam qe dataset, expanding qe research within the dravidian language family. the dataset comprises three direct assessment (da) scores assigned by human annotators, along with human post-edited translations for the ape task. the manual post-editing process further facilitates the creation of word-level qe data foot_0 , enriching its usability for fine-grained evaluation. additionally, the project aimed at a comprehensive evaluation of multiple large language models (llms) for qe of low-resource language pairs. ## project progress & impact figure 1 depicts our workflow as described below. our workflow consisted of-filtering high-quality parallel data, machine translation, ter-based validation, and human annotation for qe & ape.