Perceptions of Educators on MTQA Curriculum and Instruction (2024)

Abstract: This paper reports the preliminary resultsof a survey aimed at identifying and ex-ploring the attitudes and recommendationsof machine translation quality assessment(MTQA) educators. Drawing upon ele-ments from the literature on MTQA teach-ing, the survey explores themes that maypose a challenge or lead to successful im-plementation of human evaluation, as theliterature shows that there has not beenenough design and reporting. Results show educators’ awareness ofthe topic, awareness stemming from therecommendations of the literature on MTevaluation, and reports new challenges andissues.

## introduction academia and industry continuously make efforts to assess the quality of machine translation (mt) systems @xcite , typically using automatic evaluation metrics (aem) or human evaluation (he) @xcite , each approach possessing its own strengths and weaknesses. however, to evaluate an mt system with detailed and actionable results, it is vital to use a balanced approach incorporating he in the process in conjunction with aems @xcite . in particular, the inclusion of he must be carefully employed so as to not generate hyperbolic reports of the capabilities of mt systems in particular scenarios such as in @xcite . some studies have recommended more rigorous he design principles @xcite ) not only to dampen hype, but also to identify systems' weaknesses through an analysis of complex linguistic phenomena @xcite . while it is not recommended to rely solely on aem-based evaluations @xcite , the literature shows a common tendency to rely on aems without he @xcite in the mt community. it is understood that mt use must consider the purpose and value of translations and the expected longevity of the content @xcite , which extends to mt evaluation as well @xcite . in this manner, risks from mt systems such as grammatical errors or inappropriate words/constructions @xcite , biases in the output @xcite , which can be dangerous for specific domains such as legal and medical @xcite , can be prevented with rigorous he incorporation in mt evaluations. given these risks and the responsibility of implementing a careful evaluation, complementing automatic with he is essential to ensure ai technology is safe, beneficial and fair @xcite . it can be achieved with ethical behaviours adopted by engineers and technology developers @xcite , which can be further refined with the training of stakeholders themselves @xcite . thus, this paper focuses on the instructional training of mt quality assessment (mtqa), as part of a doctoral study that intends to create and provide training in he for natural language processing (nlp) master's students. in this paper, we report results from the qualitative findings of a survey aimed at mtqa educators with both ts and nlp educators. it inquired about the educators' attitudes and recommendations regarding he in mtqa teaching, exploring where he can be positioned pedagogically, what he content should be prioritised, and evaluates the practical considerations that may facilitate or hinder the incorporation of he into an mtqa curriculum focused on nlp students. the survey explores the following key questions: these findings can inform mtqa trainers and curriculum planners in making informed decisions to foster appropriate he teaching and deployment, and consequently, its use in mtqa. ## related work translation quality assessment (tqa) is complex, leading to much debate and different definitions of translation quality, especially in regard to translation technologies, such as mt @xcite . mtqa in translation studies (ts) curricula has been slowly introduced alongside the use of mt @xcite from a curricular standpoint @xcite to critically use and assess mt @xcite . technical aspects of mt also became an element of mtqa teaching, such as building an engine @xcite , mainly with the intent of empowering trainee translators to understand how the systems work in order to facilitate informed decisions when evaluating the output @xcite . studies have shown that translators in training gain mtqa proficiency through error analysis @xcite , and that translators' ability to identify missing contextual information in mt output and select appropriate language for specific domains is crucial @xcite . this mirrors evaluation models used in the industry @xcite , showcasing academia's efforts to prepare translators. accordingly, aem and other measures of he have been introduced in the classroom in the ts field. @xcite and @xcite introduced adequacy and fluency measures in conjunction with error typologies. @xcite created a platform on which students could rank mt outputs and generate bleu scores @xcite , focusing on the correlation of human judgement with the aem. other platforms were used in classroom settings, such as the asiya-online toolkit @xcite , which provided automatic scores, and later, mutnmt @xcite for guided building and evaluation of nmt systems. @xcite proposed jupyter notebooks to introduce translators to the technical nature of aems while generating different scores such as bleu, meteor (banerjee and lavie, 2005), chrf3 @xcite , ter @xcite and bertscore @xcite . @xcite demonstrate a case study of teaching mtqa, by using he through ranking, adequacy and fluency measures, correlating to aems provided by mateo @xcite , a platform that generates bleu, chrf, bertscore, bleurt @xcite and comet @xcite scores. these studies demonstrate the effort to introduce different evaluation approaches in the translation classroom, and how important accessible evaluation platforms are for training translators. the importance of integrating mtqa into curricula is underscored by the concept of mt literacy @xcite which entails understanding the operational mechanisms of mt systems to facilitate their use. @xcite and @xcite echo the importance of mt literacy in equipping professionals to use and evaluate mt effectively. however, the implementation of training is context-based, the pedagogical guiding principles for mtqa education tend not to be structured. in the context of nlp education, the few studies that mention mtqa do so only to a minor degree @xcite @xcite . this is due to mt being only one component within the broader spectrum of training, with evaluation assuming a secondary role. however, that does not diminish the importance of evaluation in nlp, as the reasons for its lack of implementation in training may due to absence of space in the curriculum and the lack of structured information on evaluation @xcite . as such, organising the insights and recommendations of mtqa educators, both from nlp and ts may lead to fostering mtqa education. ## methods to collect information regarding educators' insights and suggestions on mtqa, an online sur-vey was designed (approved by the university's research ethics committee, with reference dcu-fhss-2023-015). ## data results and analysis as this is an ongoing study, the results reported in this paper are qualitative and small-scale, with the intention of being exploratory, to explore possible relationships and patterns @xcite . while it is known that smaller samples are not ideal for generalisations @xcite , the qualitative components may inform better the results of the survey as it reaches a largerscale @xcite . data was visualised on qualtrics, which affords analysis of both closed-ended and openended questions. for the closed-ended questions, qualtrics automatically created graphs based on the responses to form variables, and the platform allowed a degree of customisation to change the colour of graphs and combine/separate variables (or groups) as needed. for the open-ended questions, qualtrics lists the responses by variables (or groups), allowing an interpretive qualitative analysis of the data. ## final considerations and future challenges by comparing the two groups, it can be seen that their attitudes and difficulties reflect both contextual factors of their teaching, and needs commonly associated with their profession. for ts educators, there has been an increasing effort to integrate the newest technological advancements into their teaching while still maintaining the critical approach of their use. ts educators focus on teaching mtqa for translation trainees in order to foster their mt literacy, either for more proficient use when performing pe or to prepare them to serve as consultants in the development of mt systems. for either, it places ts educators and the future translators in a position to ensure a safer use of translation systems. nlp/cl educators tend to place more attention towards the ethics, and regard the design of mt evaluation among the most chosen topics, which can be performed by translators who can serve as experts on this process. we have seen in section 4.5 that the technical aspect may present different pedagogical challenges for ts educators q8, since teaching technical elements to a non-technical audience requires accessible resources. therefore, there has been research done focused on the experience of translators performing pe, and evaluating mt systems. as a result, over the past years platforms such as mut-nmt and mateo are paramount to make aspects of evaluation accessible, especially when teaching aems. accordingly, for ts lecturers who reported faculty expertise and development as a pedagogical difficulty, those platforms are an important resource for educators. the nlp/cl group reports other difficulties with mtqa, primarily in finding room in the curricular structure to focus on evaluation (section (4.5, q8). it is worth noting that this group does not recognise either type of evaluation as less important. in fact, the survey shows that nlp/cl educators recognise the importance of he in mtqa and teach different evaluation metrics to different groups according to their profiles and roles in the evaluation process. due to the amount of technical content in development to be covered, it has been suggested by nlp/cl educators that the most appropriate way to cover evaluation would be through intensive, interactive, hands-on workshops to practise different aspects of evaluation -either the design planning, different approaches or the annotation. however, based on the results, nlp/cl educators appear to suggest that nlp master's students who are choosing to work on mt development and evaluation should know the basic approaches and should still place translators at the centre of the evaluation. these results show the efforts of the mt community at demonstrating the importance of every stakeholder in the mtqa process -from developer to evaluator. the design and implementation of mtqa still brings challenges (section 4.5), but ts, nlp and cl educators report it is essential, whether you are training translators or developers. challenges to overcome may include: as observed in section 4.3, (q4), the survey also provided some insights on what constitutes a comprehensive evaluation of mt, demonstrating the awareness of the educators. this survey shines light on the directions of mtqa education according to educators from different fields. we hope the insights and recommendations presented here can aid the mt community in fostering mtqa education. appendix a. full questionnaire questions in bold are the ones selected for this paper. how many hours do you spend teaching per week? move the slider according to the amount of hours. -(close-ended) • q7 -what are your other main work activities? -(close-ended) • q8 -in your current teaching role, how much influence do you have over the curriculum, including changes to the syllabus and teaching methods? -(close-ended) • q9 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q10 -what type of students do you work with, mostly? -(close-ended) • q11 -at what academic levels do you currently teach? please select all that apply. -(close-ended) • q12 -what modality/modalities do you deliver training in? -(close-ended) • q13 -have you taught mt quality assessment before? -(close-ended) • q14 -please rate the significance of incorporating human evaluation into the development of mt systems. rate on a scale of 1 to 5. -(close-ended) • q15 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q16 (in the survey, q3) -in your opinion, what trends do you foresee in evaluation metrics that incorporate human judgment for mt systems? select all that apply. -(close-ended) • q17 (in the survey, q4) -in your view, what constitutes a comprehensive evaluation of an mt system? please describe the key components or criteria that should be included. -(open-ended) • q18 (in the survey, q2) -what types of mt evaluation do you teach? -(close-ended) • q19 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q20 -do you teach evaluation for nlp tasks (e.g. summarisation, speech recognition, sentiment analysis) other than mt? -(closeended) • q21 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q22 -how many years have you been teaching mt quality assessment? move the slider according to the amount of years. -(closeended) • q23 -assess the importance of teaching students how to plan evaluations for mt systems in your academic curriculum. please rate the importance of integrating evaluation planning as part of the academic curriculum for mt quality assessment. -(close-ended) • q24 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q25 (in the survey, q5) -assess the importance of including evaluation metrics in your academic curriculum. please evaluate the importance of integrating evaluation metrics into the academic curriculum for mt quality assessment. you will be presented with two types of evaluation metrics. rate on a scale of 1 to 5. -(closeended) • q26 (in the survey, q6) -please, add any further comments or explanations for your previous answer here. -(open-ended) • q27 (in the survey, q8) -beyond content (such as human evaluation metrics or automatic evaluation metrics), what other ped-agogical aspects do you believe may be currently lacking in the teaching of mt quality assessment? please select all that apply. -(close-ended) • q28 -please, add any further comments or explanations for your previous answer here. -(open-ended) • q29 (in the survey, q7) -if you were to create a human evaluation module in mt quality assessment addressed to nlp students at master's level, what should be the main content? select as many as necessary. -(close-ended) • q30 -please, add any further comments or explanations for your previous answer. -(open-ended) • q31 (in the survey, q9) -if you were to create a human evaluation module in mt quality assessment addressed to nlp students at master's level, what would be the best format? -(close-ended) • q32 (in the survey, q10) -given your previous choice on the best format for a human evaluation module, what teaching modality would be most suitable? -(closeended) • q33 (in the survey, q11, in relation to q9 and q10) -please, add any further comments or explanations for your previous answers from q31 and q32 here. -(openended) • q34 -is there anything else you would like to add? -(open-ended) appendix b. full responses from q4 -in your view, what constitutes a comprehensive evaluation of an mt system? please describe the key components or criteria that should be included. judging "translation quality" through human judgements (usually produced "in vacuo") is clearly inferior to this approach.