Generating subject-matter expertise assessment questions with GPT -4: a medical translation use-case (2024)

Abstract: This paper examines the suitability of a large language model (LLM), GPT-4, for generating multiple choice questions (MCQs) aimed at assessing subject matter expertise (SME) in the domain of medical translation. The main objective of these questions is to model the skills of potential subject matter experts in a human-in-the-loop machine translation (MT) flow, to ensure that tasks are matched to the individuals with the right skill profile. The investigation was conducted at Unbabel, an artificial intelligence-powered human translation platform. Two medical translation experts evaluated the GPT-4-generated questions and answers, one focusing on English–European Portuguese, and the other on English–German. We present a methodology for creating prompts to elicit high-quality GPT-4 outputs for this use case, as well as for designing evaluation scorecards for human review of such output. Our findings suggest that GPT-4 has the potential to generate suitable items for subject matter expertise tests, providing a more efficient approach compared to relying solely on humans. Furthermore, we propose recommendations for future research to build on our approach and refine the quality of the outputs generated by LLMs.

## introduction this work presents an approach for developing an assessment tool to evaluate the subject-matter expertise (sme) of professional translators in the field of medical translation, using the large language model gpt-4. as mt becomes more predominant in translation scenarios, including specialized fields, the need for skilled experts who can identify and address quality concerns in mtgenerated output is proportionately increasing. specialized translators require sme, which involves extensive knowledge and proficiency in a specialized domain in both the source and target languages of the relevant language pair. to measure and evaluate sme effectively, a high-quality test for translators should incorporate translation questions that assess the translator's proficiency in the target language, as well as questions that evaluate their domain-specific expertise in the source language from which they are translating @xcite . evaluating the sme of potential experts in the loop across different language pairs and domains poses challenges. implementing a system that uses sme tests to pre-screen and match subject-matter experts with mt texts in the same subject matter could improve quality in a human-in-the-loop flow. however, when done entirely by humans, creating and maintaining a comprehensive and upto-date question bank for a wide array of language pairs and domains can become expensive and timeconsuming. other challenges are listed in section 5. this paper proposes a methodology for automating the creation of sme tests using gpt-4, focusing on the field of medical translation, and the english-european portuguese and english-german language pairs. opting for a multiple-choice questionnaire format allowed for the automation of both test generation and test grading, which increases the speed and scalability of the assessment process, while decreasing the costs. apart from possessing high levels of objectivity, mcq tests facilitate a fast and effective assessment process, while also being able to cover a broad range of topics @xcite . our main objective with the question banks is to distinguish between experts and non-experts, not to evaluate levels of expertise among the expert test-takers, although that might constitute a future point of research (see section 5 for insights on future work). we conducted a quality assessment of the generated questions and sets of answers, using an evaluation scorecard structured around five evaluation criteria (detailed in section 2), which was completed by medical expert translators. our analysis of the results provides insights into gpt-4's capability in generating high-quality medical expertise test items for english-european portuguese and english-german. all the data related to this work, including question banks, evaluation and prompts, is available on github. foot_0 ## study setup our plan for implementing sme tests as part of unbabel's framework for matching human experts to translation tasks involves generating different assessments for each test-taker, by randomly choosing a set number of questions from a large question bank. as such, our approach centered on compiling a large question bank generated by gpt-4, rather than individual tests, for each language pair. for each language pair, we invited a medical expert translator with more than ten years of professional experience to evaluate the generated questions and answers. the evaluators had no connection to the study and were paid according to their hourly rates with no time restrictions to conduct the experiment. ## results after generating the question banks, the next step was assessing and scoring their quality. section 3.1 presents the overall score, results and considerations for english-european portuguese, conducted by evaluator a, while section 3.2 does the same for english-german, conducted by evaluator b. ## limitations despite its preliminary positive findings, this study presents several limitations. firstly, the topics chosen for the question banks only represent a very small portion of medical knowledge, and only address a few of the most commonly translated medical documents. as mentioned in section 4, reproducing the study with different and perhaps less common language-pair combinations is likely to produce different results. we hypothesize that the less common the language-pair combination, the lower the quality achieved by gpt-4 . dac @xcite state that there is a decrease in performance for languages other than english in natural language processing (nlp) tasks, which might be verified in the use-case of mcq automation.the same can be said for less common language varieties. additionally, the evaluation of the gpt-4 output was done by only one expert per language pair. the sample size evaluated by each expert was substantial (200 question stems and 200 sets of answer choices), but extending the evaluation process to more experts can strengthen the validity of the results. furthermore, the ontology of subject matter expertise is vastly complex and multifaceted @xcite and this paper does not intend to claim that the measurement of subject matter expertise can be fully judged by the results of mcq assessments. the resulting mcqs of this study are tailored for specific use-cases, not to measure competency in general. they are also designed to be part of a larger assessment process, in which other specific tasks (such as reviewing a specialized machine translated text, for instance) contribute to a more accurate representation of the expertise level of the human-inthe-loop. what follows are recommendations for improving the relevance of the output from an llm for the use case presented in this paper: generating sme test items to be ultimately used as an optimization method of task assignment in expertin-the-loop translation flows. they can also be adapted for other contexts, mt related or otherwise. these sme test items might be relevant, for instance, in traditional translation workflows, research surveys to gather data, or in businesses, in the context of assessments and job interviews. another relevant use case is the integration of the generated test items in self-directed learning methodologies @xcite , either in corporate contexts or within freelance translator and reviewer training.