Performance Evaluation on Human-Machine Teaming Augmented Machine Translation Enabled by GPT -4 (2023)

Abstract: Translation has been modeled as a multiple-phase process where pre-editing analyses guide meaning transfer and interlingual restructure. Present-day machine translation (MT) tools provide no means for source text analyses. Generative AI with Large language modeling (LLM), equipped with prompt engineering and fine-tuning capabilities, can enable augmented MT solutions by explicitly including AI or human generated analyses/instruction, and/or human-generated reference translation as pre-editing or interactive inputs. Using an English-to-Chinese translation piece that had been carefully studied during a translator slam event, Four types of translation outputs on 20 text segments were evaluated: human-generated translation, Google Translate MT, instruction-augmented MT using GPT4-LLM, and Human-Machine-Teaming (HMT)-augmented translation based on both human reference translation and instruction using GPT4-LLM. While human translation had the best performance, both augmented MT approaches performed better than un-augmented MT. The HMT-augmented MT performed better than instruction-augmented MT because it combined the guidance and knowledge provided by both human reference translation and style instruction. However, since it is unrealistic to generate sentence-by-sentence human translation as MT input, better approaches to HMT-augmented MT need to be invented. The evaluation showed that generative AI with LLM can enable new MT workflow facilitating pre-editing analyses and interactive restructuring and achieving better performance.

## introduction the core of machine translation (mt) is the automation of the full translation process @xcite . the degree of mt automation can vary: fully automatic machine translation, human-aided machine translation, or machineaided human translation @xcite ). nearly all mt systems currently in use rely on the assistance of human operators. human involvement can come in three different modes: (1) pre-editing the source text, (2) interaction between system and human operator, or (3) post-editing. the dominant mt work model at present days is to use human translators as post-editors (correction/revision) on mt output. dominant mt tools (e.g., google translate, bing microsoft translator, or deepl) convert text from a source language into an equivalent passage in the target language, but they do not provide any pre-editing analysis results as guiding instructions or any interaction mechanisms between the human and mt system. translation has been modeled as a multiplephase process where pre-editing analyses guide meaning transfer and interlingual restructure. there are many differing theories that conceptualize different phases of the translation process. nida's translation theory of dynamic equivalence @xcite @xcite divides the translation process into three phases (illustrated in figure ): (1) analysis (based on the source language characteristics), (2) transfer (between the source and target language), and (3) restructuring (based on the target language characteristics). the analysis phase reflects the understanding and interpretation that can be used to guide the transfer and restructuring phases. present-day mt tools do not provide explicit means for users to analyze the source language, which would then be used to guide the transferring and restructuring process. another translation process model @xcite ) also has three phases: (1) recognize the problem (how to say x in target language?), (2) generate many alternative solutions, (3) select one solution (out of many). present-day mt tools do not provide explicit alternative solutions and let users select a solution; nor do they provide explanations as to why the final output it provides is more accurate than the alternatives. @xcite pointed out that a transfer architecture for mt included three separate stages: present-day mt tools do not provide mechanisms for pre-editing and human-machine interaction during the mt process. in @xcite , the human translation process was studied using eye tracking. in the end, three phases were identified: (1) preparation phase during which initial document reading was performed and translation strategy was chosen. (2) drafting phase. (3) revision phase. furthermore, different translators were found to have different styles: large versus small context planners (who read more or less content before they made strategy choices or started to translate), backtracker versus non-backtrackers (who did or did not frequently go back to revise), and various orientation styles: systematic, skimming, or head start without orientation. present-day mt tools do not provide mechanisms to support these translation style variations. following the post-editing only work model, since human experts cannot provide pre-editing and interactive guidance to mt system through the many mechanisms described above, achieving high-quality translations is challenging and timeconsuming, leading to inaccurate or unnatural results. the solution to the aforementioned limiting factors could come in the form of generative large language models (llm) such as gpt4, which could be the game changer that enables all or most mechanisms described above ( gpt models can achieve very competitive highquality translations for high resource languages, even though they were not originally designed to be used as mt tools. one of the unique features of gpt models is the prompting engineering @xcite , where precise and context-specific instructions or queries-known as prompts-can be crafted to elicit desired responses from language models. prompts guide the model and help shape its behavior and output. in this study, we focus on evaluating the approach to provide pre-editing or interactive guidance to mt using prompting engineering. using an english-to-chinese translation piece that had been carefully analyzed and discussed during a translator slam event, a human-generated chinese translation of an english story ( https://www.thecut.com/2020/03/bookexcerpt-samantha-irbys-wow-no-thankyou.html) was compared with multiple chinese translations generated using gpt4 and related plugin software (illustrated in figure ) following the steps below: unlike "traditional" mt tools, gpt4 based augmented mt can fully reflect pre-editing preparation, recognition, analyses, and humanmachine interaction by explicitly including ai and human-generated text analyses and humangenerated guidance or translation in the prompt. using 20 translation instances, two human translators assessed the pros and cons of the two augmented mt approaches using gpt4 over human-generated translation and an un-augmented mt using google translate. ## human translation slam event translation slam events have been organized by the american translators associations (ata) to have two translators go head-to-head to display their works on the same translation source text so the two translators have an opportunity to glimpse each other's work on the same content. audiences can also hear their discussions and comments during the event. in june 2023, an english-to-chinese translation slam was hosted by ata's chinese language division (cld) featuring two professional translators: sijin xian and arthur wan. they competed to translate a challenging english text into chinese and compare notes on their techniques and choices during the discussion session (see the original english text and a compilation of the two translations in the appendix). their comments on each other's work were recorded to discover how each translator analyzes, translates, and revises their translations. based on the detailed discussion during the slam event, sijin's translation focused phases of translation process models generative ai llm features supporting pre editing and human machine interaction pre-editing interaction post-editing (sager 1994) -prompting engineering (e.g. openai/gpt or google bard) can represent analysis results, transfer guidelines/strategies using instructions and few-shot examples. -multiple drafts that are distinct from each other can be generated to support multi-version selection and hmt (e.g., google bard alternative drafts feature). -llm-based chatbox (e.g. openai/bard) enables human-machine interaction. -larger and complex context can be addressed through llm finetuning. -… analysis transfer ## writing style analysis by human experts and gpt4 during the translation slam event, the two human translators discussed the writing style of the english writer and their approaches to reflect related styles in their chinese translations. in addition, chatgpt plus's link reader plugin user (prompt) please read the story at https://www.thecut.com/2020/03/book excerpt samantha irbys wow no thank you.html and make a list of the author's writing style. chatgpt (used link reader) based on the excerpt from samantha irby's essay collection "wow, no thank you.", here are some characteristics of her writing style: 1. humorous and self deprecating: irby often uses humor and self deprecation to engage her readers. she makes fun of herself and her situations, which adds a light hearted tone. 2. conversational tone: her writing style is very conversational, feeling personal and relatable. 3. ## conclusion generative ai with large language modeling (llm), equipped with prompt engineering and fine-tuning capabilities, can enable augmented mt solutions by explicitly including ai or human generated analyses, and/or human-generated reference translation as pre-editing or interactive inputs. using the english-to-chinese translation piece, we evaluated translation outputs on 20 text segments using human-generated translation, instances evaluators' ratings and reasons 1 (1) n/a (-1) wrong register level 2 (-1) incoherent meaning (1) n/a 3 (1) free translation captures the intended tone (2) n/a 4 (1) free translation captures the intended tone (2) n/a 5 (2) free translation captures the intended tone (1) good translation with the sentences re-organized 6 (0) n/a (2) good translation. exactly what means. 7 (2) free translation captures the intended tone (2) good translation. exactly what it means. 8 (1) n/a (2) good word choice 9 (0) n/a (1) n/a 10 (1) free translation captures the intended tone (1) free translation selects a word that is different from the source, but is a smooth word that can be used here 11 (0) n/a (1) n/a 12 (1) free translation captures the intended tone (1) n/a 13-16 (-1) n/a (1) n/a 17 (2) free translation is appealing (2) n/a 18 (1) free translation is appealing (2) reflects true meaning and adopt a source word seamless into the target 19 (1) free translation is simplified and appealing (0) n/a 20 (-1) direct translation but incorrect (2) smoother than other translations google translate mt, instruction-augmented mt using gpt4-llm, and human-machine-teaming (hmt) augmented translation based on both human reference translation and style instruction using gpt4-llm. the evaluation results show that while human translation had the best performance, both augmented mt approaches performed better than mt without augmentation. the hmt-augmented mt performed better than instruction-augmented mt because it combined the guidance and knowledge from both human reference translation and ai-generated style instruction. however, since it is unrealistic to generate sentence-by-sentence human translation as input to mt, better approaches to hmt-augmented mt need to be studied further. the evaluation showed that generative ai with llm can enable new mt workflow facilitating pre-editing analyses and interactive restructuring and achieving better performance. you don't have to cry for me, but listen: trying to make new friends as an adult is the hardest thing i have ever attempted. harder than multiple colonoscopi es? yes. harder than listening to the dentist pry my tooth bone away from my jawbone while i lie there wide awake? also yes! when i moved to kalamazoo from chicago, i thought for sure that i was going to be happy staying at home and never going outside. and, for the most part, i am. i get to travel and work in fancy cities with mass transit and ethiopian food, then come back and pay $1.87 for a gallon of gas for the car that i can park anywhere on my sprawling 2,000 acres of land that were practically free. okay, i'm exaggerating, but my point is fuck the city. 1 8 7 2000 1.87 2000 29 but, how does one make friends without an office to go to? or a club to participate in? or various pta meetings to grimace at each other through? are you just supposed to walk up to an interestinglooking person on the street and ask them to be your friend? i don't know if this is some kind of reverse profiling, but i can usually glance at a person and know at first sight that we're probably going to get along. i don't have it down to a science (i'm not researching shit, dude), but here are some dead giveaways: they have interesting, alternative, "cool person" hair, dumb tattoos, or are carrying a book, multiplied --" / " by a factor of ten if it happens to be one i wrote (i'm sorry -i am an egomaniac). --have you ever considered what a friendship is, or what any of your current friendships are, and thought about how to present that to a prospective new friend? you know, like how you are going to eventually be sending them selfies of you trying on 12 similar-yetslightlydifferent pairs of glasses in your ophthalmol ogist's waiting room while your garbage insurance is being processed? how do you convince a stranger to give you their real email when you are going to litter their gmail.com 12 gmail 12 gmail.com with dumb nonsense? a few years ago (before i moved to michigan and joined my wife's community of backyard composters and travelsoccer chauffeurs), my lady and i went to her friend's costume wedding and -i know you already know this, but let me just say it for anyone who is new or still has a shred of hopefulness in their heart -i did not wear a costume. the last time i wore a halloween costume was in the secondgrade costume parade at lincoln elementary school in 1986 (go, dolphins), and the only costume i could come up with was "housewife," a concept i didn't fully understand but thought i could --1986 " " -----1986 " " --approximate with my mom's tattered old robe, a halfmelted spatula, and the satin cap she slept in the nights after a fresh press 'n' curl. because i was a very large son, everyone just thought i had worn my shitty pajamas to school. " " anyway, my wife and i were at this wedding where i knew a few people well enough to say "hi" but not well enough to say "hi, ," and as we're sitting in our assigned dinner seats i'm looking around the room, taking it all in, wondering who i can latch on to in the hopes of a semiperman ent relationship once i officially moved to kalamazoo. a dude named ike " " " " " " " " "xx "