K now C omp Submission for WMT 23 Word-Level A uto C ompletion Task (2023)

Abstract: The NLP community has recently witnessed the success of Large Language Models (LLMs) across various Natural Language Processing (NLP) tasks. However, the potential of LLMs for word-level auto-completion in a multilingual context has not been thoroughly explored yet. To address this gap and benchmark the performance of LLMs, we propose an LLM-based system for the WMT23 Word-Level Auto-Completion (WLAC) task. Our system utilizes ChatGPT to represent LLMs and evaluates its performance in three translation directions: Chinese-English, German-English, and English-German. We also study the task under zero-shot and few-shot settings to assess the potential benefits of incorporating exemplars from the training set in guiding the LLM to perform the task. The results of our experiments show that, on average, our system attains a 29.8% accuracy on the test set. Further analyses reveal that LLMs struggle with WLAC in the zero-shot setting, but performance significantly improves with the help of additional exemplars, though some common errors still appear frequently. These findings have important implications for incorporating LLMs into computer-aided translation systems, as they can potentially enhance the quality of translations. Our codes for evaluation are available at https://github.com/ethanyiwu/WLAC.

## introduction recent advancements in machine translation, especially due to the development of transformers and pre-trained language models, have been significant @xcite @xcite . these methods have yielded impressive results in traditional sentencelevel translation tasks @xcite . however, challenges still exist that hinder the further progress of computer-aided translation (cat) @xcite systems. among various components that constitute cat, wordlevel autocompletion (wlac) stands out as a core function @xcite . as shown in figure 1 , wlac aims to suggest the correct word translation in the target language based on a sequence of human-typed characters and bidirectional context. while this task might seem straightforward for seasoned human translators, existing deep-learning approaches struggle to handle it effectively. this can be attributed to the fact that performant translation methods, which rely on pre-trained language models, cannot effectively interpret the typed sequence of characters as they are pre-trained at the token level. other related studies have either only considered the source contextualization @xcite or have been unable to handle multilingual translations effectively @xcite . previous works have demonstrated that transformer-based frameworks, when trained with carefully designed masking or context transformation strategies, can efficiently tackle this task @xcite . yet, these frameworks require extensive training, and their ability to transfer across different languages remains questionable. with the recent progress made by large language models (llms), such as chatgpt (openai, 2022), researchers have conducted extensive studies regarding their performances on various nlp tasks @xcite @xcite . these llms possess advantages, such as ease of deployment and robust multilingual reasoning ability, making them particularly suitable for tasks like wlac. however, a detailed study of llms' application in this context is lacking, and the systematic use of llms to assist cat, especially in performing wlac, remains unexplored. this paper addresses these gaps by introducing a system that employs llms, specifically chatgpt, for the wlac task. chatgpt has been chosen for its exceptional performance in general natural language tasks, negating the need for deploying or fine-tuning other models. our approach uses a prompt-engineering-based method to convert all wlac task inputs into comprehensible natural language sentences. following this, we synthesize exemplars from the existing training set to facilitate in-context learning with chatgpt @xcite . the generations are subsequently parsed using carefully crafted rules to yield the final "predictions" from chatgpt (section 3). we then conduct comprehensive experiments using our system, covering scenarios from zero-shot to fiveshots. our experimental results indicate that our system achieves an average accuracy of 29.8% on the testing set. through error analysis and case studies, we found that llms face challenges with wlac in the zero-shot setting and identified four common types of mistakes that can be particularly addressed in the future. however, chatgpt's performance significantly improves when provided with additional exemplars, highlighting the crucial role of in-context learning in tackling wlac for llms (section 4). we will make all codes publicly available upon acceptance of this paper. ## method figure 2 shows an overview of our framework, which consists of three steps: exemplar sampling, prompt design, and generation parsing. ## conclusions in conclusion, this paper presents a novel llmprompting system to address the wlac task. our findings demonstrate that llms are highly capable problem solvers and adept at learning in context for this particular task, albeit with performance that falls short of previous supervised learning systems. a detailed analysis uncovers several error types that contribute to the limited performance of chatgpt. therefore, we urge researchers to devote additional attention to the wlac task using llms.