A Human Perspective on GPT -4 Translations: Analysing F aroese to E nglish News and Blog Text Translations (2024)

Abstract: This study investigates the potential of Generative Pre-trained Transformer models, specifically GPT-4, to generate machine translation resources for the low-resource language, Faroese. Given the scarcity of high-quality, human-translated data for such languages, Large Language Models’ capabilities to produce native-sounding text offer a practical solution. This approach is particularly valuable for generating paired translation examples where one is in natural, authentic Faroese as opposed to traditional approaches that went from English to Faroese, addressing a common limitation in such approaches. By creating such a synthetic parallel dataset and evaluating it through the Multidimensional Quality Metrics framework, this research assesses the translation quality offered by GPT-4. The findings reveal GPT-4’s strengths in general translation tasks, while also highlighting its limitations in capturing cultural nuances.

## introduction in the past decade, the field of natural language processing (nlp) has seen a dramatic shift with the introduction of the attention mechanism and transformer models, profoundly influencing the domain of machine translation (mt) @xcite . one of the foremost challenges in mt is the scarcity of highquality, human-translated data for low-resource languages. however, large language models (llms) such as the generative pre-trained transformer (gpt) models may present a solution to this challenge. these models are trained on vast amounts of data and have an impressive ability to generate native-sounding text, which they do by adapting based on the context presented in their training material (in-context learning) @xcite . transformer models, such as gpt, are trained on multilingual data and have zeroshot translation capabilities which enables them to translate low-resource languages as well as high-resource languages. therefore, this shift towards in-context learning signifies a breakthrough in nlp, where human-quality translation pairs can be generated without the input of a human translator. this has the potential of lowering the cost of making such data and improving the scalability of such an operation, which is vital for making smaller and more cost-effective models for mt. this shift is particularly evident in the realm of mt datasets, where the gap between highresource and low-resource languages remains a critical challenge. in the past, common methods to synthesize data for mt datasets were based on backtranslation @xcite @xcite . however, the quality of gpt models indicate that it is a better choice for synthesizing mt datasets @xcite . the release of gpt-4 in march 2023 marked a significant milestone, with @xcite 's pilot study that demonstrated its enhanced translation abilities in languages including english, german, romanian, and chinese, where its mt performance was comparable with state-of-the-art neural machine translation (nmt) models. these results served as a motivation to explore the poten-tial of building mt datasets for low-resource languages with gpt models. unlike traditional translation software, gpt-4 was not trained with the explicit purpose of mt, and in fact, it is not clear to what extent it was trained in mt at all. further research, like @xcite , demonstrated the potential of applying gpt-generated synthetic data in the context of mt. their models translated from german (a high-resource language) and galician (a low-resource language). their findings revealed that while models trained solely on natural data outperformed those trained solely on synthetic data, the best performing model was the one trained on a combination of both datasets. these findings encourage the validation of translation quality in gpt models for lowresourced languages such as faroese. the population of the faroe islands is approximately 54,500 people @xcite , with the large majority speaking faroese as their l1. at this time, faroese mt resources are lacking, and the existing resources are not sufficient for training high performing mt models @xcite . focusing on faroese, this paper explores gpt-4's effectiveness in translating from faroese to english. the potential of the gpt models raises the pivotal question: can the creation of mt resources for the low-resource language, faroese, be automated, specifically through the capabilities of advanced models like gpt-4? to investigate this, the following contribution is made: these contributions provide a detailed human examination of gpt-4's proficiency in translating faroese to english, expanding on the current understanding of gpt-4's translation capabilities of low-resource languages. ## experimental setup this section outlines the methodology used to examine gpt-4's effectiveness in generating parallel data for faroese, a language with limited resources. firstly, the experiment involves generating synthetic parallel data using gpt-4. this output was then evaluated using the multidimensional quality metrics (mqm) framework with a single native speaker as an annotator. ## results the results for the mqm evaluation is summarized in as seen in the original sentence contains a spelling mistake; rørðslutarna is supposed to be spelled rørslutarnað which translates to "disabled". gpt-4 translated this term to "the pipe players". error category description accuracy addition translation includes information not present in the source. omission translation is missing content from the source. mistranslation translation does not accurately represent the source. untranslated text source text has been left untranslated. fluency punctuation incorrect punctuation (for locale or style). spelling incorrect spelling or capitalization. grammar problems with grammar, other than orthography. register wrong grammatical register. inconsistency internal inconsistency (not related to terminology). character encoding characters are garbled due to incorrect encoding. terminology inappropriate for context terminology is non-standard or does not fit context. inconsistent use terminology is used inconsistently. style awkward translation has stylistic problems. other any other issues. ## discussion according to the human evaluation of the gpt-4 translation data, gpt-4 has demonstrated its proficiency in translating from faroese to english, especially in the context of news articles. however, the translations are not perfect and we address the limitations later in this section. this finding of translation quality is consistent with recent research indicating gpt-4's effectiveness in translating from low-resource languages to english, as highlighted in studies by @xcite @xcite . the model's particular strength in news translation is likely due to its extensive training on a wide array of news texts, which is abundantly available online for collection. however, when it comes to blog texts, which are rich in idiomatic expressions and fixed phrases, gpt-4's performance dips slightly (from 94.41 to 88.38). this drop in performance suggests that while gpt-4 can generate high-quality translations, its capability diminishes with content that heavily features languagespecific idioms and cultural nuances. yet, the synthetic parallel sentences generated by gpt-4 present a "silver standard" resource for training mt models for faroese, complementing the "gold standard" human-translated data. although not a substitute for human translation, the combination of synthetic and human-generated data could potentially enhance the training materials available for faroese mt models (for german and galician, see @xcite ; for english, german and turkish, see @xcite . however, to fully assess the impact of gpt-4 generated parallel data on mt model performance, a larger dataset would be ideal. for this study, the collection was limited to 5,408 sentence pairs due to cost considerations and the licensing restrictions imposed by openai on their model's output 13 . during the study, a preliminary experiment was conducted to see how well gpt-4's translation performed from english into faroese, which revealed significant limitations. the model often failed to construct grammatically correct faroese sentences, frequently producing outputs that appeared to be an amalgamation of icelandic and faroese. this finding corroborates previous re13 at the time of usage, gpt-4-0613 cost @xmath00.06 for every output token. search indicating that gpt-4's capabilities in translating from english to low-resource languages remain constrained. studies by @xcite , @xcite , and @xcite have similarly documented these challenges, reinforcing the observation that gpt-4's performance in such translation tasks is not yet satisfactory. ## conclusion in conclusion, this paper has demonstrated the potential of gpt models like gpt-4 in generating synthetic parallel data, potentially mitigating the scarcity of high-quality, human-translated datasets. through a detailed analysis of gpt-4's translation from faroese to english, including a synthetic parallel dataset and an mqm framework-based evaluation, we have uncovered both strengths and limitations of employing gpt models for mt. while gpt-4 shows promise in generating translations that could serve as valuable training data, challenges remain, particularly with translations that involve cultural and contextual nuances. this exploration not only contributes to the understanding of gpt models' capabilities in translating low-resource languages but also sets the stage for future research directions. by integrating synthetic and humangenerated data, there's potential to enhance mt models for faroese, pushing the boundaries of accessibility and quality in mt for low-resource languages. this study underscores the necessity for ongoing research to fully leverage the capabilities of advanced models like gpt-4, aiming for a future where no language is left behind in the digital age.