Prompt Engineering Enhances Faroese MT, but Only Humans Can Tell (2025)

Abstract: This study evaluates GPT-4’s English-to-Faroese translation capabilities, comparing it with multilingual models on FLORES-200 and Sprotin datasets. We propose a prompt optimization strategy using Semantic Textual Similarity (STS) to improve translation quality. Human evaluation confirms the effectiveness of STS-based few-shot example selection, though automated metrics fail to capture these improvements. Our findings advance LLM applications for low-resource language translation while highlighting the need for better evaluation methods in this context.

## introduction historically, it has been a challenge to achieve high-quality machine translations (mt) for lowresource languages. the lack of resources has been shown to impact not only the development of high performing mt models, but also the development of high quality automated translation metrics @xcite @xcite . low-resource languages often have to rely on string-based language independent metrics such as bleu @xcite and chrf @xcite . however, these methods have shown to perform poorly when compared to neural metrics, as shown by the wmt22 metrics shared task @xcite . the lack of neural metrics developed for these languages often leaves expensive and slow human evaluation as the only high quality alternative for detecting nuanced improvement in translation quality. recent advancements in llms offer opportunities to mitigate the effect that low-resources have on translation performance, leveraging few-shot learning to achieve remarkable performances with minimal data requirements @xcite . however, there is a disparity in the translation performance when it comes to low-resource languages vs high-resource languages @xcite @xcite . therefore, optimizing the efficiency of these models in data-constrained environments demands a strategic approach in order to get the best performance. there is still much that is unknown about how prompt engineering and fewshot example selection influences translation performance. furthermore, llms have proven to be a competitive alternative also for what concerns translation evaluation @xcite . however, this ability of llms to assess translation has not been proven yet in the context of low-resource languages. we investigate how sts-driven example selection, applied with the translation query, improves translation quality in @xcite , specifically the gpt-4-1106-preview release, for faroese foot_0 , a critically low-resource language. our findings therefore demonstrate a novel approach to improve the utility of sparse data. moreover, we demonstrate how current translation metrics cannot adequately capture nuances in translation performance, advocating for the development of more robust evaluation tools. through this exploration, we provide an assessment of the state of the art of mt for faroese, and highlight how current automated evaluation metrics cannot appropriately capture nuanced improvements provided by prompt engineering. our contributions are the following: ## conclusion this study shows that selecting few-shot learning examples based on sts can improve gpt-4 turbo's faroese translation performance, as confirmed by human evaluation. however, current automated metrics fail to detect these improvements, highlighting a critical issue in low-resource language translation evaluation. while llms offer new opportunities for language generation, the inability of automated metrics to capture progress in low-resource contexts could widen digital language representation disparities. this situation necessitates expensive human evaluation, potentially hindering advancements. therefore, we call for collaborative efforts to develop metrics specifically designed for low-resource language contexts.