Comparative Evaluation of Machine Translation Models Using Human-Translated Social Media Posts as References: Human-Translated Datasets (2025)

Abstract: Machine translation (MT) of social media text presents unique challenges due to its informal nature, linguistic variations, and rapid evolution of language trends. In this paper, we propose a human-translated English dataset to Arabic, Italian, and Spanish, and a human-translated Arabic dataset to Modern Standard Arabic (MSA) and English. We also perform a comprehensive analysis of three publicly accessible MT models using human translations as a reference. We investigate the impact of social media informality on translation quality by translating the MSA version of the text and comparing BLEU and METEOR scores with the direct translation of the original social media posts. Our findings reveal that MarianMT provides the closest translations to human for Italian and Spanish among the three models, with METEOR scores of 0.583 and 0.640, respectively, while Google Translate provides the closest translations for Arabic, with a METEOR score of 0.354. By comparing the translation of the original social media posts with the MSA version, we confirm that the informality of social media text significantly impacts translation quality, with an increase of 12 percentage points in METEOR scores over the original posts. Additionally, we investigate inter-model alignment and the degree to which the output of these MT models align.

## introduction the growing demand for multilingual content has driven the development of machine translation (mt) systems, which enable fast and costeffective translation of large text volumes. evaluating mt systems is critical to ensuring accuracy across diverse languages and text types, especially with the rise of social media as a rich, multilingual data source for tasks like sentiment analysis and topic detection. however, social media text's unstructured, error-prone nature-with lin-guistic variations, informal language, and abundant slang-poses unique challenges for mt. this work contributes to the ongoing efforts to enhance mt systems for handling the intricacies of social media language, ultimately advancing the accessibility of multilingual content in an increasingly interconnected world. we propose a humantranslated set of english x posts into arabic, italian, and spanish, along with their corresponding translations by three distinct mt models (google translate, marianmt, and facebook m2m), to provide a comprehensive basis for assessing the efficacy of these systems in handling the complexities of social media language. additionally, we have curated a dataset of 500 arabic x posts translated by humans into both modern standard arabic (msa) and english. these datasets serve as valuable resources for researchers in the field of natural language processing (nlp), providing diverse and representative samples of social media text for evaluating the performance of mt systems across multiple languages. furthermore, to gain insights into the relative performance of the mt models, we evaluate each mt model against the other two by using reference translations from other models. this approach allows us to assess how similar the translations generated by each model are to those produced by the other models when using a specific reference. by conducting pairwise comparisons, we can identify strengths and weaknesses of individual mt models and gain a deeper understanding of their relative performance on social media text foot_0 . to explicitly illustrate our contributions, we list them below: we also aim to address the following question: is there a correlation between the formality of x posts and the quality of their machine translations? we hypothesise that x posts written in a formal style would yield better translation outcomes. this question is explored in section 5.3.1, where we systematically compare the output of mt models when translating dialectical arabic versus modern standard arabic (msa) version of the same data. these contributions are essential for tackling the challenges of evaluating machine translation systems, particularly in the context of social media language. the human-translated dataset serves as a valuable benchmark for assessing mt model performance in capturing the nuances and informalities of social media text. through a comprehensive evaluation using established metrics such as bleu and meteor, we provide quantitative insights into the comparative performance of mt systems, thereby contributing to ongoing efforts to enhance the quality and accuracy of multilingual content translation, especially in the challenging domain of social media language. the remainder of the paper is structured as follows: section 2 provides an overview of related work, including existing parallel corpora and machine translation evaluation metrics. section 3 explains how the data was sourced and processed. the methodology is described in section 4. section 5 covers the experimental setup, evaluation, and discusses the results. finally, section 6 concludes the paper. ## related work in recent years, the field of machine translation (mt) has witnessed significant advancements, driven by the growing demand for multilingual communication across various domains. researchers have explored diverse approaches and techniques to improve the accuracy and effectiveness of mt systems, particularly in handling the challenges posed by social media text. in this section, we provide an overview of the related work in the literature, focusing on key contributions and trends in the field. machine translation systems are being used to translate social media posts, but their performance is not always perfect. users have expressed concerns about potential inaccuracies in the translations of their posts and the lack of control over the translation process. the use of machine translation tools, such as google translate @xcite , in social media is a relatively recent phenomenon, and it presents new challenges for translating informal language, dialects, and platformspecific language phenomena @xcite . while these systems can help overcome language barriers, they also have limitations, and users often desire higher translation quality. therefore, there is a need for machine translation systems specifically trained on social media text to improve their performance on informal language and address the concerns of users. furthermore, machine translation has proven beneficial for creating parallel corpora to train large language models (llms) for downstream tasks in natural language processing, reinforcing the importance of tailored translation systems for social media content @xcite . ## data the english data was sourced from the crisisnlp dataset, a valuable resource containing english x posts collected and annotated specifically for crisisrelated analysis and classification @xcite . this dataset provides a rich source of realtime social media data, offering insights into how individuals communicate and respond during crises. from this dataset, we selected a sample of 1000 x posts posted during various crisis events to enrich the diversity of content. we sourced the arabic data from the kawarith dataset, another valuable resource containing arabic x posts collected and annotated specifically for crisis-related analysis and classification @xcite . we sampled 500 x posts to generate parallel human translations. in ## methodology the data was sampled to cover diverse events worldwide, ensuring the inclusion of a variety of linguistic features, as each event took place in a different part of the world. the english dataset was sampled from the crisisnlp dataset and cleaned following the process outlined in section 3.1. following this, human translators were engaged to translate the x posts into arabic, italian, and spanish. simultaneously, three distinct machine translation models-marianmt (helsinki-nlp/opus-mt), google translate (4.0.0-rc1), and facebook's m2m (m2m100_418m)-were chosen for their popularity and accessibility to generate corresponding mt translations. for the arabic data sourced from the kawarith dataset, human translators translated the x posts to english and converted the arabic x posts to msa. similar to the english data, the arabic data was translated by the same three mt models to english, italian, and spanish. subsequently, an evaluation was conducted to assess these models against both human reference translations and each other. in addition to evaluating the translation quality of the original x posts to the various languages, we introduce an additional dimension to our study by investigating the influence of the informal nature of x posts on translation quality. to assess this, we translated the modern standard arabic (msa) version of the x posts into english and used these translations as candidate translations against the human reference translations. subsequently, we calculated bleu and meteor scores for both sets of translations. the comparison between translating the original arabic x posts and their standardised counterpart can provide valuable insight into the impact of tweet informality on translation quality. ## experiments in this section, we outline the experimental setup designed to systematically evaluate and compare machine translation models. we describe the process of data sampling, cleaning, and translation, followed by an overview of the evaluation metrics employed. subsequently, we present the results of the evaluation, including bleu and meteor scores, and analyse the inter-model alignment to gain insights into the performance of the translation models. ## conclusion our study examined the translation quality of social media text across multiple languages using human and machine translation. formal language, such as modern standard arabic, improved accuracy compared to informal text, and translation quality varied across languages and models, emphasising the role of formality and linguistic context. the human-translated datasets we are publishing will support improvements in machine translation, cross-lingual research, and crisis communication. future human evaluations of machine-generated translations will provide deeper insights into fluency and adequacy, refining our understanding of translation performance in social media contexts.