Rosetta Balcanica: Deriving a “Gold Standard” Neural Machine Translation ( NMT ) Parallel Dataset from High-Fidelity Resources for W estern B alkan Languages (2024)

Abstract: The Rosetta Balcanica is an ongoing effort in resource expansion for low-resource Western Balkans languages. This effort focuses on discovering and using accurately translated, officially mapped, and curated parallel language resources and their preparation and use as neural machine translation (NMT) datasets. Some of the guiding principles, practices, and methods employed by Rosetta Balcanica are generalizable and could apply to other low-resource language resource expansion efforts. With this goal in mind, we present our rationale and approach to discovering and using meticulously translated and officially curated low-resource language resources and our use of these resources to develop a parallel “gold standard” translation training resource. Secondly, we describe our specific methodology for NMT dataset development from these resources and its publication to a widely-used and accessible repository for natural language processing ( Hugging Face Hub ). Finally, we discuss the trade-offs and limitations of our current approach, and the roadmap for future development and the expansion of the current Rosetta Balcanica language resource.

## introduction many underresourced languages are spoken by ethnic groups residing in regions affected by eco nomic, social, or other crises. these circumstances frequently necessitate the involvement of interna tional institutions focused on economic assistance, the promotion of human rights, the advancement of democratic structures and processes, humanitar ian aid, peacekeeping, and regional economic and political stabilization. as part of these activities, these institutions issue reports and studies that sup port their missions and goals. the documents produced by international orga nizations operating in regions affected by these factors require precise translation to: a) preserve the original meaning of the master communique, and b) accurately convey that meaning across the languages of the regional groups. such documents constitute a "golden standard" for any downstream crosslingual computational linguistics tasks, par ticularly machine translation training. for a data set to fit such a standard, it must map across par allel data sets at the phrase and paragraph levels with high accuracy, be produced by professional translators, and be issued by an authoritative pub lication body. neural machine translation (nmt) @xcite is a stateoftheart approach to ma chine translation that uses an endtoend encoder decoder architecture with attention mechanisms @xcite , to translate source lan guage sentences into target language sentences. attentionbased models are particularly well suited for translation tasks as they support the training of models that can maintain attention on complex relationships and dependencies between two related sequences. for example, attention based models excel at learning relationships be tween words and phrases in two languages, even when the languages do not have a onetoone map ping. accurate and complete sentencetosentence alignment between the source and target languages in a training dataset is particularly important for ef fectively training neural machine translation mod els because it helps the nmt model form accu rate patterns of attention. therefore, a "golden" training set in lowresource languages is of great value for effective lowresource nmt model de velopment, as the training content is meticulously translated and aligned at the phrase, sentence, and paragraph levels. this effort focuses on the western balkan lan guages, a region in south central europe compris ing countries formed by the breakup of former yu goslavia and albania. the languages spoken in this region include dialects of southern slavic lan guages (serbian, croatian, bosnian, macedonian, slovenian, etc.), albanian, and others @xcite . these languages and resources were se lected due to our familiarity with the region's cul tural, historical, and ethnic circumstances, and our fluency in several of these language groups (in cluding croatian, bosnian, serbian, and to some degree, macedonian). furthermore, given the significant similarity be tween croatian, serbian, and bosnian languages @xcite , some of these parallel re sources, with translations from english, shqip, and macedonian to either serbian, croatian, or bosnian, can potentially be used to crossmap the translations between all three languages. this sig nificantly boosts these three language resources and their use for other multilingual tasks @xcite . the development of highquality, meticulously translated parallel datasets for lowresource lan guages such as those in the western balkans is crucial for advancing neural machine translation. in this paper, we outline the methodology, chal lenges, and future directions for creating these datasets, which serve as foundational resources for enhancing multilingual and crosslingual compu tational linguistics. ## background many of the languages of the western balkans are spoken by small national groups or ethnic minorities and are considered lowresource lan guages @xcite . alba nian, or shqip, is spoken by 7.5 million people worldwide, with approximately 4.5 million speak ers in the region. although an indoeuropean lan guage, it is unique and is not closely related to any other indoeuropean language. macedonian @xcite , an eastsouth slavic language, is spoken by approximately two million people. furthermore, the region has been severely af fected by a series of regional wars, which, in addi tion to causing significant population losses, have been followed by economic depopulation @xcite , a trend that continues today @xcite . it is estimated that 4.4 million peo ple have emigrated from the region (world bank group and wiiw, 2018, p.42). ## approach our overall approach to the development of rosetta balcanica was to identify the spe cific, officially translated multilanguage resources amenable to raw language processing and then convert them into a general and neutrally formatted parallel language resource. from there, we chose to postformat this neutral resource into a specific format (e.g. hugging face hub format), with the intention of making it accessible to a large group of nlp practitioners. we started the collection process by attempt ing to automate the search, retrieval, and prepro cessing of downloaded resources. we encountered a number of errors related to standard document formatting (page numbers, header/footer dividers) that made automation complex, uncertain and error prone, so we delayed the entire automation process in favor of manual collection and preprocessing to make initial progress in corpus development (first stage/release in a roadmap (see figure 4 ). we in tend to develop and proceed with a more robust automation starting with the second stage. ## dataset statistics the first release of the rosetta balcanica focuses on the parallel documents in shqip and macedo nian. for this release, we used only shqip re sources available through osce. ## challenges and limitations maceodonian and serbian entries are written in cyrillic script. to compare the entries between serbian, and, for example, croatian and bosnian, which are otherwise very similar languages, an ad ditional transliteration step is required. osce is a trusted dataset so we did not perform translitera tion of all the data. we checked 1% of randomly selected samples and found those accurately trans lated. ## future work the scope of the work presented in this encom passes the work on the osce dataset, which is topically narrow, and primarily performing man ual preprocessing to minimize "debugging" time. future work will encompass a) inclusion of other, similarly structured parallel language datasets, and b) automation of dataset retrieval and pre processing. once we process osce resources for all west ern languages, we will work to process hague tri bunal documents which are, like osce, profes sionally translated and available in most western balkan languages. after that, we will focus on the general multilingual resources from official me dia sources. we will focus on the latter mostly to diversify the topics and balance out representation. rosetta balcanica is on an ongoing project, and the development of new resources continues. al though it is an ongoing process, we have identified releases and milestones (figure 4 ) in a roadmap that maps to the inclusion of specific language re sources and the dataset development tools (e.g. resource retrieval and preprocessing automation, quality assurance, dataset registration, etc.). we mentioned earlier that we attempted to au tomate the dataset collection/retrieval and pre processing step. we found that the complexi ties and errorproneness of automation attempts, at that time, were slowing us down in the process of dataset development more than they were helping us. it is evident, though, that this process needs to be automated in order to be scalable and easily reusable, and it is on our roadmap to automate the process. in fact, we are in the process of improving the automation process, and we expect that the au tomation scripts will be available by the time this paper is published (mid2022). ## conclusion there are three takeaways from the rosetta bal canica effort of that are, we presume, of interest to the natural language processing community: