Grammar-based Data Augmentation for Low-Resource Languages: The Case of G uarani- S panish Neural Machine Translation (2024)

Abstract: One of the main problems low-resource languages face in NLP can be pictured as a vicious circle: data is needed to build and test tools, but the available text is scarce and there are not powerful tools to collect it.In order to break this circle for Guarani, we explore if text automatically generated from a grammar can work as a Data Augmentation technique to boost the performance of Guarani-Spanish Machine Translation (MT) systems.After building a grammar-based system that generates Spanish text and syntactically transfers it to Guarani, we perform several experiments by pretraining models using this synthetic text.We find that the MT systems that are pretrained with synthetic text perform better, even outperforming previous baselines.

## introduction from an early age, millions of south americans grow up and build their reality speaking languages that were used in their region for hundreds of years before the first contact with europeans. many of them also have formal schooling in their indigenous languages, as a way to encourage them to forge their identity according to that world vision, carrying with them the rich culture of their communities and passing it on to the next generations. although some of these indigenous languages are massively spoken, their speakers might not benefit from the recent advances in language technologies (e.g. high-performance machine translators, large language models) due to the lack of data to train models and other difficulties @xcite . most of these languages are considered low-resource languages in nlp research @xcite . low-resource languages suffer from a vicious circle that is hard to break: there is not enough data to build tools, but also there are no tools to massively collect text from the web to train them. that is the case of guarani (avañe'ẽ), an indigenous language spoken by nearly 10 million people mainly in paraguay, but also in some regions of argentina, bolivia and brazil. back in the 17th century, the tupi-guarani language family was so widespread in the region that it worked as a lingua franca @xcite . that legacy remains to this day, as evidenced by loanwords used in regions where guarani is not directly spoken, like uruguay @xcite . many years of contact with spanish resulted in a considerable number of varieties, jopara being the most widely spoken nowadays, used daily by millions of speakers in paraguay @xcite . jopara is a variety that exhibits frequent code-switching between guarani and spanish (the two official languages of paraguay), as well as many adapted and unadapted loanwords and phrases. the resources we use have a strong prevalence of this variety. in this work, we present an approach that aims to break the aforementioned low-resource vicious circle: through generating synthetic parallel data based on grammar knowledge, we try to augment the available data to enhance guarani-spanish machine translation (mt). we then train guarani-spanish mt systems using different combinations of the datasets in a pretraining and fine-tuning strategy, to understand which datasets could contribute the most to improve the translation and in what scenarios. all the resources used and developed for this work are available on github foot_0 . ## related work although guarani is still considered a low-resource language in the nlp community, it is currently an active area of research and some work has focused on this language in the past few years. in particular, considerable effort has been made to build guarani-spanish parallel corpora. @xcite created a parallel corpus based on paraguayan news, which was later expanded and enriched using other parallel data sources @xcite @xcite , resulting in the jojajovai guarani-spanish benchmarking corpus @xcite . since that corpus was made up of several subsets, the text represents different guarani varieties that show varying degrees of guarani-spanish code-switching, such as jopara or jehe'a. furthermore, guarani is included in meta's initiative no language left behind as an available language in the dataset @xcite . there is also a small bert model trained on the guarani wikipedia @xcite . previous efforts have been made to specifically work on guarani-spanish mt. vázquez et al. (2021) won the americasnlp 2021 shared task @xcite , which included the guarani-spanish pair. @xcite tried to take advantage of the rich morphology of guarani, incorporating morphological information to the models. @xcite performed translation experiments to evaluate if pretrained static word embeddings made a difference in mt performance. google translate includes guarani in the available languages since june 2022 @xcite . in low-resource scenarios, it is usual to explore strategies to increase the available data by creating synthetic text or reusing the existing text in clever ways. this process is known as data augmentation (da). one of the most used techniques for da is back-translation, consisting in translating monolingual data to the target language using an already available mt system for the considered language pair @xcite . some works explore more complex strategies to improve the effect of back-translation @xcite . other works explore completely different approaches for da, such as obtaining extra sentence pairs by changing common words for rare words in some original sentence pairs from the training set @xcite , transforming sentence pairs by cautiously changing the order of words or phrases @xcite , generating new data using different statistic-based algorithms @xcite , or creating multilingual lexicons and following different strategies to use them @xcite . in this work we follow a different approach: we try to create a large parallel corpus of synthetic data from grammar knowledge, and use it to pretrain a neural translation model. as far as we know, this is the first time an ad hoc grammar is used as da process for a south american indigenous language. in preliminary experiments, we observed that mt systems for the guarani-spanish pair showed low performance when dealing with short sentences. a similar observation can be inferred from @xcite , where the blue score for the short-length buckets is lower than those for the mid and long-length buckets. in our case, this could be caused by the nature of the used parallel data @xcite , consisting mainly of long, complex and somewhat formal sentences often found in journalistic writing. obtaining pairs of short sentences is not always easy, since they are generally found in social media (usually very noisy), subtitles (usually not available for low-resource languages) or in simple texts for children, which in general have no translation available, and are often hard to find (as digital text) for a low-resource language like guarani. to tackle this problem, we decided to build a synthetic collection of short texts with their translations, starting from a formal grammar based on the official guarani grammar (academia de la lengua guaraní, 2018). ## neural translation experiments our neural translation experiments were done in three phases: first we trained simple models with default parameters using the jojajovai training data to establish basic baselines; secondly we performed a hyperparameter tuning phase where we tried to find the best possible configurations for both the transformer and the seq2seq architectures; finally we experimented with scenarios using separate pretraining and fine-tuning stages, varying the pretraining data and the number of pretraining steps, and then fine-tuning with the jojajovai training data. all our experiments were done using the mariannmt (junczys-dowmunt et al., 2018) framework, which allows to train neural translation models based on seq2seq (lstm or gru with attention) and transformer based models. ## results section 4 presented the results of our experiments against the dev split of jojajovai. in this section, we evaluate the models over the test split and compare it to the results found on the original jojajovai benchmark experiments @xcite . we also ran the test data on the google translate api on october 15, 2023. as of this date, google translate is the only available translation platform that includes the guarani-spanish pair. we also notice that the models pretrained only using the synthetic ancora set, come in a very close second position for all metrics. this is very interesting, as this dataset is not very large but seemingly still manages to obtain great results, perhaps because it could leverage the linguistically inspired syntactic-transfer information with the realworld ancora text, creating a kind of artificial codeswitched data that was very suitable for this dataset. in order to understand better if this is the case, we performed an analysis of performance broken down by subsets in jojajovai, as shown in our models beat the google translate baseline on the abc, libro_gn and libro_td subsets, and have almost the same performance on the spl subset. the abc and spl subsets are by far the largest of the corpus, and also abc is composed of news text that has frequent code-switching between guarani and spanish. we think that our dataset, especially the synthetic ancora, would be able to emulate data that is similar to this code-switched data (e.g. sentences with guarani structure but where named entities and other non-frequent words are kept in spanish) which could explain the performance gain in this type of text. on the other hand, the dataset for which we have the lowest performance is anlp, from the americasnlp @xcite shared task, which is very different in nature and was built with the intention of having as little interference from spanish words as possible. fig. 2 shows a graphical comparison of the @xmath1, where we can clearly see the performance differences across the subsets. ## conclusions in this work we presented two main contributions. first, we built grammars for guarani and spanish that model roughly the same sets of simple sentences, and transfer rules between the languages, with a strong inspiration in the hpsg formalism. the grammar can be transformed into a probabilistic cfg, with subsequent fg filtering and syntactic transfer, that allows to generate guarani-spanish sentence pairs. additionally, this grammar-based system was applied to the spanish ancora corpus to obtain a silver standard guarani-spanish parallel corpus. second, we evaluate if the generated text has enough quality to boost the performance of guarani-spanish mt systems when used as pretraining data, i.e. if grammar-based text generation is a suitable alternative as a da methodology. we find that the seq2seq model pretrained on all the synthetic text plus the contents of the bible, and later fine-tuned on the jojajovai corpus, outperformed the previous jojajovai baselines in both directions. our results also outperformed the google translate translations for the @xmath2, while showing competitive performance for the @xmath3. even if the sentences in the synthetic corpus sometimes did not sound completely natural to a native speaker, there is no doubt that their content had a positive impact on the models. therefore, we think this approach can benefit other languages with a well-documented grammar, and also that it can be combined with other strategies, such as backtranslation. unlike back-translation, this synthetic text generation strategy does not need previously trained models or even digitized text to train them. we hope this work can inspire researchers working in low-resource settings, showing that research on grammars for mt could still be very relevant, especially as a way to alleviate data scarcity.