On the Robustness of Neural Models for Full Sentence Transformation (2024)

Abstract: This paper describes the LECS Lab submission to the AmericasNLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages. The task requires transforming a base sentence with regards to one or more linguistic properties (such as negation or tense). We observe that this task shares many similarities with the well-studied task of word-level morphological inflection, and we explore whether the findings from inflection research are applicable to this task. In particular, we experiment with a number of augmentation strategies, finding that they can significantly benefit performance, but that not all augmented data is necessarily beneficial. Furthermore, we find that our character-level neural models show high variability with regards to performance on unseen data, and may not be the best choice when training data is limited.

## introduction morphological inflection is an nlp task with a rich history of rule-based, statistical, and neural methods @xcite @xcite @xcite @xcite . typically, systems must predict an inflected form of a word (such as "cats") given a lemma form ("cat") and an inflectional change (plural). in the americasnlp 2024 shared task on creation of educational materials for indigenous languages @xcite , systems must convert a base sentence into a target sentence by changing one or more linguistic properties (example in ## background in 2021, the first edition of the workshop (and shared task) on natural language processing for indigenous languages of the americas (americasnlp) was proposed. for this edition, the task of machine translation was presented to the participants. the goal of this shared task was to learn machine translation models for ten indigenous languages. the participants were given ten sets of language pairs: quechua-spanish, wixarika-spanish, shipibo-konibo-spanish, asháninka-spanish, raramuri-spanish, nahuatl-spanish, otomí-spanish, aymara-spanish, guarani-spanish, and bribri-spanish @xcite . for the 2022 edition, the participants were asked to present novel speech-to-text translation systems for bribri-spanish, guaraní-spanish, kotiria-portuguese, wa'ikhana-portuguese, and quechua-spanish @xcite . finally, in 2023, the task was machine translation for the transformer. @xcite also finds that in many cases, the transformer can outperform recurrent networks at character-level tasks. thus, we also compare with an encoder-decoder transformer. linguistic changes are treated as in the lstm. ## data augmentation in very low-resource settings, data augmentation can be highly effective at improving output quality and performance. we employ a number of strategies for augmentation. ## shared task submission we selected a number of systems for final submission to the shared task, based on our evaluation results. we use the chatgpt system with a chunk size of 20, the mbart system, and several of the augmented character-level neural systems. we aim to select a diverse set of augmented systems, so we select the copy ext , copy tr + copy ext , and copy ext + per systems for the lstm model and the copy ext , copy ext +trans, and copy ext + per + con systems for the pointergenerator model. we train final models using the training data and specified synthetic dataset. we perform hyperparameter search and select the optimal model architecture for each language and model, which we report in appendix a. we train models for 1000 epochs, selecting the best model according to vali-6 164 20 30 40 50 60 70 base + copy_all, trans + copy_all, per + copy_all, con + copy_all, per, con + copy_ext, trans + copy_ext, per + copy_ext, con + copy_ext, per, con lstm pg lstm (a) bribri 20 40 60 80 100 base + copy_all, trans + copy_all, per + copy_all, con + copy_all, per, con + copy_ext, trans + copy_ext, per + copy_ext, con + copy_ext, per, con lstm pg lstm (b) guaraní 20 40 60 80 100 base + copy_all, trans + copy_all, per + copy_all, con + copy_all, per, con + copy_ext, trans + copy_ext, per + copy_ext, con + copy_ext, per, con lstm pg lstm (c) maya figure 3: chrf results for combinations of strategies. we report results from the covered test set in first, all of the datasets involved are quite small, and it is possible that through random variability, the test set was meaningfully different in distribution from the evaluation set. neural models can be vulnerable to distributional shift, particularly when training data is scarce (linzen, 2020), which may explain why the non-neural baseline model fared better. we briefly investigate whether this is the case by examining the types of linguistic changes in each data split. specifically, for each desired linguistic change in the evaluation and test datasets (which might include multiple changes from a single example), we compute the number of times that change occurs in the training dataset, and average over all changes. this gives us a rough estimate of how common the linguistic changes are in the model's training data. we report these results in the other potential contributing factor is that due to the small datasets and difficult nature of the task, the performance of our models was highly variable. for augmentation strategies such as synonym replacement, the base assumption that synonyms are even present in a dataset of this size might not be accurate. during training, we often observed dev accuracy curves that swung wildly, sometimes jumping up or down by 10 points in a single epoch. furthermore, since we performed a large number in the future, this could be avoided by using manyfold cross-validation to select models rather than a single dev set. 11 finally, we saw significant performance benefits to including sentence copying in figure 2 , and we employed this in all of our submitted characterlevel systems. however, this strategy is most beneficial when it includes the sentences and lemmas that appear in the data being evaluated. it is possible that our external corpora happened to contain more overlap with the dev set examples than those in the test set, which could significantly impact performance. we suspect the strategy of retraining including the test inputs as synthetic examples could alleviate this. overall, these results serve as a cautionary example of the risks of selecting final systems based on limited evaluation metrics in extremely lowresource scenarios. 11 we considered this, but it was ultimately too resourceintensive for the number of experiments we wished to run. ## conclusion we describe our systems for the 2024 americas-nlp shared task on the creation of educational materials for indigenous languages, which include llm-based systems, character-level neural networks, and finetuned multilingual models. we observe potential benefits from augmentation strategies for character-level models, particularly the sentence copying strategy, which helps a model adapt to new examples. however, we find that nearly all of our systems, with the exception of the llm system, do not generalize well to the covered test set, resulting in poor performance on the shared task. these results reaffirm the difficulty of training robust neural models in low-resource scenarios and the importance of thorough validation.