The unreasonable effectiveness of large language models for low-resource clause-level morphology: In-context generalization or prior exposure? (2024)

Abstract: This paper describes the submission of Team “Giving it a Shot” to the AmericasNLP 2024 Shared Task on Creation of Educational Materials for Indigenous Languages. We use a simple few-shot prompting approach with several state of the art large language models, achieving competitive performance on the shared task, with our best system placing third overall. We perform a preliminary analysis to determine to what degree the performance of our model is due to prior exposure to the task languages, finding that generally our performance is better explained as being derived from in-context learning capabilities.

## introduction this paper describes the submission of team "giving it a shot" to the americasnlp 2024 shared task on creation of educational materials for indigenous languages @xcite . this task covers three indigenous languages of the americas: yucatec maya (yua), guaraní (grn), and bribri (bzd). the task is similar to the clauselevel reinflection task described by @xcite and explored in a 2022 mrl shared task @xcite . however, it is more challenging in a number of ways. the first is structural: the present shared task provides an input sentence, and what values of features should be changed, while the previous task provided all feature values present in the input and what they should be changed to. as such, the features describing the source must be learned latently. other challenges come from differences in the languages covered: all three languages in this shared task are relatively low-resource, and correspondingly the training data in the shared task is also very limited (595 training examples at most). however, as a morphological/morphosyntactic task 1 , the input-output functions are relatively sim-1 note that, as in prior work on clause-level morphology, ple compared to many common tasks in nlp, being in all likelihood context-free or even regular @xcite @xcite . increasingly in nlp, even computationally complex tasks such as sentiment analysis are being framed as few-shot tasks for large language models (llms), with impressive results being obtained by presenting a few examples to a language model and allowing it to perform next-token prediction @xcite @xcite . the ability of such paradigms to improve performance over raw language model probabilities has been termed incontext learning; however, this term has been the subject of controversy, as it is not learning in the traditional machine learning sense, nor is it clear exactly how much information is being extracted from the context. for example, in the "in-context learning" of sentiment analysis, much of the relation between a sentence and a sentiment label is presumably latent in the pre-trained weights, and the examples serve moreso to "extract" that information from the model, enabling better generalization than could be expected from the information in the provided in-context examples alone. this setting therefore represents an interesting case: if few-shot prompting works well here, will it be due to prior language exposure, or an ability to generalize simple functions from limited data? to explore this question, we create three simple few-shot prompting-based systems, based on two closed-source llms (gpt-3.5 and gpt-4) and one openly available model (command r+), finding they perform competitively on the shared task. we permute the characters in the dataset to preserve the problem stucture while ablating language information, finding some evidence that the models primarily generalize in-context data, rather than using prior language exposure. the functions involved sometimes operate at the clause level here's some examples. source,change,target táan a bin koonol tu k'íiwikil koonol,type:neg,ma' táan a bin koonol tu k'íiwikil koonoli' táan u bin koonol tu k'íiwikil koonol,type:neg,leti'e' ma' táan u bin koonol tu k'íiwikil koonoli' jach k'a'abéet in bin tu k'íiwikil koonol,type:neg,ma' jach k'a'abéet in bin tu k'íiwikil koonoli' táan a bine'ex ich kool,type:neg,ma' táan a bine'ex ich kooli' teche' ka bin xíimbal tu yotoch,type:neg,teche' ma' ta bin xíimbal tu yotochi' ... now fill in the third column: te'exe' táan a bine'ex koonol tu k'íiwikil koonol,type:neg,te'exe' ma' táan a bine'ex koonol tu k'íiwikil koonoli' ## method we treat the task as a simple few-shot prompting problem, using no external data. we consider three models: two closed-source (gpt-4-0125 and gpt-3.5-turbo-0125 from openai) and one open-source (command r+ from cohere). our prompt is minimal, de-emphasising problem specific factors. it simply presents relevant examples from the training data in a csv format, then asks the model to complete the third column of a test item csv row. we use no additional data besides the provided training set, and perform no fine-tuning. a sample prompt is shown in figure 1 . in contrast to prior work showing the utility of expert prompting @xcite , describing the task domain @xcite , and tipping @xcite , preliminary evidence showed limited effects of any of these techniques when augmenting our prompt format. indeed, treating the problem as a simple csv completion task seems to have triggered interesting behavior in all 3 models: almost without exception, the first line contained either just the predicted target, or all three completed columns separated by commas. indeed, even on our worst-performing model and language pair, bribri using the cohere model, only 4/480 examples in the test set are miss-parsed by these heuristics (i.e., yielding something other than the model's prediction), in contrast to prior works where large language models typically place their completions unpredictably in unstructured text, causing parsing errors. while the format of the prompt is simple, some heuristics are required to best make use of the provided training data and compute costs. typically, (e.g. the addition of a particle to express negation) only examples of the requested change are shown. in cases where more than 10 examples of a particular change occur in the training data, the training data exhibiting this change is sorted according to the sum of bleu @xcite and chrf @xcite with the source of the test item as a reference, and the 10 examples with the highest score used in the prompt. further, when the specific change occurs fewer than 3 times in the training data (as is often the case in bribri), we back off to similar changes: we break the queried change into the component feature changes, and take up to 3 instances of each component change. if a feature change does not occur on its own in the training data, we add one example containing the feature change which maxmizes the sum of bleu and chrf between its source and the target source. finally, we add up to 8 examples which contain some of the component changes, again chosen by their source bleu+chrf similarity. we use temperature 0.1 for the openai models and temperature 0.3 for cohere models. preliminary evidence suggested that lower temperatures aided consistency. ## results our results on the test set are shown in full in while our results suggest that large language models can solve complex clause-level reinflection tasks for some indigenous languages, it is unclear what drives this behavior. one hypothesis is that it is driven largely by prior exposure to these languages. the glot500 dataset, which attempts to collate large amounts of data for low-resource languages for language model pretraining, contains 610,052 maya sentences; 87,568 guaraní sentences, and none for bribri @xcite . attempts to develop a large corpus for bribri have so far maxed out at just around 100,000 tokens, even with manual gathering of data from books not on the internet @xcite . this lines up relatively neatly with our results, with @xmath0. another possibility is that the model is primarily generalizing the patterns of in-context examples it is provided. support for this account is provided from the observation that the same pat-tern of @xmath1, which has no prior language exposure; suggesting that the inherent difficulty of the task may vary between the languages/their datasets. as such, our primary results alone are ambiguous between these two hypotheses. to differentiate these hypotheses, we develop a simple test involving permuting the alphabet for each language, such that most characters are mapped to other characters. this should provide a problem of an equivalent difficulty to the original, but which has a very different distribution over tokens, which should limit the degree to which the model uses information from prior exposure to the languages. to ensure the difficulty characteristics of the problem are preserved, letters with diacritics are permuted analogously to their counterparts without diacritics. this is due to the observation that a positive quality of our systems is their tendency to generalize patterns that apply to one set of diacritics on a letter to different diacritics on that letter. as an example, here is a real maya sentence from the dataset followed by its permuted counterpart: as the structure of the shared task prevents us from evaluating on a permuted test set, we present results on the development set for this experiment, shown in ## conclusion we present a simple few-shot learning setup for the americasnlp 2024 shared task on the creation of educational materials for indigenous languages, applied to three state-of-the-art large language models. we find even simple few-shot prompting techniques are able to beat the baseline, with our best system (gpt-4) placing third in the shared task. we investigate the extent to which the performance of our approach is due to a model's prior exposure to the language, by using a character-permuted version of the development set to maintain the problem structure while ablating the language information. we find from this preliminary evidence that the performance of these systems is driven more by in-context learning capabilities than prior exposure to these lowresource indigenous languages. we also find preliminary evidence of performance sensitivity to subwords, as we find that sometimes the model performs better on the permuted language than the original language. one question not addressed here is the cause of the relative performance of the models on each of the three languages. the differences in performance mirror the performance of the baseline, suggesting that in some sense perhaps e.g. the maya data is simpler or the training data is more informative than for some of the other languages. however, future work could characterize this further, investigating what kind of data sparsity these systems can generalize over and what kinds of functions they are better or worse at generalizing. for example, anecdotally for bribri we found the systems struggled to generalize morphophonological stem changes (e.g., sú + ök should be sawök, but the model produces súök).