Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource B razilian Languages (2024)

Abstract: Large Language Models are transforming NLP for a lot of tasks. However, how LLMs perform NLP tasks for LRLs is less explored. In alliance with the theme track of the NAACL’24, we focus on 12 low-resource languages (LRLs) from Brazil, 2 LRLs from Africa and 2 high-resource languages (HRLs) (e.g., English and Brazilian Portuguese). Our results indicate that the LLMs perform worse for the labeling of LRLs in comparison to HRLs in general. We explain the reasons behind this failure and provide an error analyses through examples from 2 Brazilian LRLs.

## introduction despite numerous advancements in the nlp research due to large language models (llms), available resources mainly cover 20 out of the estimated 7,000 languages @xcite . as a result, majority of world languages could still be considered as "low-resource". being a low-resource language (lrl) encompasses different types of inadequacies with respect to the availability of data for creating language technologies @xcite . focusing on multilingual linguistic scene in south america, we test the performance of llms for annotating part-ofspeech (pos) tagging for 12 lrls from brazil, make a comparison with 2 lrls from africa and 2 high resource languages (hrls) (e.g., english and brazilian portuguese) through human evaluation. the evaluation is challenging for two reasons. first, there is a lack of benchmark datasets for the lrls in brazil in general. the ones we were able to find in universal dependencies (ud) data base, 1 do not have the training data to fine-tune multilingual language models. hence, we can only 1 https://universaldependencies.org/ leverage prompting llms or cross-lingual transfer through multilingual language models. secondly, there is a lack of large monolingual data to benefit from effective multilingual and cross-lingual transfer techniques @xcite @xcite . we could only find the bible corpora with less than 35k sentences for 7 out of the 12 languages. we perform the evaluation on 12 brazilian lrls by prompting gpt-4 llm and cross-lingual transfer individually from english and brazilian portuguese leveraging xlm-r. we preferred gpt-4 because the other open multilingual models (e.g., mt0 @xcite , aya @xcite ) do not support the lrls in this study. the results of both methods indicate low performance (less than 34.0% while high-resource languages achieved over 90.0%). however, gpt-4 leads to better results and brazilian portuguese performs better than english in zero-shot evaluation. furthermore, to boost the performance of cross-lingual transfer, we perform language adaptation using xlm-r on each language, before finetuning brazilian portuguese, and evaluating on that language. this boosts the performance by +3 to +12.0 points on six out of seven languages. our findings suggest that cross-lingual transfer to these languages is very challenging and having few training examples may further boost the performance. therefore, there is a need for building nlp resources across different tasks for these lrls. ## literature overview in terms of labelled datasets for brazilian lrls, we only found datasets from the ud tasks: gerardi et al. (2022) developed for tudet ud treebanks covering 8 tupian languages, other languages covered in ud are apurina @xcite , bororo, madi-jarawara, and xavante (contributed by the tudet team). for the monolingual data, we found seven bible corpora on the ebible corpus (akerman et al., 2023) that are freely available. all languages lack a large monolingual corpus which makes it very challenging for cross-lingual transfer and multilingual pre-training of llms. in terms of evaluation, some studies have already shown the potential of prompting llms in multilingual settings @xcite , including some lrls @xcite . however, evaluations covering brazilian lrls are lacking. to the best of our knowledge, our study is the first to fill this gap. ## experimental setup we focus our evaluation of pos tagging (a subtask of universal dependencies (ud)) on brazilian lrls due to the simplicity of the task, its popularity, and the availability of the test evaluation datasets in ud . foot_2 ## results zero-shot evaluation results while pos tagging has a performance of 98% (e.g. for english and portuguese) when training data are available (especially for hrls), the performance decreases while performing zero-shot transfer to other lan-guages because pos tagging is language-specific. the transfer performance is low for both brazilian and african languages (probably) because they are not typologically related whereas english and portuguese are slightly related (i.e., being in the same indo-european family) and covered by xlm-r, thus achieving an impressive transfer performance (> +83%). gpt-4 vs. basic cross-lingual transfer gpt-4 performed slightly better than the zero-shot transfer from other languages in our experiments indicating better abilities of llms for this task. for english and portuguese, the performance reaches to 90% (although it is not on par with fully-supervised setting). for african languages, the performance was lower than the hrls, but it was still decent (64.8-75.4) probably because the llms were exposed to some african languages during pre-training. the struggle of gpt-4 for brazilian lrls can be explained with the fact that these languages were probably not included during the pre-training. the generation is often not useful for some examples, where gpt-4 declines to give answers like "as an ai, i'm unable to provide the pos tags for words in languages i'm not programmed to understand. ". thus, we had to remove such examples from our evaluation. however, this was not the case for african lrls and the hrls. ## error analysis in this section, we provide examples from 2 brazilian languages (karo and guajajara) where the llms made errors with the pos tagging. the first line refers to the original sentence, the second line refers to the gold-standard ud pos tag; the third line refers to the gpt-4 pos tag.) in example (1), the auxiliary verb (in karo) has the same orthographic form as the english interjection okay. in example (2), the guajajara verb has (partially) the same orthographic form as the english interjection (oh). due to these similarities, gpt-4 seems to tag the pos for these words according to english instead of the pos tagged in ud for karo and guajajara. ## ethics issues since we used publicly available data sets, we do not foresee any major issues in terms of ethical concerns.