COPAL - ID : I ndonesian Language Reasoning with Local Culture and Nuances (2024)

Abstract: We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and therefore, provides a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere. Professionally written by natives from scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID. In addition, we present COPALID in both standard Indonesian and in Jakartan Indonesian–a dialect commonly used in daily conversation. COPAL-ID poses a greater challenge for existing open-sourced and closedstate-of-the-art multilingual language models, yet is trivially easy for humans. Our findings suggest that general multilingual models struggle to perform well, achieving 66.91% accuracy on COPAL-ID. South-East Asian-specific models achieve slightly better performance of 73.88% accuracy. Yet, this number still falls short of near-perfect human performance. This shows that these language models are still way behind in comprehending the local nuances of Indonesian.

## introduction a predominant challenge in multilingual nlp is to capture the sociolinguistic nuances and contexts that vary from culture to culture @xcite . this is especially important in localized language reasoning tasks where knowledge of local context and culture is crucial. for example, while the fact that a "chanting crowd" logically follows "super bowl" might be obvious within the us cultural sphere, the same cannot be said in japan, where "summer koshien" is the more locally appropriate context for a "chanting crowd". as another example, while "wearing suits" follows "attending a wedding", within the indonesian culture "wearing batik" is probably the more appropriate consequence. existing multilingual language reasoning datasets such as xnli @xcite and xcopa @xcite do not capture such local nuances because of two reasons. first, they are largely sanitized from localized and cultural elements. general, common-sense instances such as "water flows" following "opening faucet" are the ones typically found in the datasets. second, any cultural element appearing in the dataset is typically based on us/western context. even when translated to other languages, it retains the original context while ignoring the cultural mismatch between the context and the common culture of the people speaking the target language. as a consequence, the current language reasoning benchmarks for multilingual models are lacking the crucial aspect of local and cultural context. to provide a better benchmark for multilingual models that also capture local nuances for indonesian, we introduce copal-id. 1 it follows copa's @xcite commonsense causal reasoning format. copal-id is handcrafted by native long-term jakarta residents to capture indonesian cultural and local nuances, especially in jakarta. specifically, we define three categories of locality: culture, which captures local customs or norm; local terminology, terms commonly known by locals, yet not for outsiders; language, which tests the nuance of the language, including uses of homonymy and non-compositionality. each category contains data that can be considered uniquely indonesian and can be understood as common customs or general knowledge by locals. additionally, the dataset comes in pairs of two forms: standard indonesian and colloquial indonesian @xcite , with the latter being the go-to form in day-to-day contexts. we find that the copal-id dataset is trivially easy for native jakartans, with our human scorers category premise correct option incorrect option note t pria itu memperbaharui kk miliknya (the man updated his kk) ia baru saja menikah (he just got married) ia baru saja lulus kuliah (he had just graduated from college) kk is a legal document that lists all the family members in a household. l rumah tetangga saya baru saja dibobol maling dia cuma bisa gigit jari dia menggigit jarinya gigit jari is a figure of speech to express helplessness. the 2 nd option is more literal. (my neighbor's house was just broken into by thieves) (he can only bite his fingers) (he bit his finger) t + c anak itu diterima masuk ui sekeluarga makan nasi kuning sekeluarga makan nasi uduk ui is one of the top universities in indonesia. nasi kuning is often served for celebrations. (that kid was accepted into ui) ## data creation copal-id is created through several steps. as a preliminary step, we formulate the end-to-end plan shown in figure 1 . in the data creation step, each of the five data creators (native and raised in indonesia, accustomed to jakarta culture) is tasked to create data following the definition in section 3.3, i.e., to write the premise and both alternatives, in standard indonesian. each person is set a target of 110 data (but more is welcome). the resulting data then goes through a check-and-review process. this involves (1) an automated duplicate checker using tf-idf vectors and (2) a double-blind crossreview process where each created data instance is assigned to two other creators for review. the assignment is distributed uniformly, meaning that there is no bias in the creator-reviewer pairings. the reviewers see the data with the alternatives shuffled, so they do not know the correct answer. ## experiment setup our data evaluation involves three different setups: monolingual, cross-lingual, translate-test, and colloquial test. each setup is tailored to a specific scenario. for monolingual setup, we use the same language (indonesian and colloquial indonesian) for training and testing. zero-shot cross-lingual utilizes the english copa dataset as the few-shot examples or training data for zero-shot classification and uses the copal-id dataset for testing. in contrast, the translate-test employs an englishtranslated version of the copal-id for testing instead. we use seamless-m4t (seamless communication, 2023) for translation. the evaluation under the monolingual setup is performed twice, one using copal-id with standard indonesian and the other with the colloquial dataset instead. we forego performing the colloquial testing on the two other setups, noting that the high lexical distinction (section 4.4) would hamper the translation quality on the translate-test setup and that prior work has noted big degradation in cross-lingual performances of indonesian local languages even for those with high lexical similarity with standard indonesian @xcite . detailed data and model setup can be seen in ## experiment results our experiment results can be seen in among open-source models, xlmr-large exhibits the best performance among the finetuned models, surpassing indobert, which is pre-trained using the indonesian dataset. on the other hand, for in-context learning, sea-specific models generally achieve the best performance. for general models, bloomz outperforms others, followed by polylm. polylm does not surpass bloomz despite having more parameters, showing that it takes other factors, not just a bigger size, for this task. regarding the closed-source models, gpt-4 outperforms chatgpt by a significant margin yet cannot beat human performance. does colloquial indonesian pose an additional difficulty? copal-id-c tests the impact of colloquial data on models' performance. it is evident that human performance has comparable results for both colloquial and non-colloquial forms of copal-id, unaffected by it. on the other hand, all of the open models used for prompting performance dropped by about 3-9 percent. surprisingly, the differences are negligible for each fine-tuned model, demonstrating that finetuning them with the gen-x dataset is quite robust to colloquial texts. this also indicates a lack of representation of colloquial indonesian within the pretraining data of these open models. on the other hand, both chat-gpt and gpt-4 models exhibit minimal degradation when evaluated under colloquial indonesian. however, due to their proprietary nature, it is impossible to pinpoint the reason. for copal-id, does using zero-shot crosslingual instead of a translate-test improve the score? performing translate-test on our dataset tends to degrade the model's performance notably. for instance, xlmr-large exhibits a significant decline from its standard one by approximately 12%. we hypothesize that this drop is due to translation errors caused by long-tail words or local terminologies. in this scenario, we recommend using a local-nuance-aware machine translation model to obtain optimal performance, though admittedly such a model (or an open parallel data to build it) currently does not exist. for finetuned models, the cross-lingual approach (train on en copa, then evaluate on copal-id) yields comparably bad performance similar to the translate-test, aside from sailor. therefore in this scenario, english data is not really helpful, whether we use it for cross-lingual or translate tests. interestingly, cross-lingual prompting is visibly better than translate-test, noting that language models can perform in-context learning from a different language. do the in-context learning setting and prompt template matter for copal-id? in how is the models' performance across different categories? based on figure 3 , the language category of copal-id achieves the best accuracy across all models, while the culture category performs the worst. this underscores the challenges posed in this category. the terminology category has a slightly higher overall score than the culture category. we posit that since the dataset incorporates local nuances that require prior knowledge to answer, it might either conflict with other regions' cultural knowledge or be nonexistent within the embedded knowledge of the pre-training models. using colloquial test data also tends to hinder the model performance, especially in the language category, where the open-source prompting model is more affected. in contrast, fine-tuned models are relatively robust to colloquial text, with some categories demonstrating comparable scores between the original and the colloquial. would explicitly instructing the models to reason from a local point-of-view help? we ran our experiment using previously existing prompts. however, these prompts are in english, and none are explicitly designed for indonesian causal reasoning. the closest indication for these llms to reason based on local aspects is when we translate the prompt into indonesian. in ## qualitative analysis upon analyzing the output of the models, we discover some interesting findings where some in- for nasi uduk and nasi kucing (lit: cat rice), both are dish names with a cultural context where one is often consumed as breakfast while the latter is for dinner. the second example is more intricate. the term jaga lilin (translated: keeping the candle), a common black magic practice widely known in indonesia, requires the model to possess the right cultural knowledge. other examples show some activities that are not commonly practiced outside indonesia, such as kissing our parents' hands. although the llms are trained on multilingual data, some long-tail terms and local nuances are bound to be missed. without imposing their context explicitly on the models, it would remain challenging for them to infer the correct reasoning. ## conclusion we release copal-id to benchmark common sense reasoning with local nuance in indonesian that consists of culture, terminology, and language in copa format. we maintain our data quality by having several cross-reviews and automatic duplicate checks. we then benchmark copal-id by evaluating it through in-context learning and finetuning. our experiment shows that copal-id proves to be challenging for current open-source models across different experiment setups, yet easy for natives. we also provide additional insight that in-context learning is sensitive to the template, and using few-shot examples helps improve the accuracy of the models. we believe that each region has a different culture, which results in different common sense on some specific things, such as norms and values. hence, we hope that publishing this dataset encourages the nlp community to build new datasets and models that incorporate diverse local nuances, including indonesia. we leave out how to build a local nuance-aware model for future work.