Cloze Evaluation for Deeper Understanding of Commonsense Stories in I ndonesian (2022)

Abstract: Story comprehension that involves complex causal and temporal relations is a critical task in NLP, but previous studies have focused predominantly on English, leaving open the question of how the findings generalize to other languages, such as Indonesian. In this paper, we follow the Story Cloze Test framework of Mostafazadeh et al. (2016) in evaluating story understanding in Indonesian, by constructing a four-sentence story with one correct ending and one incorrect ending. To investigate commonsense knowledge acquisition in language models, we experimented with: (1) a classification task to predict the correct ending; and (2) a generation task to complete the story with a single sentence. We investigate these tasks in two settings: (i) monolingual training and ii) zero-shot cross-lingual transfer between Indonesian and English.

## introduction commonsense reasoning is a key component of natural language understanding (nlu), which previous work @xcite @xcite ) has attempted to model through tasks such as story comprehension. while humans can easily comprehend temporal and causal relations to understand a story narrative, machines tend to struggle due to implicit information and story premises. often, world knowledge such as social conventions, the laws of nature, and common logic are required to connect the premises to draw appropriate conclusions or closure @xcite . @xcite and @xcite introduced the story cloze test framework to empirically evaluate commonsense reasoning, based on english short stories about daily-life events. the task is to choose the correct ending of a four-sentence story based on a two-way multiple choice. @xcite published 3,700 data pairs, and the dataset has been used to model commonsense reasoning @xcite @xcite @xcite and perform discourse probing of pretrained language models @xcite . there is a lack of research modeling story comprehension in languages beyond english. @xcite argued that current progress over english may not generalize to other languages because of its anglocentric bias both linguistically, and also in terms of cultural and social conventions @xcite . motivated by this, we explore commonsense reasoning in indonesian by constructing a dataset based on the framework of @xcite . xcopa @xcite is perhaps the most closely-related work to ours, wherein 600 instances of the copa dataset @xcite were manually translated into 11 languages, including indonesian. copa is an open-domain commonsense causal reasoning task that consists of two-sentence pairs, and does not include complex narrative comprehension. moreover, the translation approach also has its own limitations, in entrenching anglocentric social contexts in other languages. to summarize, we introduce the first story cloze test in indonesian, and perform preliminary studies based on: (1) a classification task to predict the correct ending @xcite ; and (2) a single-sentence generation task to complete the story @xcite . we perform these two tasks in two settings: (1) monolingual training, and (2) zero-shot cross-lingual transfer, between indonesian and english. our data and code are available at https://github.com/fajri91/ indocloze . ## dataset construction following @xcite , we construct an indonesian story cloze test dataset. each instance consists of a four-sentence premise, and two candidates for the fifth sentence: an appropriate person location organization (#unique: 1962) (#unique: 114) (#unique: 166) rio, acha, reno, mamat, hana, gina, juju, tarra, maria, elisa indonesia, jakarta, bandung, kenya, bali, jogja, surabaya, korea, monas sd harapan, kai, sma harapan, sma angkasa, bobo, bimbel, smp harapan and inappropriate ending. similar to @xcite and @xcite , our corpus consists of daily-life events, but in indonesian contexts (e.g. locations, places, names, food, culture). data creation. we hired seven indonesian university students to each write 500 short stories over a period of one month. as part of the recruitment, candidates were provided with story requirements and several examples, 1 and asked to write a 5sentence story, as well as an inappropriate fifth sentence. from ten applicants, we hired the seven best candidates based on their submitted stories. after one month, four workers completed the job and were paid rp 750,000. 2 the three who did not complete the task were paid a prorated salary, based on the number of completed stories. this resulted in a dataset of 2,335 stories (see quality control. we additionally assessed the dataset by employing two indonesian university students that were not involved in the data construction. 3 based on 100 random samples, we asked each worker to choose the correct fifth sentence for a given four-sentence premise, and found that both 1 see appendix for more details. 2 the monthly minimum wage in indonesia is around rp 4,000,000, and the workload to write 500 short stories equates to roughly 5-days of full-time work. 3 we paid rp 150,000 to each. we used an indobert model @xcite to train pos and ner models, based on the datasets of @xcite and @xcite , resp., and used them to predict verb, person, location, and organization tags. 5 first, we found that the dataset contains 21,447 verb tokens (3,723 unique tokens), with the top-3 most frequent verbs having a frequency of 2% (see figure 2 in appendix). we also observe that person, location, and organization nes are mostly local indonesian expressions, with common person names being reno and mamat, and organization names being kai and bobo, as captured in ## experimental setup similar to @xcite experiments in english commonsense reasoning, we conducted two tasks: (1) a classification task to predict the correct ending; and (2) a single-sentence generation task to complete the story. we perform these two tasks in two settings: (1) monolingual training, and (2) zero-shot cross-lingual transfer, between indonesian and english. the data split is presented in ## results and analysis classification. in overall, the best performance is achieved by in-dobert when using all sentences (s 1 → s 4 ) as context, outperforming mbert with 81% accu-8 each worker was paid rp 250,000. racy. compared to the english story cloze test, our corpus is arguably harder, as @xcite reported bert accuracies of 78% and 88.1% in the english corpus when using none and s 1 → s 4 as the premise. we acknowledge that there is a spurious correlation of sentence-5 candidates with the commonsense labels, indicated by indobert accuracy of 76.1% when having context of none. this phenomenon is worse in the english dataset @xcite where the bert accuracy of using context of none is 88.1% @xcite . in to further observe whether the transferability is affected by factors beyond language, we translate the training data with google translate. in although zero-shot cross-lingual transfer of @xmath6, we notice that mbart is capable of generating plausibly codemixed sentences made up of indonesian and english @xcite . based on our manual evaluation on the same 50 indonesian test set, we found that 41% of generated fifth sentences contain code-mixing, of which 75% are naturalistic (see ## conclusion in this paper, we introduced the first indonesian story cloze dataset, and performed preliminary analysis in classification and generation settings in two scenarios: monolingual training and zeroshot cross-lingual transfer between indonesian and 9 indonesian cuisine. 10 please see appendix for the adequacy and fluency scores (including pearson correlations) of each translation system. english. from both experiments, we found that the cross-lingual transfer of commonsense from english to indonesian does not perform well, motivating the construction of commonsense reasoning resources in different languages.