m CSQA : Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans (2024)

Abstract: It is very challenging to curate a dataset for language-specific knowledge and common sense in order to evaluate natural language understanding capabilities of language models. Due to the limitation in the availability of annotators, most current multilingual datasets are created through translation, which cannot evaluate such language-specific aspects. Therefore, we propose Multilingual CommonsenseQA (mCSQA) based on the construction process of CSQA but leveraging language models for a more efficient construction, e.g., by asking LM to generate questions/answers, refine answers and verify QAs followed by reduced human efforts for verification. Constructed dataset is a benchmark for cross-lingual language-transfer capabilities of multilingual LMs, and experimental results showed high language-transfer capabilities for questions that LMs could easily solve, but lower transfer capabilities for questions requiring deep knowledge or commonsense. This highlights the necessity of language-specific datasets for evaluation and training. Finally, our method demonstrated that multilingual LMs could create QA including language-specific knowledge, significantly reducing the dataset creation cost compared to manual creation. The datasets are available at https://huggingface.co/datasets/yusuke1997/mCSQA.

## introduction can you choose the correct answer in they cannot be solved without such knowledge, as the translations consolidate the nuances into a single word. a few major languages such as english, and thus, many languages lack such datasets. when focusing on the cross-lingual capability of lms, datasets created from scratch in multiple languages are lim-1 14182 ited, and currently, evaluations mostly use datasets created through translation. however, as can be seen from the example in to tackle this problem, as shown in figure 1 , we propose a method to efficiently create multilingual nlu datasets from multilingual resources by replacing some of the manual annotation processes with generative multilingual lms. in this study, we focus on commonsenseqa (csqa) @xcite , a dataset for evaluating commonsense reasoning capabilities within nlu evaluations. csqa is a major commonsense reasoning question-answering dataset manually created from the multilingual knowledge base concept-net @xcite . however, due to such limitations, csqa has been created from scratch only in english and japanese, jcommonsenseqa (jcsqa) @xcite . therefore, we create a multilingual commonsenseqa (mcsqa) that extends csqa to eight languages foot_0 using our proposed method. furthermore, we evaluated the cross-lingual language-transfer capabilities of multilingual lms focusing on language-specific common sense and knowledge using mcsqa. the results showed high language-transfer capabilities for questions that lms could easily solve, but lower transfer capabilities for questions requiring deep knowledge or commonsense. the total cost per question in mcsqa was reduced to one-hundredth of that for csqa. to summarize, our contributions are as follows: commonsense reasoning task this task evaluates how an lm can understand and infer object recognition, visual information, and cultural or societal common sense, which are not typically described in textual information. csqa is a multiplechoice question task that asks for the most plausible choice as an answer with some variants: jc-sqa is in japanese, commonsenseqa 2.0 @xcite is a more challenging dataset, ecqa @xcite requires explaining the process of deriving an answer, etc. there exist other types of commonsense tasks: copa @xcite and balancedcopa @xcite ask about causal relationships between everyday events; socialiqa @xcite asks about social common sense; piqa @xcite evaluates procedural knowledge; hot-potqa @xcite requires multi-hop inference; drop @xcite captures arithmetic operation capabilities; and tasks like understanding language information @xcite @xcite , understanding causal relationships within documents @xcite @xcite @xcite , and common-gen @xcite , which asks to generate common sentences from given keywords. the above datasets primarily focus on english, but there exist datasets in japanese @xcite @xcite , chinese @xcite @xcite , russian @xcite , and indonesian @xcite . the evaluation datasets are almost exclusively created through three methods, as shown in ## datasets creation our mcsqa construction involves three main steps (see figure 2 ): extraction of sub-graphs from conceptnet, creation of question and choice pairs with lms, and verification of their quality by both lms and humans. we basically follow the creation processes of csqa and jcsqa, but modified to allow for unified processing to support multiple languages. ## evaluation for mcsqa we verify that the mcsqa dataset is meaningful for evaluating the common sense reasoning capability of lms by using various multilingual lms. ## conclusion and future directions we proposed an efficient and low-cost method for creating nlu datasets from structured data by utilizing generative lms as an alternative to traditional human annotation, often crowdsourced. inspired by csqa and jcsqa, we created the multilingual commonsense reasoning task dataset, mcsqa, using gpt-3.5 from the structured multilingual knowledge base conceptnet. we demonstrated that mcsqa is useful for evaluating the commonsense reasoning capabilities of lms. we also analyzed the language-transfer capability beyond english with mcsqa and examined the language-specific learning from two aspects: question difficulty and language information. moreover, our study has shown that the use of multilingual lms enables the construction of multilingual datasets. therefore, our method can significantly reduce human labor and financial costs. in this study, we used a single multilingual lm, but since each step is independent, it is possible to replace the lm used in each step with another one. furthermore, each step can be applied modularly to other methods, making it possible to use this method for creating multilingual datasets, such as those expanded through translation and manual refinement @xcite . we aim to extend this method to other types of commonsense reasoning tasks and nlu tasks, to efficiently create multilingual data and conduct a more comprehensive analysis of transfer capabilities across a broader range of tasks and languages. we focused on language-specific commonsense, but languages are shared across various regions. for example, english is spoken in the united states, the united kingdom, india, australia, and many other regions each of which is geographically distant and diverse in terms of climate, food, and culture. therefore, it will be necessary to create more detailed commonsense tasks that consider cultural differences rather than just language such as @xcite al. (2023