CN - A uto MIC : Distilling C hinese Commonsense Knowledge from Pretrained Language Models (2022)

Abstract: Commonsense knowledge graphs (CKGs) are increasingly applied in various natural language processing tasks. However, most existing CKGs are limited to English, which hinders related research in non-English languages. Meanwhile, directly generating commonsense knowledge from pretrained language models has recently received attention, yet it has not been explored in non-English languages. In this paper, we propose a large-scale Chinese CKG generated from multilingual PLMs, named as **CN-AutoMIC**, aiming to fill the research gap of non-English CKGs. To improve the efficiency, we propose generate-by-category strategy to reduce invalid generation. To ensure the filtering quality, we develop cascaded filters to discard low-quality results. To further increase the diversity and density, we introduce a bootstrapping iteration process to reuse generated results. Finally, we conduct detailed analyses on CN-AutoMIC from different aspects. Empirical results show the proposed CKG has high quality and diversity, surpassing the direct translation version of similar English CKGs. We also find some interesting deficiency patterns and differences between relations, which reveal pending problems in commonsense knowledge generation. We share the resources and related models for further study.

## introduction compiling large-scale commonsense knowledge resources is a long-term goal of the ai community. recent efforts focus on constructing commonsense knowledge graphs (ckgs) via manually compiling @xcite @xcite or automatic extraction @xcite @xcite . these projects have shown benefits for a wide range of downstream tasks @xcite @xcite . however, most ckg projects are limited to english, which hinders further research in other languages. to go beyond such an english-centric trend in commonsense knowledge research, there are some challenging issues. first, directly translating english ckgs is not enough. for example, (personx is a little girl, xwant, to ask christmas presents) is a triple from a ckg crowdsourced by english users @xcite , but it is not common in non-christian cultures. in fact, such resources reflect only the commonsense perspectives of english-speaking communities. the translation will omit the cultural differences in other languages, and even implicitly exacerbate the english-centric bias @xcite . therefore, when creating ckgs for new languages, it is better to rely on native speakers and corpora. second, current common practices in english ckg construction, i.e. manually compiling and automatic extraction, are difficult to generalize. for manually compiling, creating human-authored ckgs for each new language is very expensive. for automatic extraction, current pipelines rely on english-specific hand-craft patterns or language processing tools, which are not available to other languages. therefore, when creating ckgs for new languages, the cost and availability of construction scheme should also be concerned. recent work reveals pretrained language models (plms) can be a new source to generate commonsense knowledge @xcite . the latest research indicates the ckg built from huge plms (e.g. gpt-3) can surpass the crowdsourced ones in quantity and quality @xcite , and only a small amount of human-authored data are required for prompting and filtering. interestingly, we find this way could be the ideal start point to construct ckgs for new languages, since plms can be trained on the corpora of target languages, and the generation process is low-cost and independent of language-specific tools. however, up to now, work in this thread has not extended to non-english languages. the main challenge of this paradigm is that the generation quality and diversity are often conflicting and difficult to control. to sample diverse results, the generation should be run extensive times, and a large number of generated results are invalid and need to be filtered out. for new languages, as there is often no comparable plm to @xcite in english, the generated results will be even noisier. therefore, we need to reduce unnecessary generation to increase the efficiency and take measures to ensure the quality. in this paper, we propose a framework to distill commonsense knowledge from multilingual pretrained language models. to increase the generation efficiency, we propose a generate-by-category strategy to reduce invalid generation. to ensure the filtering quality, we propose cascaded filters to discard low-quality results. to further increase the diversity and density, we introduce a bootstrapping process. based on the framework, we propose a large-scale chinese commonsense knowledge graph, cn-automic (automatically obtained machine commonsense). to the best of our knowledge, it is the first non-english ckg built from pretrained language models. the empirical results show the proposed ckg has high quality and diversity. we also discuss some interesting deficiencies that need further solutions. we summarize the contribution as follows. besides verifying the quality and diversity, we also get some valuable observations about the deficiencies of generation. considering generating commonsense knowledge with plms is still not fully explored, our findings can provide more insights into this topic. ## construction of cn-automic for clarity, in this section, we first show the overview of the construction task, then describe the construction process of cn-automic in detail. ## evaluation and analysis in this section, we evaluate and analyze the resources in three parts. first, we comprehensively evaluate cn-automic in size, quality and diversity. in this step, we also evaluate the commonsense model (cn-comet) trained on it. second, we conduct specific analyses for different construction steps. third, we inspect the cases of culturespecific commonsense knowledge and generation deficiencies. ## discussion what is the upper-bound size of the generation? the plms can conduct ever-lasting generation, but it seems there is a soft upper bound. the results generated later are easy to repeat previous results. therefore, the cost of novel results would gradually increase and eventually become unaffordable. in this work, we have generated tens of millions of triples. it has still not reached the limit, but similar or repeated content has appeared in large numbers. for example, during generating head phrases, we observe that the proportion of non-repetitive results keeps descending (figure 5 ). if plms have already learned commonsense knowledge, why is it necessary to extract the knowledge from it first, according to the results in this paper, the quality of direct generated results is still poor. it indicates that even for a model with 13b+ parameters, the learned knowledge is still rough and full of mistakes. a distilling process can distinguish the clean knowledge and benefit smaller applicable models. besides, recent work shows that even though explicitly training plms with knowledge, there is no guarantee that they can actually use such knowledge in target tasks @xcite . therefore, we can exploit explicit symbolic knowledge as auxiliary information. ## conclusion considering the dilemma of lacking non-english commonsense knowledge resources, in this paper, we propose cn-automic, the first chinese commonsense knowledge graph that is totally generated by pretrained language models. during the construction, we use prompting generation to obtain head and tail items, as well as introduce categorized generation, cascaded filtering and bootstrapping iteration to improve the quantity, quality and diversity. through human evaluation, the resource is shown to have better quality than directly translated resources from english language. we discuss the culture-related phenomena and common deficiency patterns in the generated knowledge graph. although our work is limited to chinese, the basic framework and methods can be used to populate ckgs in more languages.