Do Children Texts Hold The Key To Commonsense Knowledge? (2022)

Abstract: Compiling comprehensive repositories of commonsense knowledge is a long-standing problem in AI. Many concerns revolve around the issue of reporting bias, i.e., that frequency in text sources is not a good proxy for relevance or truth. This paper explores whether children’s texts hold the key to commonsense knowledge compilation, based on the hypothesis that such content makes fewer assumptions on the reader’s knowledge, and therefore spells out commonsense more explicitly. An analysis with several corpora shows that children’s texts indeed contain much more, and more typical commonsense assertions. Moreover, experiments show that this advantage can be leveraged in popular language-model-based commonsense knowledge extraction settings, where task-unspecific fine-tuning on small amounts of children texts (childBERT) already yields significant improvements. This provides a refreshing perspective different from the common trend of deriving progress from ever larger models and corpora.

## introduction compiling commonsense knowledge (csk) is a long-standing problem in ai @xcite . automated text-extraction-based approaches to csk compilation, like knext @xcite , tuplekb @xcite , quasimodo @xcite , comet @xcite or ascent @xcite typically struggle with reporting bias @xcite , in particular an under-reporting of basic commonsense assertions. this is a crux of commonsense: if knowledge is assumed to be commonplace, such as that rain is wet or cars have wheels, there is little need to utter it explicitly. in contrast, statements that contradict commonsense are more frequently reported, leading to inappropriate images of the real world, e.g., that fires are more often cold than hot (e.g., 238 vs. 173 literal occurrences in the english wikipedia). children's material may partially counter this bias: as children's knowledge is still growing, seemingly obvious assertions may still be frequently expressed explicitly in such material. note that this is not a binary question of whether some knowledge is expressed or not, but more a ranking problem: prominent csk repositories often do not struggle to recall relevant statements (e.g., ascent @xcite contains 2800 assertions for "elephant"), but struggle to rank them properly. this is especially true for language-model based approaches of csk compilation @xcite , which by design can assign every token in the vocabulary a probability, but should do so in sensible order. this paper investigates (i) whether children's texts are a promising source for csk and (ii) whether small corpora can still boost knowledge extraction from large language models. specifically, we analyze the density and typicality of csk assertions in children's text corpora and show how fine-tuning existing language models on them can improve csk compilation. data and models, including a childbert variant, can be found at https://www.mpi-inf.mpg.de/ children-texts-for-commonsense . ## background prominent manual efforts towards csk compilation include conceptnet @xcite , atomic @xcite , and the integrated cskg @xcite . prominent text extraction projects are knext @xcite , tuplekb @xcite , quasimodo @xcite and ascent @xcite . each carefully selects extraction corpora, like wikipedia texts, user query logs, or targeted web search, to minimize extraction noise and maximize salience. nonetheless, all struggle with extracting very basic csk that is generally deemed 10954 too obvious to state explicitly. the utilized corpora are also small compared to what is typically used in language model pre-training. therefore, pre-trained language models (ptlms) have been employed directly for csk extraction in a setting called prompting/probing (cf. the lama benchmark) @xcite , where the bert lm showed promising results in predicting concept-net assertions. they can also be employed with supervision, like in the comet and the atomic 10x system @xcite . however, both ptlm-paradigms are grounded in frequencies observed in the original text corpora used for lm training, which are again subject to reporting bias. it generally ranges between 0 and 100, with 0-30 being considered difficult to read, 60-70 assumed standard, and above 80 easy. we investigate three children text corpora: 1. children book test (cbt) the cbt dataset @xcite ) contains 108 children books such as alice's adventures in wonderland extracted from the gutenberg project. it targets children around 12-14 years old and is about 30 mb in total. 2. c4-easy c4 @xcite is a cleaned version of common crawl's web crawl corpus that was used to train the t5 language model. it is approximately 305 gb in size. we derive c4-easy by restricting the corpus to documents with an fre greater than 80, retaining 40.827.011 documents, which are 11% of c4. 3. infantbooks we newly introduce the in-fantbooks dataset, composed of 496 books targeted at kids from 1-6 years. it is based on ebooks from websites like freekidsbooks.org, monkeypen.com and kidsworldfun.com, which we collected, transcribed, and cleaned. the final dataset consists of 496 books with 2 mb of text. foot_0 as a baseline, and to rule out that observed improvements stem only from general training on more data, we also compare with employing the whole c4 corpus. in to further explore the relation of text simplicity and csk density, we grouped c4 documents into buckets based on their fre. for a sample of 10k documents per bucket, figure 1 reports the perword frequencies of csk assertions, considering all spotted csk assertions (blue) or only distinct ones (red). the results are shown in figure 1 . as one can see, csk density increases significantly with easier readability, and only the most simple documents suffer from a lack of diversity (decrease in blue line). dataset mean freq./sent. mean freq./word c4 (sample) 9.99e -3 4.86e -4 cbt 1.03e -2 4.94e -4 c4-easy (sample) 1.10e -2 7.65e -4 infantbooks 3.07e -2 2.56e -3 ## csk generation we next evaluate whether pre-training on children texts helps in two ptlm-based methods for csk generation: (i) via prompting from a lm pretrained on those corpora (lama method) @xcite , and (ii) via supervised learning, based on a lm fine-tuned on those corpora (comet method) @xcite . ## conclusion our results positively confirm both starting questions: dedicated children corpora contain more typical csk assertions, and even small quality text corpora can boost the performance of large lms for csk extraction, especially concerning basic csk assertions. while the overall gains are still modest, these results affirm the role of data selection in commonsense knowledge compilation. along with the exploitation of even stronger pre-trained language models, the quest for relevant text corpora thus continues.