Do Neural Language Models Overcome Reporting Bias? (2020)

Abstract: Mining commonsense knowledge from corpora suffers from reporting bias, over-representing the rare at the expense of the trivial (Gordon and Van Durme, 2013). We study to what extent pre-trained language models overcome this issue. We find that while their generalization capacity allows them to better estimate the plausibility of frequent but unspoken of actions, outcomes, and properties, they also tend to overestimate that of the very rare, amplifying the bias that already exists in their training corpus.

## introduction apart from several notable efforts to collect commonsense knowledge from experts @xcite or through crowdsourcing @xcite , most work has been on extracting such knowledge from large text corpora @xcite . while the latter approach is scalable and low cost, it also suffers from reporting bias: due to grice's conversational maxim of quantity @xcite , people rarely state the obvious, thus many trivial facts ("people breathe") are rarely mentioned in text, while uncommon events ("people murder") are reported disproportionately @xcite . traditionally, knowledge acquisition from text was extractive. in recent years, the generalization capacity of neural language models (lms) and their ability to aggregate knowledge across contexts have facilitated estimating the plausibility of facts, even when they don't appear in the corpus explicitly. recent pre-trained lms such as gpt-2 @xcite and bert @xcite , trained on massive texts, dominate the nlp leaderboards, and are considered a source of commonsense knowledge @xcite . does this mean that pre-trained lms overcome reporting bias? in this paper we revisit the experiments conducted by @xcite (henceforth g&v), applying them to various pre-trained lms (based on the nature of the experiment, we test either masked lms or standard left-to-right lms). we find that lms, compared to extractive methods: foot_0 1. provide a worse estimate of action frequency, mostly due to overestimating very rare actions. 2. predict both expected outcomes as well as sensational and unlikely outcomes. 3. are capable of learning associations between concepts and their properties indirectly, but tend to over-generalization, which leads to confusing semantically-similar but mutually exclusive values. ## actions and events g&v demonstrate the discrepancy between corpus occurrences and actual action frequency by showing that if you believe the corpus, people murder more than they breathe. breathing is an activity we take for granted and thus rarely talk about @xcite . that murder is frequent in the corpus is a reflection of the same issue: we talk more about uncommon or newsworthy events @xcite . we follow g&v's qualitative analysis of actions and events performed by or which happen to people by comparing real-world frequency to corpus-based and lm-based frequency. we estimate real-world figure 1 : frequency of actions performed or occurring to people during their lifetime from very frequent (daily), through once in a lifetime events, to very rare (don't happen to most people). note that actual frequencies of rare events are too small to show. see appendix a for the exact frequencies. frequency (e.g. how many times does a person breathe in their lifetime?) from published statistics based on us data, as detailed in appendix a. corpus frequency is computed using the google n-gram corpus @xcite . specifically, we compute the normalized frequency of the verbs appearing in the 3-gram "person @xmath0, falling back to the bigram "@xmath1. we use spacy to determine parts of speech, keeping non auxiliary verbs @xcite . while lm scores don't represent frequency or probability, they are often used in practice as a proxy for plausibility. thus, we would expect lm scores to correlate with real-world frequency. we query masked lms for substitutes of the mask in several templates describing actions, foot_1 and left-to-right lms by greedily decoding the next token (e.g. for "the person is"), taking the maximum score for each word across templates. specifically, we use bert large uncased @xcite , roberta large @xcite , and gpt-2 xl @xcite from the transformers package @xcite . we keep the non auxiliary verbs among the top 5000 predictions. 3 figure 1 visualizes the relative frequency of each action as estimated by the various sources, where the scores for all actions are normalized for each source. actions are sorted by their real-world frequency from very frequent to very rare. first, we observe that lms assign non-zero scores for all actions, as opposed to the non-smoothed corpus frequencies from google ngrams. however, the scores they produce diverge further from the actual distribution, measuring with kl-divergence: google ngrams -2.94, bert and gpt-2 -3.77, and roberta -3.08. lms produce a more accurate estimate for some frequent actions (blinking, eating) but worse for others (thinking, breathing). at the same time, lms also exaggerate the frequencies of rare events (e.g. dying), producing estimates not only higher than the actual frequency but even higher than the corpus frequency. the same patterns emerge for both lms, but some exceptions stand out. for example, bert overestimates the frequency of dying, which may be due to being trained on wikipedia, which consists of many entries describing historically important-and dead-people. roberta, on the other hand, which ## event outcomes g&v argue that an event outcome is more likely to be mentioned in text if it's not certain. for instance, "the man turned on the faucet. the water started running in a steady stream." makes an awfully boring story, while "water gushed out of the sink" builds up to a turn in events. do lms learn the proportional outcome distribution in the corpus, or can they overcome it by implicitly learned commonsense? a good testbed for event outcomes is the copa dataset (choice of plausible alternatives) @xcite . given an event (context), the goal is to predict its cause or effect among two candidate answers. we focus on lms typically used for generation: gpt @xcite , gpt-2 @xcite , and xlnet @xcite . ## properties according to g&v, people are more likely to state unusual properties of a concept (blue pencil) than usual ones (yellow pencil). recently, @xcite studied lms' ability to associate concepts with their properties, by providing the lm the concept and predicting the properties and vice versa. overall, lms performed reasonably well, with roberta outperforming bert. both performed better on encyclopedic and functional properties ("a bear is an animal") than on perceptual properties, which are less often mentioned in text @xcite . we hypothesize that while lms are to some extent capable of learning association between concepts and their properties indirectly by aggregating across contexts, during this process, they often overgeneralize, predicting semantically-similar but mutually exclusive values. we verify that by evaluating bert and roberta's ability to predict colors. we constructed a list of 11 common colors and extracted all sentences in wikipedia in which a color modifies a noun, masking the color tokens (e.g. "a bear is [mask]"). we then split the data into train (1,169,590 sentences) and test (10,000 sentences). ## related work commonsense in pre-trained lms. there is ongoing research on extracting commonsense knowledge from pre-trained lms, providing mixed results. on the one hand, @xcite and @xcite somewhat successfully used pre-trained lms to complete commonsense kbs. on the other hand, @xcite have shown that lms are limited in their ability to generate accurate factual knowledge, and @xcite and @xcite pointed out that lms are not sensitive to negation, resulting in generating incorrect facts ("birds can't fly"). finally, @xcite showed that despite being noisy, knowledge generated by lms can be used to improve performance on commonsense tasks. similarly to our color experiment, @xcite developed a lm-based relation classification model that included a color relationship. the model starts with a seed of known word pairs for a given relationship, uses it to find template sentences indicative of the relationship, and fine-tunes bert on these retrieved sentences. their experiment had a different purpose from ours, in which we probed the lms for knowledge already captured by their pre-training phase. learning from other modalities. much of our world knowledge is innate or acquired through modalities such as vision, including physical commonsense ("physical objects can't be in different places at the same time") and social commonsense ("people do and say things for reasons"). there has been little work on learning meaning from other modalities @xcite , but there is a shared understanding in the community that this is the imperative next step @xcite . ## conclusion we show that pre-trained lms to some extent overcome reporting bias in the sense that they possess knowledge that wasn't explicitly stated, including trivial facts. unfortunately, they also over-represent rare and newsworthy events, amplifying the bias that already exists in their training corpus. the results in this paper are in line with prior work that showed that lms amplify social bias @xcite and knowledge about named entities that are prominent in the corpus @xcite . going forward, it is important to study how the choice of training corpus, model size, and other factors affect the type and extent of biases the lm would have.