RICA : Evaluating Robust Inference Capabilities Based on Commonsense Axioms (2021)

Abstract: Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference using Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense.

## introduction smooth and effective communication requires the ability to make various forms of commonsense inferences @xcite . when a friend texts, "i'm going to perform in front of thousands tomorrow," you may reply reassuringly, "deep breaths, you'll do great!" implicit to this communication is a commonsense logical inference that a person performing in front of a crowd may feel anxious, and that a reassuring remark helps ease anxiety (figure 1 ). a growing body of literature @xcite text message: "i'm going to perform in front of thousands tomorrow…" ## the rica challenge the rica challenge is posed as a set of textual statements (sentences), each expressing a latent commonsense relationship in the "premiseconclusion" format (see stage 5 in fig. 3 for examples). these statements use generated novel entities such as "prindag" and "fluberg" instead of real-world entities such as "thimble" and "elephant" to separate factual recalling from reasoning. each statement can be viewed as an instantiation of a commonsense principle, such as "smaller objects cannot contain larger objects." we express these commonsense principles in first-order logic, further generalizing statements through the use of general predicates for object properties (e.g., size) and object-object relations (e.g., containment). we turn these logical formulae into the associated textual statements using a set of perturbation operators and a conversion module, which together produce a logically-equivalent set of commonsense statements. in the rest of this sec- ## results and analysis we examine the performance of multiple language models on each evaluation setting on our probe data, including zero-shot and fine-tuning on various splits, and present ablation studies to analyze performance more thoroughly. all of our results are averages of testing on 3 seeds. ## conclusion we design rica as an ai challenge to test robust inference capabilities on linguistically-varied probes covering different commonsense axioms. rica is built on a systematic process to construct probes using fol formulae, perturbation operators, and novel entities. following this approach, we generate and verify more than 10k statements from 1.7k axioms and test multiple ptlms in various settings. we find that ptlms perform on par with random guessing on zero-shot setting, have strong positivity bias, and are not robust under linguistic perturbations. net and atomic, we follow their github repo 5 . we use ernie from their original github 6 . fine-tuning details we fine-tune bert-baseuncased, bert-large-uncased, roberta-base, and roberta-large based on happytransformers 7 framework, using a consistent learning rate of 1e-5. we fine-tune gpt-2 based on huggingface transformers library's example code 8 , using their default parameters. we train them on one nvidia quadro rtx 6000 gpu for 10 epochs and after each epoch we test the finetuned model on our validation set, and save the model with the highest validation set performance. fine-tuning roberta-base and gpt-2 takes around 30 minutes for each epoch and roberta-large takes around 1 hour. the best validation performance for roberta-base is the fourth epoch, with perplexity 1.3378140926361084 and evaluation loss': 0.2910370217429267. for roberta-large, the best is epoch 5, with perplexity 1.3949965238571167 and evaluation loss 0.3328918993473053. for gpt-2, the best is epoch 3, with perplexity 1.2786548795017285. interpretation details we use the alleninterpret demo 9 . to identify important context words, we run the algorithm over the same probe for 5 times, each with different entity names, and select the words that are ranked in the top 5 most important words at least 3 times. we find that the interpretations are not very consistent as the most important words change when we input the same sentence for multiple times and will also change when different names are used, so we conduct 5 trials with different names for each probe and pick the words that appear in the majority of the trials.