Revisiting Pathologies of Neural Models under Input Reduction (2023)

Abstract: We revisit the question of why neural models tend to produce high-confidence predictions on inputs that appear nonsensical to humans. Previous work has suggested that the models fail to assign low probabilities to such inputs due to model overconfidence. We evaluate various regularization methods on fact verification benchmarks and find that this problem persists even with well-calibrated or underconfident models, suggesting that overconfidence is not the only underlying cause. We also find that regularizing the models with reduced examples helps improve interpretability but comes with the cost of miscalibration. We show that although these reduced examples are incomprehensible to humans, they can contain valid statistical patterns in the dataset utilized by the model.

## introduction during the development stage, we put much effort into tuning neural models to achieve high accuracy on held-out data. however, when deploying such tuned models in real-world scenarios, it is also important for them to be reliable. for example, when a fact verification model judges that a claim is true with a confidence of 0.95, it should have a 95% chance of being correct. meanwhile, lowconfidence predictions can be passed onto humans to be double-checked manually. if the model can align its confidence with the correctness, it is considered calibrated. despite achieving human-level performance on various tasks, recent studies @xcite @xcite have shown that modern neural models tend to be miscalibrated. miscalibration further reveals an anomaly of neural models in which they tend to produce highconfidence predictions on inputs that appear nonsensical to humans. figure 1 shows examples from evidence conclusions : in our cohort of covid-19 patients, immunosuppression was associated with a lower risk of moderate-severe ards. ## input reduction model interpretation methods offer explanations for model predictions @xcite @xcite . the goal is to understand why the model made specific predictions. a brute-force method is to look at model weights, but they are incomprehensible. because most modern neural architectures (including ours) rely on attention mechanisms, attention weights over inputs are often used as explanations. however, subsequent ## regularization methods in this section, we review widely used regularization methods, inspect their properties, and introduce our extension. ## conclusion we revisited the pathological behaviors of neural models in which they tend to be overconfident on inputs that appear meaningless to humans. we first analyzed the commonly used fact verification benchmarks with input reduction @xcite and found that we could only shorten particular types of claims into a few tokens without changing the model's predictions. we explored various entropy regularization methods and also proposed our extensions. we found that regularizing the objective function with the reduced examples improves interpretability but deteriorates calibration. training neural models that use more meaningful features while being well-calibrated is an important direction for future work.