  Singh et al. (2020) point out the dangers of contextual bias in visual
recognition datasets. They propose two methods, CAM-based and feature-split,
that better recognize an object or attribute in the absence of its typical
context while maintaining competitive within-context accuracy. To verify their
performance, we attempted to reproduce all 12 tables in the original paper,
including those in the appendix. We also conducted additional experiments to
better understand the proposed methods, including increasing the regularization
in CAM-based and removing the weighted loss in feature-split. As the original
code was not made available, we implemented the entire pipeline from scratch in
PyTorch 1.7.0. Our implementation is based on the paper and email exchanges
with the authors. We found that both proposed methods in the original paper
help mitigate contextual bias, although for some methods, we could not
completely replicate the quantitative results in the paper even after
completing an extensive hyperparameter search. For example, on COCO-Stuff,
DeepFashion, and UnRel, our feature-split model achieved an increase in
accuracy on out-of-context images over the standard baseline, whereas on AwA,
we saw a drop in performance. For the proposed CAM-based method, we were able
to reproduce the original paper's results to within 0.5$\%$ mAP. Our
implementation can be found at
https://github.com/princetonvisualai/ContextualBias.
