  Augmenting pre-trained language models with knowledge graphs (KGs) has
achieved success on various commonsense reasoning tasks. Although some works
have attempted to explain the behavior of such KG-augmented models by
indicating which KG inputs are salient (i.e., important for the model's
prediction), it is not always clear how these explanations should be used to
make the model better. In this paper, we explore whether KG explanations can be
used as supervision for teaching these KG-augmented models how to filter out
unhelpful KG information. To this end, we propose SalKG, a simple framework for
learning from KG explanations of both coarse (Is the KG salient?) and fine
(Which parts of the KG are salient?) granularity. Given the explanations
generated from a task's training set, SalKG trains KG-augmented models to solve
the task by focusing on KG information highlighted by the explanations as
salient. Across two popular commonsense QA benchmarks and three KG-augmented
models, we find that SalKG's training process can consistently improve model
performance.
