Symbolic knowledge (e.g., entities, relations, and facts in a knowledge graph) has become an increasingly popular component of neural-symbolic models applied to machine learning tasks, such as question answering and recommender systems. Besides improving downstream performance, these symbolic structures (and their associated attention weights) are often used to help explain the model's predictions and provide " insights " to practitioners . In this paper, we question the faithfulness of such symbolic explanations . We demonstrate that, through a learned strategy (or even simple heuristics), one can produce deceptively perturbed symbolic structures which maintain the downstream performance of the original structure while significantly deviating from the original semantics . In particular, we train a reinforcement learning policy to manipulate relation types or edge connections in a knowledge graph, such that the resulting downstream performance is maximally preserved. Across multiple models and tasks, our approach drastically alters knowledge graphs with little to no drop in performance. These results raise doubts about the faithfulness of explanations provided by learned symbolic structures and the reliability of current neural-symbolic modelsin leveraging symbolic knowledge .
Knowledge graphs (KGs) have helped neural-symbolic models improve performance on various knowledge-intensive tasks, like question answering and item recommendation. By using attention over the KG, such models can also " explain " which KG information was most relevant for making a given prediction . In this paper, we question whether these models are really behaving as we expect . We demonstrate that, through a reinforcement learning policy (or even simple heuristics), one can produce deceptively perturbed KGs which maintain the downstream performance of the original KG while significantly deviating from the original semantics and structure. Our findings raise doubts about KG-augmented models' ability to leverage KG information and provide plausible explanations .
R
Symbolic knowledge (e.g., entities, relations, and facts in a knowledge graph) has become an increasingly popular component of
Knowledge graphs (KGs) have helped
R
applied to machine learning tasks, such as
improve performance on various knowledge-intensive tasks, like
R
recommender systems. Besides improving downstream performance, these symbolic structures (and their associated attention weights) are often used to help explain the model's predictions and provide
item recommendation. By using attention over the KG, such models can also
R
insights
explain
R
to practitioners
which KG information was most relevant for making a given prediction
R
the faithfulness of such symbolic explanations
whether these models are really behaving as we expect
R
learned strategy
reinforcement learning policy
R
symbolic structures
KGs
R
structure
KG
R
. In particular, we train a reinforcement learning policy to manipulate relation types or edge connections in a knowledge graph, such that the resulting downstream performance is maximally preserved. Across multiple models and tasks, our approach drastically alters knowledge graphs with little to no drop in performance. These results
and structure. Our findings
R
the faithfulness of explanations provided by learned symbolic structures and the reliability of current neural-symbolic modelsin leveraging symbolic knowledge
KG-augmented models' ability to leverage KG information and provide plausible explanations