Original: G LOBAL E XPLAINABILITY OF GNN S VIA L OGIC C OMBINATION OF L EARNED C ONCEPTS
Revised: G LOBAL E XPLAINABILITY OF GNN S VIA L OGIC C OMBINATION OF L EARNED C ONCEPTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While instance-level explanation of GNN is a well-studied problem with plenty of approaches being developed, providing a global explanation for the behaviour of a
Revised: While instance-level explanation of GNN is a well-studied problem with plenty of approaches being developed, providing a global explanation for the behaviour of a

Original: GNN is much less explored, despite its potential in interpretability and debugging.
Revised: GNN is much less explored, despite its potential in interpretability and debugging.

Original: Existing solutions either simply list local explanations for a given class, or generate a synthetic prototypical graph with maximal score for a given class, completely missing any combinatorial aspect that the GNN could have learned.
Revised: Existing solutions either simply list local explanations for a given class, or generate a synthetic prototypical graph with maximal score for a given class, completely missing any combinatorial aspect that the GNN could have learned.

Original: In this work, we propose GLGExplainer (Global Logic-based GNN Explainer), the first Global
Revised: In this work, we propose GLGExplainer (Global Logic-based GNN Explainer), the first Global

Original: Explainer capable of generating explanations as arbitrary Boolean combinations of learned graphical concepts.
Revised: Explainer capable of generating explanations as arbitrary Boolean combinations of learned graphical concepts.

Original: GLGExplainer is a fully differentiable architecture that takes local explanations as inputs and combines them into a logic formula over graphical concepts, represented as clusters of local explanations.
Revised: GLGExplainer is a fully differentiable architecture that takes local explanations as inputs and combines them into a logic formula over graphical concepts, represented as clusters of local explanations.

Original: Contrary to existing solutions, GLGExplainer provides accurate and human-interpretable global explanations that are perfectly aligned with ground-truth explanations (on synthetic data) or match existing domain knowledge (on real-world data).
Revised: Contrary to existing solutions, GLGExplainer provides accurate and human-interpretable global explanations that are perfectly aligned with ground-truth explanations (on synthetic data) or match existing domain knowledge (on real-world data).

Original: Extracted formulas are faithful to the model predictions, to the point of providing insights into some occasionally incorrect rules learned by the model, making GLGExplainer a promising diagnostic tool for learned GNNs.
Revised: Extracted formulas are faithful to the model predictions, to the point of providing insights into some occasionally incorrect rules learned by the model, making GLGExplainer a promising diagnostic tool for learned GNNs.

Original:  I NTRODUCTION & R ELATED W ORK
Revised:  I NTRODUCTION & R ELATED W ORK

Original: Graph Neural Networks (GNNs) have become increasingly popular for predictive tasks on graph structured data.
Revised: Graph Neural Networks (GNNs) have become increasingly popular for predictive tasks on graph structured data.

Original: However, as many other deep learning models, their inner working remains a black box.
Revised: However, as many other deep learning models, their inner working remains a black box.

Original: The ability to understand the reason for a certain prediction represents a critical requirement for any decision-critical application, thus representing a big issue for the transition of such algorithms from benchmarks to real-world critical applications.
Revised: The ability to understand the reason for a certain prediction represents a critical requirement for any decision-critical application, thus representing a big issue for the transition of such algorithms from benchmarks to real-world critical applications.

Original: Over the last years, many works proposed Local Explainers (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021;
Revised: Over the last years, many works proposed Local Explainers (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021;

Original: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019) to explain the decision process of a GNN in terms of factual explanations, often represented as subgraphs for each sample in the dataset.
Revised: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019; Magister et al., 2021) to explain the decision process of a GNN in terms of factual explanations, often represented as subgraphs for each sample in the dataset.

Original: We leave to Yuan et al.
Revised: We leave to Yuan et al.

Original: a detailed overview about Local Explainers, who recently proposed a taxonomy to categorize the heterogeneity of those.
Revised: a detailed overview about Local Explainers, who recently proposed a taxonomy to categorize the heterogeneity of those.

Original: Overall, Local Explainers shed light over why the network predicted a certain value for a specific input sample.
Revised: Overall, Local Explainers shed light over why the network predicted a certain value for a specific input sample.

Original: However, they still lack a global understanding of the model.
Revised: However, they still lack a global understanding of the model.

Original: Global Explainers, on the other hand, are aimed at capturing the behaviour of the model as a whole, abstracting individual noisy local explanations in favor of a single robust overview of the model.
Revised: Global Explainers, on the other hand, are aimed at capturing the behaviour of the model as a whole, abstracting individual noisy local explanations in favor of a single robust overview of the model.

Original: Nonetheless, despite this potential in interpretability and debugging, little has been done in this direction.
Revised: Nonetheless, despite this potential in interpretability and debugging, little has been done in this direction.

Original: GLocalX (Setzu et al., 2021) is a general solution to produce global explanations of black-box models by hierarchically aggregating local explanations into global rules.
Revised: GLocalX (Setzu et al., 2021) is a general solution to produce global explanations of black-box models by hierarchically aggregating local explanations into global rules.

Original: This solution is however not readily applicable to GNNs as it requires local explanations to be expressed as logical rules.
Revised: This solution is however not readily applicable to GNNs as it requires local explanations to be expressed as logical rules.

Original: Yuan et al. Yuan et al.
Revised: Yuan et al. Yuan et al.

Original:  proposed XGNN, which frames the Global Explanation problem for GNNs as a form of input optimization (Wu et al., 2020), using policy gradient to generate synthetic prototypical graphs for each class.
Revised: (2020) proposed XGNN, which frames the Global Explanation problem for GNNs as a form of input optimization (Wu et al., 2020), using policy gradient to generate synthetic prototypical graphs for each class.

Original: The approach requires prior domain knowledge, which is not always available, to drive the generation of valid prototypes.
Revised: The approach requires prior domain knowledge, which is not always available, to drive the generation of valid prototypes.

Original: Additionally, it cannot identify any compositionality in the returned explanation, and has no principled way to generate alternative explanations for a given class.
Revised: Additionally, it cannot identify any compositionality in the returned explanation, and has no principled way to generate alternative explanations for a given class.

Original: Indeed, our experimental evaluation shows that XGNN fails to generate meaningful global explanations in all the tasks we investigated.
Revised: Indeed, our experimental evaluation shows that XGNN fails to generate meaningful global explanations in all the tasks we investigated.

Original: Concept-based Explainability (Kim et al., 2018; Ghorbani et al., 2019;
Revised: Concept-based Explainability (Kim et al., 2018; Ghorbani et al., 2019;

Original: Yeh et al., 2020) is a parallel line of research where explanations are constructed using “concepts” i.e., intermediate, high-level and semantically meaningful units of information commonly used by humans to explain their decisions.
Revised: Yeh et al., 2020) is a parallel line of research where explanations are constructed using “concepts” i.e., intermediate, high-level and semantically meaningful units of information commonly used by humans to explain their decisions.

Original: Concept Bottleneck Models (Koh et al., 2020) and Prototypical Part Networks (Chen et al., 2019a) are two popular architectures that leverage concept learning to learn explainable-by-design neural networks.
Revised: Concept Bottleneck Models (Koh et al., 2020) and Prototypical Part Networks (Chen et al., 2019a) are two popular architectures that leverage concept learning to learn explainable-by-design neural networks.

Original: In addition, similarly to Concept Bottleneck Models, Logic Explained Networks (LEN) (Ciravegna et al., 2021a) generate logic-based explanations for each class expressed in terms of a set of input concepts.
Revised: In addition, similarly to Concept Bottleneck Models, Logic Explained Networks (LEN) (Ciravegna et al., 2021a) generate logic-based explanations for each class expressed in terms of a set of input concepts.

Original: Such concept-based classifiers improve human understanding as their input and output spaces consist of interpretable symbols (Wu et al., 2018; Ghorbani et al., 2019; Koh et al., 2020).
Revised: Such concept-based classifiers improve human understanding as their input and output spaces consist of interpretable symbols (Wu et al., 2018; Ghorbani et al., 2019; Koh et al., 2020).

Original: Those approaches have been recently adapted to GNNs (Zhang et al., 2022;
Revised: Those approaches have been recently adapted to GNNs (Zhang et al., 2022;

Original: Georgiev et al., 2022; Magister et al., 2022).
Revised: Georgiev et al., 2022; Magister et al., 2022).

Original: However, these solutions are not conceived for explaining already learned GNNs.
Revised: However, these solutions are not conceived for explaining already learned GNNs.

Original: Our contribution consists in the first Global Explainer for GNNs which i) provides a Global Explanation in terms of logic formulas, extracted by combining in a fully differentiable manner graphical concepts derived from local explanations; ii) is faithful to the data domain, i.e., the logic formulas, being derived from local explanations, are intrinsically part of the input domain without requiring any prior knowledge.
Revised: Our contribution consists in the first Global Explainer for GNNs which i) provides a Global Explanation in terms of logic formulas, extracted by combining in a fully differentiable manner graphical concepts derived from local explanations; ii) is faithful to the data domain, i.e., the logic formulas, being derived from local explanations, are intrinsically part of the input domain without requiring any prior knowledge.

Original: We validated our approach on both synthetic and real-world datasets, showing that our method is able to accurately summarize the behaviour of the model to explain in terms of concise logic formulas.
Revised: We validated our approach on both synthetic and real-world datasets, showing that our method is able to accurately summarize the behaviour of the model to explain in terms of concise logic formulas.

Original:  B ACKGROUND 2. G RAPH N EURAL N ETWORKS
Revised:  B ACKGROUND 2. G RAPH N EURAL N ETWORKS

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V , E ) with adjacency matrix A
Revised: ( V , E ) with adjacency matrix A

Original: where A ij = 1 if there exists an edge between nodes i and j , and a node feature matrix X ∈ R |V|× r where X i is the r -dimensional feature vector of node i , a GNN layer aggregates the node’s neighborhood information into a d -dimensional refined representation H ∈ R |V|× d .
Revised: where A ij = 1 if there exists an edge between nodes i and j , and a node feature matrix X ∈ R |V|× r where X i is the r -dimensional feature vector of node i , a GNN layer aggregates the node’s neighborhood information into a d -dimensional refined representation H ∈ R |V|× d .

Original: The most common form of aggregation corresponds to the GCN (Kipf & Welling, 2016) architecture, defined by the following propagation rule:
Revised: The most common form of aggregation corresponds to the GCN (Kipf & Welling, 2016) architecture, defined by the following propagation rule:

Original: [Equation]
Revised: [Equation]

Original: where ˜ A =
Revised: where ˜ A =

Original: A + I , ˜ D is the degree matrix relative to ˜ A , σ an activation function, and W ∈ R F × F is a layer-wise learned linear transformation.
Revised: A + I , ˜ D is the degree matrix relative to ˜ A , σ an activation function, and W ∈ R F × F is a layer-wise learned linear transformation.

Original: However, the form of Eq 1 is heavily dependent on the architecture and several variants have been proposed (Kipf & Welling, 2016;
Revised: However, the form of Eq 1 is heavily dependent on the architecture and several variants have been proposed (Kipf & Welling, 2016;

Original: Veli ˇ ckovi ´ c et al., 2017;
Revised: Veli ˇ ckovi ´ c et al., 2017;

Original: Gilmer et al., 2017)
Revised: Gilmer et al., 2017)

Original: 2. L OCAL E XPLAINABILITY FOR GNN S
Revised: 2. L OCAL E XPLAINABILITY FOR GNN S

Original: Many works recently proposed Local Explainers to explain the behaviour of a GNN (Yuan et al., 2022).
Revised: Many works recently proposed Local Explainers to explain the behaviour of a GNN (Yuan et al., 2022).

Original: In this work, we will broadly refer to all of those whose output can be mapped to a subgraph of the input graph (Ying et al., 2019;
Revised: In this work, we will broadly refer to all of those whose output can be mapped to a subgraph of the input graph (Ying et al., 2019;

Original: Luo et al., 2020; Yuan et al., 2021;
Revised: Luo et al., 2020; Yuan et al., 2021;

Original: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019).
Revised: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019).

Original: For the sake of generality, let LE XP ( f, G )
Revised: For the sake of generality, let LE XP ( f, G )

Original: =
Revised: =

Original: ˆ G be the weighted graph obtained by applying the local explainer LE XP to generate a local explanation for the prediction of the GNN f over the input graph G , where each ˆ A ij relative to ˆ G is the likelihood of the edge ( i, j ) being an important edge.
Revised: ˆ G be the weighted graph obtained by applying the local explainer LE XP to generate a local explanation for the prediction of the GNN f over the input graph G , where each ˆ A ij relative to ˆ G is the likelihood of the edge ( i, j ) being an important edge.

Original: By binarizing the output of the local explainer ˆ G with threshold θ ∈ R we achieve a set of connected components ¯ G i such that (cid:83)
Revised: By binarizing the output of the local explainer ˆ G with threshold θ ∈ R we achieve a set of connected components ¯ G i such that (cid:83)

Original: i ¯ G i ⊆
Revised: i ¯ G i ⊆

Original: ˆ G .
Revised: ˆ G .

Original: For convenience, we will henceforth refer to each of these ¯ G i as local explanation.
Revised: For convenience, we will henceforth refer to each of these ¯ G i as local explanation.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: The key contribution of this paper is a novel Global Explainer for GNNs which allows to describe the behaviour of a trained GNN f by providing logic formulas described in terms of humanunderstandable concepts (see Fig. 1).
Revised: The key contribution of this paper is a novel Global Explainer for GNNs which allows to describe the behaviour of a trained GNN f by providing logic formulas described in terms of humanunderstandable concepts (see Fig. 1).

Original: In the process, we use one of the available Local Explainers (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021; Vu & Thai, 2020;
Revised: In the process, we use one of the available Local Explainers (Ying et al., 2019; Luo et al., 2020; Yuan et al., 2021; Vu & Thai, 2020;

Original: Shan et al., 2021; Pope et al., 2019) to obtain a local explanation for each sample in the dataset.
Revised: Shan et al., 2021; Pope et al., 2019) to obtain a local explanation for each sample in the dataset.

Original: We then map those local explanations to some learned prototypes which will represent the final high-level concepts (e.g. motifs in a graph).
Revised: We then map those local explanations to some learned prototypes which will represent the final high-level concepts (e.g. motifs in a graph).

Original: Finally, for the formulas extraction, we input the vector of concept activations to an Entropy-based LEN (E-LEN) (Barbiero et al., 2021;
Revised: Finally, for the formulas extraction, we input the vector of concept activations to an Entropy-based LEN (E-LEN) (Barbiero et al., 2021;

Original: Ciravegna et al., 2021b) which is trained to match the predictions of f .
Revised: Ciravegna et al., 2021b) which is trained to match the predictions of f .

Original: In the following, we will describe every step in more detail.
Revised: In the following, we will describe every step in more detail.

Original: Local Explanations Extraction: The first step of our pipeline consists in extracting local explanations.
Revised: Local Explanations Extraction: The first step of our pipeline consists in extracting local explanations.

Original: In principle, every Local Explainer whose output can be mapped to a subgraph of the input sample is compatible with our pipeline (Ying et al., 2019;
Revised: In principle, every Local Explainer whose output can be mapped to a subgraph of the input sample is compatible with our pipeline (Ying et al., 2019;

Original: Luo et al., 2020; Yuan et al., 2021;
Revised: Luo et al., 2020; Yuan et al., 2021;

Original: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019).
Revised: Vu & Thai, 2020; Shan et al., 2021; Pope et al., 2019).

Original: Nonetheless, in this work, we relied on PGExplainer (Luo et al., 2020) since it allows the extraction of arbitrary disconnected motifs as explanations and it gave excellent results in our experiments.
Revised: Nonetheless, in this work, we relied on PGExplainer (Luo et al., 2020) since it allows the extraction of arbitrary disconnected motifs as explanations and it gave excellent results in our experiments.

Original: The result of this preprocessing step consists in a list D of local explanations, which are provided as input to the GLGExplainer architecture.
Revised: The result of this preprocessing step consists in a list D of local explanations, which are provided as input to the GLGExplainer architecture.

Original: More details about the binarization are available in Section 4.2.
Revised: More details about the binarization are available in Section 4.2.

Original: Embedding Local Explanations: The following step consists in learning an embedding for each local explanation that allows to cluster together functionally similar local explanations.
Revised: Embedding Local Explanations: The following step consists in learning an embedding for each local explanation that allows to cluster together functionally similar local explanations.

Original: This is achieved with a standard GNN h which maps any graph ¯ G into a fixed-sized embedding h ( ¯ G ) ∈ R d .
Revised: This is achieved with a standard GNN h which maps any graph ¯ G into a fixed-sized embedding h ( ¯ G ) ∈ R d .

Original: Since each local explanation ¯ G is a subgraph of an input graph G , in our experiments we used the original node features of the dataset.
Revised: Since each local explanation ¯ G is a subgraph of an input graph G , in our experiments we used the original node features of the dataset.

Original: Note, however, that those feature can be arbitrarily augmented to make the aggregation easier.
Revised: Note, however, that those feature can be arbitrarily augmented to make the aggregation easier.

Original: The outcome of this aggregation consists in a set E = { h ( ¯ G ) , ∀ ¯ G ∈
Revised: The outcome of this aggregation consists in a set E = { h ( ¯ G ) , ∀ ¯ G ∈

Original: D } of graph embeddings.
Revised: D } of graph embeddings.

Original: Concept Projection : Inspired by previous works on prototype learning (Li et al., 2017; Chen et al., 2019b), we project each graph embedding e ∈ E into a set P of m prototypes { p i ∈
Revised: Concept Projection : Inspired by previous works on prototype learning (Li et al., 2017; Chen et al., 2019b), we project each graph embedding e ∈ E into a set P of m prototypes { p i ∈

Original: R
Revised: R

Original: d | i = 1 , . . .
Revised: d | i = 1 , . . .

Original: , m } via the following distance function:
Revised: , m } via the following distance function:

Original: [Equation]
Revised: [Equation]

Original: Prototypes are initialized randomly from a uniform distribution and are learned along with the other parameters of the architecture.
Revised: Prototypes are initialized randomly from a uniform distribution and are learned along with the other parameters of the architecture.

Original: As training progresses, the prototypes will align as prototypical representations of every cluster of local explanations, which will represent the final groups of graphical concepts.
Revised: As training progresses, the prototypes will align as prototypical representations of every cluster of local explanations, which will represent the final groups of graphical concepts.

Original: The output of this projection is thus a set V
Revised: The output of this projection is thus a set V

Original: = { v e , ∀ e ∈ E } where v e =
Revised: = { v e , ∀ e ∈ E } where v e =

Original: [ d ( p 1 , e ) , .., d ( p m , e )] is a vector containing a probabilistic assignment of graph embedding e (thus the local explanation that maps to it) to the m concepts, and will be henceforth referred to as concept vector .
Revised: [ d ( p 1 , e ) , .., d ( p m , e )] is a vector containing a probabilistic assignment of graph embedding e (thus the local explanation that maps to it) to the m concepts, and will be henceforth referred to as concept vector .

Original: Formulas Learning: The final step consists of an E-LEN, i.e., a Logic Explainable Network (Ciravegna et al., 2021a) implemented with an Entropy Layer as first layer (Barbiero et al., 2021).
Revised: Formulas Learning: The final step consists of an E-LEN, i.e., a Logic Explainable Network (Ciravegna et al., 2021a) implemented with an Entropy Layer as first layer (Barbiero et al., 2021).

Original: An E-LEN learns to map a concept activation vector to a class while encouraging a sparse use of concepts that allows to reliably extract Boolean formulas emulating the network behaviour.
Revised: An E-LEN learns to map a concept activation vector to a class while encouraging a sparse use of concepts that allows to reliably extract Boolean formulas emulating the network behaviour.

Original: We train an E-LEN to emulate the behaviour of the GNN f feeding it with the graphical concepts extracted from the local explanations.
Revised: We train an E-LEN to emulate the behaviour of the GNN f feeding it with the graphical concepts extracted from the local explanations.

Original: Given a set of local explanations ¯ G 1 . . .
Revised: Given a set of local explanations ¯ G 1 . . .

Original: ¯ G n i for an input graph
Revised: ¯ G n i for an input graph

Original: G i and the corresponding set of the concept vectors v 1 . . .
Revised: G i and the corresponding set of the concept vectors v 1 . . .

Original: v n i , we aggregate the concept vectors via a pooling operator and feed the resulting aggregated concept vector to the E-LEN, providing f ( G i ) as supervision.
Revised: v n i , we aggregate the concept vectors via a pooling operator and feed the resulting aggregated concept vector to the E-LEN, providing f ( G i ) as supervision.

Original: In our experiments we used a max-pooling operator.
Revised: In our experiments we used a max-pooling operator.

Original: Thus, the Entropy Layer learns a mapping from the pooled concept vector to (i) the embeddings z (as any linear layer) which will be used by the successive MLP for matching the predictions of f .
Revised: Thus, the Entropy Layer learns a mapping from the pooled concept vector to (i) the embeddings z (as any linear layer) which will be used by the successive MLP for matching the predictions of f .

Original: (ii) a truth table T explaining how the network leveraged concepts to make predictions for the target class.
Revised: (ii) a truth table T explaining how the network leveraged concepts to make predictions for the target class.

Original: Since the input pooled concept vector will constitute the premise in the truth table T , a desirable property to improve human readability is discreteness, which we achieve using the Straight-Through (ST) trick used for discrete Gumbel-Softmax Estimator (Jang et al., 2016).
Revised: Since the input pooled concept vector will constitute the premise in the truth table T , a desirable property to improve human readability is discreteness, which we achieve using the Straight-Through (ST) trick used for discrete Gumbel-Softmax Estimator (Jang et al., 2016).

Original: In practice, we compute the forward pass discretizing each v i via argmax , then, in the backward pass to favor the flow of informative gradient we use its continuous version.
Revised: In practice, we compute the forward pass discretizing each v i via argmax , then, in the backward pass to favor the flow of informative gradient we use its continuous version.

Original: Supervision Loss: GLGExplainer is trained end-to-end with the following loss:
Revised: Supervision Loss: GLGExplainer is trained end-to-end with the following loss:

Original: [Equation]
Revised: [Equation]

Original: where L surr corresponds to a Focal BCELoss (Lin et al., 2017) between the prediction of our E-LEN and the predictions to explain, while L R 1 and L R 2 are respectively aimed to push every prototype p j to be close to at least one local explanation and to push each local explanation to be close to at least one prototype (Li et al., 2017).
Revised: where L surr corresponds to a Focal BCELoss (Lin et al., 2017) between the prediction of our E-LEN and the predictions to explain, while L R 1 and L R 2 are respectively aimed to push every prototype p j to be close to at least one local explanation and to push each local explanation to be close to at least one prototype (Li et al., 2017).

Original: The losses are defined as follows:
Revised: The losses are defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where p and γ represent respectively the probability for positive class prediction and the focusing parameter which controls how much to penalize hard examples.
Revised: where p and γ represent respectively the probability for positive class prediction and the focusing parameter which controls how much to penalize hard examples.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted an experimental evaluation on synthetic and real-world datasets aimed at answering the following research questions:
Revised: We conducted an experimental evaluation on synthetic and real-world datasets aimed at answering the following research questions:

Original: • Q1:
Revised: • Q1:

Original: Can GLGExplainer extract meaningful Global Explanations?
Revised: Can GLGExplainer extract meaningful Global Explanations?

Original: • Q2:
Revised: • Q2:

Original: Can GLGExplainer extract faithful Global Explanations?
Revised: Can GLGExplainer extract faithful Global Explanations?

Original: The source code of GLGExplainer, including the extraction of local explanations, as well as the datasets and all the code for reproducing the results are freely available online 1 .
Revised: The source code of GLGExplainer, including the extraction of local explanations, as well as the datasets and all the code for reproducing the results will be made freely available online.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We tested our proposed approach on three datasets, namely:
Revised: We tested our proposed approach on three datasets, namely:

Original: BAMultiShapes: BAMultiShapes is a newly introduced extension of some popular synthetic benchmarks (Ying et al., 2019) aimed to assess the ability of a Global Explainer to deal with logical combinations of concepts.
Revised: BAMultiShapes: BAMultiShapes is a newly introduced extension of some popular synthetic benchmarks (Ying et al., 2019) aimed to assess the ability of a Global Explainer to deal with logical combinations of concepts.

Original: In particular, we created a dataset composed of 1,000 Barab ´ asi-Albert (BA) graphs with attached in random positions the following network motifs: house, grid, wheel.
Revised: In particular, we created a dataset composed of 1,000 Barab´asi-Albert (BA) graphs with attached in random positions the following network motifs: house, grid, wheel.

Original: Classcontains plain BA graphs and BA graphs enriched with a house, a grid, a wheel, or the three motifs together.
Revised: Classcontains plain BA graphs and BA graphs enriched with a house, a grid, a wheel, or the three motifs together.

Original: Class 1 contains BA graphs enriched with a house and a grid, a house and a wheel or a wheel and a grid.
Revised: Class 1 contains BA graphs enriched with a house and a grid, a house and a wheel or a wheel and a grid.

Original: Mutagenicity: The Mutagenicity dataset is a collection of 4,337 molecule graphs where each graph is labelled as either having a mutagenic effect or not.
Revised: Mutagenicity: The Mutagenicity dataset is a collection of 4,337 molecule graphs where each graph is labelled as either having a mutagenic effect or not.

Original: Based on Debnath et al.
Revised: Based on Debnath et al.

Original: (1991), the mutagenicity of a molecule is correlated with the presence of electron-attracting elements conjugated with nitro groups (e.g. NO2 ).
Revised: (1991), the mutagenicity of a molecule is correlated with the presence of electron-attracting elements conjugated with nitro groups (e.g. NO2 ).

Original: Moreover, compounds with three or more fused rings tend to be more mutagenic than those with one or two.
Revised: Moreover, compounds with three or more fused rings tend to be more mutagenic than those with one or two.

Original: Most previous works on this dataset have focused on finding the functional group NO2 , since most of them struggle in finding compounds with more than two carbon rings.
Revised: Most previous works on this dataset have focused on finding the functional group NO2 , since most of them struggle in finding compounds with more than two carbon rings.

Original: Hospital Interaction Network (HIN) :
Revised: Hospital Interaction Network (HIN) :

Original: HIN is a new real-world benchmark proposed in this work.
Revised: HIN is a new real-world benchmark proposed in this work.

Original: It represents the third order ego graphs for doctors and nurses in a face-to-face interaction network collected in an hospital (Vanhems et al., 2013).
Revised: It represents the third order ego graphs for doctors and nurses in a face-to-face interaction network collected in an hospital (Vanhems et al., 2013).

Original: There are four type of individuals in the network: doctors (D), nurses (N), patients (P) and administrators (A).
Revised: There are four type of individuals in the network: doctors (D), nurses (N), patients (P) and administrators (A).

Original: Such typologies constitute the feature vector for each node, represented as one-hot encoding.
Revised: Such typologies constitute the feature vector for each node, represented as one-hot encoding.

Original: Each ego network is an instance, and the task is to classify whether the ego in the ego network is a doctor or a nurse (without knowing its node features, which are masked).
Revised: Each ego network is an instance, and the task is to classify whether the ego in the ego network is a doctor or a nurse (without knowing its node features, which are masked).

Original: More details about the dataset construction are available in the
Revised: More details about the dataset construction are available in the

Original: Appendix.
Revised: Appendix.

Original: For Mutagenicity we replicated the setting in the PGExplainer paper (Luo et al., 2020), while for
Revised: For Mutagenicity we replicated the setting in the PGExplainer paper (Luo et al., 2020), while for

Original: BAMultiShapes and HIN we trained until convergence a 3-layers GCN.
Revised: BAMultiShapes and HIN we trained until convergence a 3-layers GCN.

Original: Details about the training of the networks and their accuracies are in the Appendix.
Revised: Details about the training of the networks and their accuracies are in the Appendix.

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: Local Explanations Extraction: As discussed in Section 3, we used PGExplainer (Luo et al., 2020) as the Local Explainer.
Revised: Local Explanations Extraction: As discussed in Section 3, we used PGExplainer (Luo et al., 2020) as the Local Explainer.

Original: However, we modified the procedure for discretizing weighted graphs into a set of disconnected motifs.
Revised: However, we modified the procedure for discretizing weighted graphs into a set of disconnected motifs.

Original: Indeed, the authors in Luo et al.
Revised: Indeed, the authors in Luo et al.

Original: (2020) limited their analysis to graphs that contained the ground truth motifs, and proposed to keep the top-k edges as a rule-of-thumb for visualization purposes.
Revised: (2020) limited their analysis to graphs that contained the ground truth motifs, and proposed to keep the top-k edges as a rule-of-thumb for visualization purposes.

Original: For Mutagenicity, over which PGExplainer was originally evaluated, we simply selected the threshold θ that maximises the F1 score of the local explainer over all graphs, including those that do not contain the ground-truth motif.
Revised: For Mutagenicity, over which PGExplainer was originally evaluated, we simply selected the threshold θ that maximises the F1 score of the local explainer over all graphs, including those that do not contain the ground-truth motif.

Original: For the novel datasets BAMultiShapes and HIN, we adopted a dynamic algorithm to select θ that does not require any prior knowledge about the ground truth motifs.
Revised: For the novel datasets BAMultiShapes and HIN, we adopted a dynamic algorithm to select θ that does not require any prior knowledge about the ground truth motifs.

Original: This algorithm resembles the elbow-method, i.e., for each local explanation sort weights in decreasing order and choose as threshold the first weight that differs by at least 40% from the previous one.
Revised: This algorithm resembles the elbow-method, i.e., for each local explanation sort weights in decreasing order and choose as threshold the first weight that differs by at least 40% from the previous one.

Original: We believe that threshold selection for Local Explainers is a fundamental problem to make local explainers actionable, but it is often left behind in favor of top- k selections where k is chosen based on the ground-truth motif.
Revised: We believe that threshold selection for Local Explainers is a fundamental problem to make local explainers actionable, but it is often left behind in favor of top- k selections where k is chosen based on the ground-truth motif.

Original: In the Appendix we show some examples for each dataset along with their extracted explanations.
Revised: In the Appendix we show some examples for each dataset along with their extracted explanations.

Original: GLGExplainer:
Revised: GLGExplainer:

Original: We implemented the Local Explanation Embedder h as a 2-layers GIN (Xu et al., 2018) network for each dataset except HIN, for which we found a GATV2 (Brody et al., 2021) to provide better performance as the attention mechanism allows the network to account for the relative importance of the type of neighbouring individuals.
Revised: We implemented the Local Explanation Embedder h as a 2-layers GIN (Xu et al., 2018) network for each dataset except HIN, for which we found a GATV2 (Brody et al., 2021) to provide better performance as the attention mechanism allows the network to account for the relative importance of the type of neighbouring individuals.

Original: All layers consist in 20 hidden units followed by a non-linear combination of max, mean, and sum graph pooling.
Revised: All layers consist in 20 hidden units followed by a non-linear combination of max, mean, and sum graph pooling.

Original: We set the number of prototypes m to 6, 2, and 4 for BAMultiShapes, Mutagenicity, and HIN respectively (see Section 4.4 for an analysis showing how these numbers were inferred), keeping the dimensionality d to 10.
Revised: We set the number of prototypes m to 6, 2, and 4 for BAMultiShapes, Mutagenicity, and HIN respectively (see Section 4.4 for an analysis showing how these numbers were inferred), keeping the dimensionality d to 10.

Original: We trained using ADAM optimizer with early stopping and with a learning rate of 1 e − 3 for the embedding and prototype learning components and a learning rate of 5 e − 4 for the E-LEN.
Revised: We trained using ADAM optimizer with early stopping and with a learning rate of 1 e − 3 for the embedding and prototype learning components and a learning rate of 5 e − 4 for the E-LEN.

Original: The batch size was set to 128, the focusing parameter γ to 2, while the auxiliary loss coefficients λ 1 and λ 2 were set respectively to 0.09 and 0.00099.
Revised: The batch size was set to 128, the focusing parameter γ to 2, while the auxiliary loss coefficients λ 1 and λ 2 were set respectively to 0.09 and 0.00099.

Original: The E-LEN consists of an input Entropy Layer ( R m → R 10 ), a hidden layer ( R 10 → R 5 ), and an output layer with LeakyReLU activation function.
Revised: The E-LEN consists of an input Entropy Layer ( R m → R 10 ), a hidden layer ( R 10 → R 5 ), and an output layer with LeakyReLU activation function.

Original: We turned off the attention mechanism encouraging a sparse use of concepts from the E-LEN, as the end-toend architecture of GLGExplainer already promotes the emergence of predictive concepts, and the discretization step preceding the E-LEN encourages each concept activation vector to collapse on a single concept.
Revised: We turned off the attention mechanism encouraging a sparse use of concepts from the E-LEN, as the end-toend architecture of GLGExplainer already promotes the emergence of predictive concepts, and the discretization step preceding the E-LEN encourages each concept activation vector to collapse on a single concept.

Original: All these hyper-parameters were identified via internal cross-validation over the training set.
Revised: All these hyper-parameters were identified via internal cross-validation over the training set.

Original: 4. E VALUATION M ETRICS
Revised: 4. E VALUATION M ETRICS

Original: In order to show the robustness of our proposed methodology, we have evaluated GLGExplainer on the following three metrics, namely: i) FIDELITY, which represents the accuracy of the formulas in 
Revised: In order to show the robustness of our proposed methodology, we have evaluated GLGExplainer on the following three metrics, namely: i) FIDELITY, which represents the accuracy of the formulas in matching the predictions of the model f to explain; ii) ACCURACY, which represents the accuracy of the formulas in matching the ground-truth labels of the graphs; iii) CONCEPT PURITY, which is computed for every cluster independently and measures how good the embedding is at clustering

Original: [Table]
Revised: [Table]

Original: matching the predictions of the model f to explain; ii) ACCURACY, which represents the accuracy of the formulas in matching the ground-truth labels of the graphs; iii) CONCEPT PURITY, which is computed for every cluster independently and measures how good the embedding is at clustering the local explanations.
Revised: the local explanations. A more detailed description of those metrics is available in the Appendix.

Original: For BAMultiShapes and Mutagenicity, every local explanation was annotated with its corresponding ground-truth motif, or Others in case it did not match any ground-truth motif.
Revised: For BAMultiShapes and Mutagenicity, every local explanation was annotated with its corresponding ground-truth motif, or Others in case it did not match any ground-truth motif.

Original: For HIN, since no ground truth explanation is available, we labeled each local explanation with a string summarizing the users involved in the interaction (e.g., in a local explanation representing the interaction between a nurse and a patient, the label corresponds to NP ).
Revised: For HIN, since no ground truth explanation is available, we labeled each local explanation with a string summarizing the users involved in the interaction (e.g., in a local explanation representing the interaction between a nurse and a patient, the label corresponds to NP ).

Original: Since labelling each possible combination of interactions between 4 types of users would make the interpretation of the embedding difficult, we annotated the most frequent seven ( D, N, NA, P, NP, MN, MP ), assigning the rest to Others .
Revised: Since labelling each possible combination of interactions between 4 types of users would make the interpretation of the embedding difficult, we annotated the most frequent seven ( D, N, NA, P, NP, MN, MP ), assigning the rest to Others .

Original: Note that such unsupervised annotation may negatively impact the resulting Concept Purity, since it ignores the correlation between similar but not identical local explanations.
Revised: Note that such unsupervised annotation may negatively impact the resulting Concept Purity, since it ignores the correlation between similar but not identical local explanations.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: In this section we will go through the experimental results with the aim of answering the research questions defined above.
Revised: In this section we will go through the experimental results with the aim of answering the research questions defined above.

Original: Table 1 presents the raw formulas extracted by the Entropy Layer where, to improve readability, we have dropped negated literals from clauses.
Revised: Table 1 presents the raw formulas extracted by the Entropy Layer where, to improve readability, we have dropped negated literals from clauses.

Original: This means that for each clause in a formula, missing concepts are implicitly negated.
Revised: This means that for each clause in a formula, missing concepts are implicitly negated.

Original: Those formulas can be further rewritten in a more human-understandable format after inspecting the representative elements of each cluster, as shown in Figure 2.
Revised: Those formulas can be further rewritten in a more human-understandable format after inspecting the representative elements of each cluster, as shown in Figure 2.

Original: The resulting Global Explanations in human-understandable format are reported in Figure 3, where we included a qualitative comparison with the global explanations generated by
Revised: The resulting Global Explanations in human-understandable format are reported in Figure 3, where we included a qualitative comparison with the global explanations generated by

Original: XGNN (Yuan et al., 2020), the only available competitor for global explanations of GNN.
Revised: XGNN (Yuan et al., 2020), the only available competitor for global explanations of GNN.

Original: Finally,
Revised: Finally,

Original: Table 2 shows the results over the three metrics as described in Section 4.3.
Revised: Table 2 shows the results over the three metrics as described in Section 4.3.

Original: Note that XGNN is not shown in the table as it cannot be evaluated according to these metrics.
Revised: Note that XGNN is not shown in the table as it cannot be evaluated according to these metrics.

Original: Q1: Can GLGExplainer extract meaningful Global Explanations?
Revised: Q1: Can GLGExplainer extract meaningful Global Explanations?

Original: The building blocks of the Global Explanations extracted by GLGExplainer are the graphical concepts that are learned in the concept projection layer.
Revised: The building blocks of the Global Explanations extracted by GLGExplainer are the graphical concepts that are learned in the concept projection layer.

Original: Figure 2 clearly shows that each concept represents local explanations with specific characteristics, thus achieving the desired goal of creating 
Revised: Figure 2 clearly shows that each concept represents local explanations with specific characteristics, thus achieving the desired goal of creating interpretable concepts.

Original: [Table]

Original: interpretable concepts. Note that clusters corresponding to concepts are on average quite homogeneous (see Concept Purity in Table 2)
Revised: Note that clusters corresponding to concepts are on average quite homogeneous (see Concept Purity in Table 2), and the concept representatives in the figure are faithful

Original: [Table]
Revised: [Table]

Original: and the concept representatives in the figure are faithful representations of the instances in their corresponding cluster.
Revised: representations of the instances in their corresponding cluster.

Original: See the Appendix for further details
Revised: See the Appendix for further details

Original: [Table]
Revised: [Table]

Original: where we report five random instances for each concept.
Revised: where we report five random instances for each concept.

Original: It is worth noting that this clustering emerges solely based on the supervision defined by Eq 3, while no specific supervision was added to cluster local explanations based on their similarity.
Revised: It is worth noting that this clustering emerges solely based on the supervision defined by Eq [Table] while no specific supervision was added to cluster local explanations based on their similarity.

Original: This is the reason behind the emergence of the Mix cluster around p 2 in the upper part of Figure 2, which represents all local explanations with at least two motifs that are present solely in Class 1 of BAMultiShapes.
Revised: This is the reason behind the emergence of the Mix cluster around p 2 in the upper part of Figure [Table] which represents all local explanations with at least two motifs that are present solely in Class 1 of BAMultiShapes.

Original: Additionally, as shown in Figure 3, GLGExplainer manages to combine these building blocks into highly interpretable explanations.
Revised: Additionally, as shown in Figure 3, GLGExplainer manages to combine these building blocks into highly interpretable explanations.

Original: The explanation for BAMultiShapes almost perfectly matches the ground-truth formula, where the only difference is the conjunction of all motifs being assigned to
Revised: The explanation for BAMultiShapes almost perfectly matches the ground-truth formula, where the only difference is the conjunction of all motifs being assigned to

Original: Class 1 rather than Class 0 .
Revised: Class 1 rather than Class 0 .

Original: This however is due to a discrepancy between the ground-truth formula and what the GNN learned, as will be discussed in the answer to Q2 .
Revised: This however is due to a discrepancy between the ground-truth formula and what the GNN learned, as will be discussed in the answer to Q2 .

Original: Note that the Mix cluster has been rewritten as the conjunction of the shapes it contains when extracting the human-interpretable formulas.
Revised: Note that the Mix cluster has been rewritten as the conjunction of the shapes it contains when extracting the human-interpretable formulas.

Original: For Mutagenesis, the GLGExplainer manages to recover the well-known NO2 motif as an indicator of mutagenicity ( Class 0 ).
Revised: For Mutagenesis, the GLGExplainer manages to recover the well-known NO2 motif as an indicator of mutagenicity ( Class 0 ).

Original: It is worth reminding that in all formulas in Figure 3, negative literals have been dropped from clauses for improved readability.
Revised: It is worth reminding that in all formulas in Figure 3, negative literals have been dropped from clauses for improved readability.

Original: Having formulas for Mutagenesis only two concepts, this implies that the formula for Class 0 actually represents NO2 itself ((NO ∧ OTHERS) ∨ (NO2 ∧ ¬ OTHERS)
Revised: Having formulas for Mutagenesis only two concepts, this implies that the formula for Class 0 actually represents NO2 itself ((NO ∧ OTHERS) ∨ (NO2 ∧ ¬ OTHERS)

Original: ⇐⇒ NO2).
Revised: ⇐⇒ NO2).

Original: For HIN, the global explanations match the common belief that nurses tend to interact more with other nurses or with patients, while doctors tend to interact more frequently with other doctors (or patients, but less frequently than nurses).
Revised: For HIN, the global explanations match the common belief that nurses tend to interact more with other nurses or with patients, while doctors tend to interact more frequently with other doctors (or patients, but less frequently than nurses).

Original: Conversely to our results, XGNN (Yuan et al., 2020) fails to generate explanations matching either the ground truth or common belief about the dataset in most cases, while failing to generate any graph in others.
Revised: Conversely to our results, XGNN (Yuan et al., 2020) fails to generate explanations matching either the ground truth or common belief about the dataset in most cases, while failing to generate any graph in others.

Original: Q2: Can GLGExplainer extract faithful Global Explanations?
Revised: Q2: Can GLGExplainer extract faithful Global Explanations?

Original: The high Fidelity reported in Table 2 shows that the extracted formulas are correctly matching the behaviour of the model in most samples, while being defined over fairly pure concept as shown by the Concept Purity in the same Table.
Revised: The high Fidelity reported in Table 2 shows that the extracted formulas are correctly matching the behaviour of the model in most samples, while being defined over fairly pure concept as shown by the Concept Purity in the same Table.

Original: It is worth highlighting that GLGExplainer is not simply generating an explanation for the ground-truth labeling of the dataset, bypassing the GNN it is supposed to explain, but it is indeed capturing its underlying predictive behaviour.
Revised: It is worth highlighting that GLGExplainer is not simply generating an explanation for the ground-truth labeling of the dataset, bypassing the GNN it is supposed to explain, but it is indeed capturing its underlying predictive behaviour.

Original: This can be seen by observing that over the training set, Fidelity is higher than Accuracy for all datasets.
Revised: This can be seen by observing that over the training set, Fidelity is higher than Accuracy for all datasets.

Original: Note that on BAMultiShapes, training accuracy is lower than test accuracy.
Revised: Note that on BAMultiShapes, training accuracy is lower than test accuracy.

Original: The reason for this train-test discrepancy can be found in the fact that the GNN fails to identify the logical composition of all three motifs (which are rare and never occur in the test set) as an indicator of Class 0 .
Revised: The reason for this train-test discrepancy can be found in the fact that the GNN fails to identify the logical composition of all three motifs (which are rare and never occur in the test set) as an indicator of Class 0 .

Original: This can be seen by decomposing the GNN accuracy with respect to the motifs that occur in the data (Table 3), and observing that the accuracy for the group with all three motifs (All) is exactly zero.
Revised: This can be seen by decomposing the GNN accuracy with respect to the motifs that occur in the data (Table 3), and observing that the accuracy for the group with all three motifs (All) is exactly zero.

Original: Quite remarkably, GLGExplainer manages to capture this anomaly in the GNN, as the clause involving all three motifs is learned as part of the formula for Class 1 instead of Class 0 , as shown in Figure 3.
Revised: Quite remarkably, GLGExplainer manages to capture this anomaly in the GNN, as the clause involving all three motifs is learned as part of the formula for Class 1 instead of Class 0 , as shown in Figure 3.

Original: The ability of GLGExplainer to explain this anomalous behaviour is a promising indication of its potential as a diagnostic tool for learned GNNs. 
Revised: The ability of GLGExplainer to explain this anomalous behaviour is a promising indication of its potential as a diagnostic tool for learned GNNs. to be added to the Barab ´ asi-Albert base graph.

Revised: H, G, W respectively for House, Grid, and Wheel.

Original: [Table]
Revised: [Table]

Original: In the rest of this section we show how the number of prototypes, that critically affects the interpretability of the explanation, can be easily inferred by trading-off Fidelity, Concept Purity and sparsity, and we provide an ablation study to demonstrate the importance of the Discretization trick.
Revised: In the rest of this section we show how the number of prototypes, that critically affects the interpretability of the explanation, can be easily inferred by trading-off Fidelity, Concept Purity and sparsity, and we provide an ablation study to demonstrate the importance of the Discretization trick.

Original: Choice of the number of prototypes : The number of prototypes in the experiments of this section was determined by selecting the smallest m which achieves satisfactory results in terms of Fidelity and Concept Purity, as measured on a validation set.
Revised: Choice of the number of prototypes : The number of prototypes in the experiments of this section was determined by selecting the smallest m which achieves satisfactory results in terms of Fidelity and Concept Purity, as measured on a validation set.

Original: Specifically, we aim for a parsimonious value of m to comply to the human cognitive bias of favoring simpler explanations to more complex ones (Miller, 1956).
Revised: Specifically, we aim for a parsimonious value of m to comply to the human cognitive bias of favoring simpler explanations to more complex ones (Miller, 1956).

Original: Table 4 reports how different values of m impact Fidelity and Concept Purity.
Revised: Table 4 reports how different values of m impact Fidelity and Concept Purity.

Original: The number of prototypes achieving the best trade-off between the different objectives was identified as 6, 2 and 4 for BAMultiShapes, Mutagenicity, and HIN respectively.
Revised: The number of prototypes achieving the best trade-off between the different objectives was identified as 6, 2 and 4 for BAMultiShapes, Mutagenicity, and HIN respectively.

Original: Role of the Discretization trick : The Discretization trick was introduced in Section 3 to enforce a discrete prototype assignment, something essential for an unambiguous definition of the concepts on which the formulas are based.
Revised: Role of the Discretization trick : The Discretization trick was introduced in Section 3 to enforce a discrete prototype assignment, something essential for an unambiguous definition of the concepts on which the formulas are based.

Original: We ran an ablation study to evaluate its contribution to the overall performance of GLGExplainer.
Revised: We ran an ablation study to evaluate its contribution to the overall performance of GLGExplainer.

Original: Figure 4 (left) shows the reduction in concept vector entropy achieved by GLGExplainer with the discretization trick (red curve, zero entropy by construction) as compared to GLGExplainer with the trick disabled (orange curve).
Revised: Figure 4 (left) shows the reduction in concept vector entropy achieved by GLGExplainer with the discretization trick (red curve, zero entropy by construction) as compared to GLGExplainer with the trick disabled (orange curve).

Original: Figure 4 (middle) reports the Fidelity over the training epochs for the two variants.
Revised: Figure 4 (middle) reports the Fidelity over the training epochs for the two variants.

Original: The figure shows the effectiveness of the discretization trick in boosting Fidelity of the extracted formulas, which is more than double the one achieved disabling it.
Revised: The figure shows the effectiveness of the discretization trick in boosting Fidelity of the extracted formulas, which is more than double the one achieved disabling it.

Original: We conjecture that the reason for this behaviour is the fact that the discretization trick forces the hidden layers of the E-LEN to focus on the information relative to the closest prototype, ignoring[Table] other positional information of local explanations.
Revised: We conjecture that the reason for this behaviour is the fact that the discretization trick forces the hidden layers of the E-LEN to focus on the information relative to the closest prototype, ignoring other positional information of local explanations.

Original: Thus, the E-LEN predictions are much better aligned with the discrete formulas being extracted, and indeed the Fidelity of the formulas matches the Fidelity of the E-LEN alone, which is shown in the right plot.
Revised: Thus, the E-LEN predictions are much better aligned with the discrete formulas being extracted, and indeed the Fidelity of the formulas matches the Fidelity of the E-LEN alone, which is shown in the right plot.

Original: On the other hand, GLGExplainer  without discretization has a high Fidelity of the E-LEN, but fails to extract highly faithful formulas.
Revised: On the other hand, GLGExplainer [Table] without discretization has a high Fidelity of the E-LEN, but fails to extract highly faithful formulas.

Original: Note that simply adding an entropy loss over the concept vector to the overall loss (Eq. 3) fails to achieve the same performance obtained with the discretization trick.
Revised: Note that simply adding an entropy loss over the concept vector to the overall loss (Eq. 3) fails to achieve the same performance obtained with the discretization trick.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced GLGExplainer, the first Global Explainer for GNNs capable of generating explanations as logic formulas, represented in terms of learned human-interpretable graphical concepts.
Revised: We introduced GLGExplainer, the first Global Explainer for GNNs capable of generating explanations as logic formulas, represented in terms of learned human-interpretable graphical concepts.

Original: The approach is inherently faithful to the data domain, since it processes local explanations as extracted by an off-the-shelf Local Explainer.
Revised: The approach is inherently faithful to the data domain, since it processes local explanations as extracted by an off-the-shelf Local Explainer.

Original: Our experiments showed that GLGExplainer, contrary to existing solutions, can faithfully describe the predictive behaviour of the model, being able to aggregate local explanations into meaningful high-level concepts and combine them into formulas achieving high Fidelity.
Revised: Our experiments showed that GLGExplainer, contrary to existing solutions, can faithfully describe the predictive behaviour of the model, being able to aggregate local explanations into meaningful high-level concepts and combine them into formulas achieving high Fidelity.

Original: GLGExplainer even managed to provide insights into some occasionally incorrect rules learned by the model.
Revised: GLGExplainer even managed to provide insights into some occasionally incorrect rules learned by the model.

Original: We believe that this approach can constitute the basis for investigating how GNNs build their predictions and debug them, which could substantially increase human trust in this technology.
Revised: We believe that this approach can constitute the basis for investigating how GNNs build their predictions and debug them, which could substantially increase human trust in this technology.

Original: The proposed GLGExplainer is inherently faithful to the data domain since it processes local explanations provided by a Local Explainer.
Revised: The proposed GLGExplainer is inherently faithful to the data domain since it processes local explanations provided by a Local Explainer.

Original: However, the quality of those local explanations, in terms of representativeness and discriminability with respect to the task-specific class, has a direct effect on the Fidelity.
Revised: However, the quality of those local explanations, in terms of representativeness and discriminability with respect to the task-specific class, has a direct effect on the Fidelity.

Original: If the generated concept vector does not exhibit any class-specific pattern, then the
Revised: If the generated concept vector does not exhibit any class-specific pattern, then the

Original: E-LEN will not be able to emulate the predictions of the model to explain.
Revised: E-LEN will not be able to emulate the predictions of the model to explain.

Original: Despite being a potential limitation of GLGExplainer, this can actually open to the possibility of using the Formula Fidelity as a proxy of local explanations quality, which is notoriously difficult to assess.
Revised: Despite being a potential limitation of GLGExplainer, this can actually open to the possibility of using the Formula Fidelity as a proxy of local explanations quality, which is notoriously difficult to assess.

Original: We leave this investigation to future work.
Revised: We leave this investigation to future work.

Original: Despite tailoring our discussion on graph classification, our approach can be readily extended to any kind of classification task on graphs, provided that a suitable Local
Revised: Despite tailoring our discussion on graph classification, our approach can be readily extended to any kind of classification task on graphs, provided that a suitable Local

Original: Explainer is available.
Revised: Explainer is available.

Original: A A PPENDIX A. D ATASETS DETAILS
Revised: A A PPENDIX A. D ATASETS DETAILS

Original: While BAMultiShapes and Mutagenicity were already described in detail in Section 4.1, here we report more details about the newly introduced dataset for graph Explainability.
Revised: While BAMultiShapes and Mutagenicity were already described in detail in Section 4.1, here we report more details about the newly introduced dataset for graph Explainability.

Original: Figure 7-9 presents some random examples for each dataset, with their extracted explanation in bold.
Revised: Figure 7-9 presents some random examples for each dataset, with their extracted explanation in bold.

Original: HIN: Hospital Interaction Networks is a new real benchmark that we propose in this work.
Revised: HIN: Hospital Interaction Networks is a new real benchmark that we propose in this work.

Original: The dataset was collected using a wearable sensors, equipped with radio-frequency identification devices (RFIDs) capturing face-to-face interactions.
Revised: The dataset was collected using a wearable sensors, equipped with radio-frequency identification devices (RFIDs) capturing face-to-face interactions.

Original: The devices record an interaction if and only if there is at least one exchanged signal within 20 seconds.
Revised: The devices record an interaction if and only if there is at least one exchanged signal within 20 seconds.

Original: The dataset was collected by Sociopatterns collaboration 2 in the geriatric ward of a university hospital (Vanhems et al., 2013) in Lyon, France, over four days in December 2010.
Revised: The dataset was collected by Sociopatterns collaboration 1 in the geriatric ward of a university hospital (Vanhems et al., 2013) in Lyon, France, over four days in December 2010.

Original: The individuals belong to four categories: medical doctors (M), nurses (N), administrative staff (A) and patients (P).
Revised: The individuals belong to four categories: medical doctors (M), nurses (N), administrative staff (A) and patients (P).

Original: Since the interaction network GT evolves over time, we convert the temporal network into a sequence of graph snapshots, aggregating interactions every five minutes ( GT =
Revised: Since the interaction network GT evolves over time, we convert the temporal network into a sequence of graph snapshots, aggregating interactions every five minutes ( GT =

Original: [ G 1 , G 2 , . . . , G n ] ).
Revised: [ G 1 , G 2 , . . . , G n ] ).

Original: For each static graph G i , we extract the ego graph with radius 3 centered in each doctor and each nurse, obtaining a set of ego graphs.
Revised: For each static graph G i , we extract the ego graph with radius 3 centered in each doctor and each nurse, obtaining a set of ego graphs.

Original: A GNN is trained in classifying between ego networks of doctor and nurse, where the features of each ego node is masked.
Revised: A GNN is trained in classifying between ego networks of doctor and nurse, where the features of each ego node is masked.

Original: An illustrative example of this procedure is shown in Figure 5.
Revised: An illustrative example of this procedure is shown in Figure 5.

Original: In particular, the top of the figure shows a static snapshot ( G i ) of the temporal graph.
Revised: In particular, the top of the figure shows a static snapshot ( G i ) of the temporal graph.

Original: In the middle, we show three ego graphs (with radius equal to 2 for convenience) centered in two doctors and a nurse respectively.
Revised: In the middle, we show three ego graphs (with radius equal to 2 for convenience) centered in two doctors and a nurse respectively.

Original: Finally, the ego node features are masked, as depicted at the bottom of the Figure 5.
Revised: Finally, the ego node features are masked, as depicted at the bottom of the Figure 5.

Original: At the end of this extraction procedure, we obtain 880 and 2009 ego graphs for doctors and nurses respectively.
Revised: At the end of this extraction procedure, we obtain 880 and 2009 ego graphs for doctors and nurses respectively.

Original: To avoid class imbalance, we sub-sample the nurse class, obtaining a balanced dataset of 1760 graphs.
Revised: To avoid class imbalance, we sub-sample the nurse class, obtaining a balanced dataset of 1760 graphs.

Original: A. I MPLEMENTATION D ETAILS A.2. T RAINING THE GNN f
Revised: A. I MPLEMENTATION D ETAILS A.2. T RAINING THE GNN f

Original: In this section we will provide more details about the training of the GNN to explain f .
Revised: In this section we will provide more details about the training of the GNN to explain f .

Original: While for
Revised: While for

Original: Mutagenicity we limited to reproduce the results presented in (Luo et al., 2020) both in terms of model to explain (a 3 layers GCN Kipf & Welling (2016)) and local explanations, for BAMultiShapes and HIN we trained our own networks.
Revised: Mutagenicity we limited to reproduce the results presented in (Luo et al., 2020) both in terms of model to explain (a 3 layers GCN Kipf & Welling (2016)) and local explanations, for BAMultiShapes and HIN we trained our own networks.

Original: For BAMultiShapes we adopted a 3-layers GCN (20-20-20 hidden units) with mean graph pooling for the final prediction, whilst for HIN we employed a 3-layers GCN (20-20-20 hidden units) with non-linear combination of sum, mean, and max graph pooling.
Revised: For BAMultiShapes we adopted a 3-layers GCN (20-20-20 hidden units) with mean graph pooling for the final prediction, whilst for HIN we employed a 3-layers GCN (20-20-20 hidden units) with non-linear combination of sum, mean, and max graph pooling.

Original: The final model performances are reported in Table 5.
Revised: The final model performances are reported in Table 5.

Original: In both cases we used ADAM optimizer, training until convergence and using the validation set to select the best epoch.
Revised: In both cases we used ADAM optimizer, training until convergence and using the validation set to select the best epoch.

Original: A.2. E XPLAINERS
Revised: A.2. E XPLAINERS

Original: In this work we relied mainly on two off-the-shelf explainers, namely, PGExplainer (Luo et al., 2020) and XGNN (Yuan et al., 2020).
Revised: In this work we relied mainly on two off-the-shelf explainers, namely, PGExplainer (Luo et al., 2020) and XGNN (Yuan et al., 2020).

Original: Here we report some details about their usage.
Revised: Here we report some details about their usage.

Original: PGExplainer: For Mutagenicity and BAMultiShapes, we used the original implementation as provided by Luo et al.
Revised: PGExplainer: For Mutagenicity and BAMultiShapes, we used the original implementation as provided by Luo et al.

Original: For BAMultiShapes we changed to original hyper-parameters to { epochs:5, args.elr=0.007, args.coff t0=1.0, args.coff size=0.
Revised: For BAMultiShapes we changed to original hyper-parameters to { epochs:5, args.elr=0.007, args.coff t0=1.0, args.coff size=0.

Original: args.coff ent=0.000 } .
Revised: args.coff ent=0.000 } .

Original: For HIN we instead used the PyTorch implementation provided by
Revised: For HIN we instead used the PyTorch implementation provided by

Original: Agarwal et al. (2022), using as custom hyper-parameters { t0=1, t1=1, max epochs=30 } .
Revised: Agarwal et al. (2022), using as custom hyper-parameters { t0=1, t1=1, max epochs=30 } .

Original: Finally, for BAMultiShapes and HIN, for which we extracted our own local explanations, we trained PGExplainer on the train split of the original dataset.
Revised: Finally, for BAMultiShapes and HIN, for which we extracted our own local explanations, we trained PGExplainer on the train split of the original dataset.

Original: It is worth mentioning that for HIN, after the local explanations extraction, we removed every local explanation not containing the ego node.
Revised: It is worth mentioning that for HIN, after the local explanations extraction, we removed every local explanation not containing the ego node.

Original: XGNN: The official code provided here 3 is specifically tailored for generating explanations for the
Revised: XGNN: The official code provided here 2 is specifically tailored for generating explanations for the

Original: Mutagenicity dataset.
Revised: Mutagenicity dataset.

Original: For the other two, despite the algorithm accepting an heavy optimization for the task at hand (like defining custom rewards functions for each task), we made minimal changes to the architecture and to the original hyper-parameters in order not to input any a-priori knowledge.
Revised: For the other two, despite the algorithm accepting an heavy optimization for the task at hand (like defining custom rewards functions for each task), we made minimal changes to the architecture and to the original hyper-parameters in order not to input any a-priori knowledge.

Original: For HIN, specifically, we adapted the node type generation to match the node types of the dataset, and the custom check validity function to define whether the generated graph is valid, i.e., it must contain a single Ego node. 
Revised: For HIN, specifically, we adapted the node type generation to match the node types of the dataset, and the custom check validity function to define whether the generated graph is valid, i.e., it must contain a single ego node. For what concern the evaluation metrics presented in Section 4.3, since XGNN and GLGExplainer return explanations in two substantially different formats, we could not compare quantitatively the explanations provided by XGNN with ours. Thus, a direct comparison of [Table]

Original: A.2. C HOOSING λ 1 AND λ
Revised: A.2. C HOOSING λ 1 AND λ

Original: In Eq 3 we introduced the two parameters regulating the importance of the two auxiliary losses described in Section 3.
Revised: In Eq 3 we introduced the two parameters regulating the importance of the two auxiliary losses described in Section 3.

Original: Those parameters were kept fixed to, respectively, 0 . 09 and 0 .
Revised: Those parameters were kept fixed to, respectively, 0 . 09 and 0 .

Original: 00099 for all the experiments
Revised: 00099 for all the experiments

Original: and were chosen via cross validation.
Revised: and were chosen via cross validation.

Original: In Figure 6 we show how the Fidelity over the validation set changes with different combinations of such hyper-parameters. 
Revised: In Figure 6 we show how the Fidelity over the validation set changes with different combinations of such hyper-parameters. [Figure] A. L OCAL E XPLANATIONS E MBEDDING

Revised: For layout reasons

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: we did not report in the main paper the 2D embedding plot for each dataset. However, we believe it is of great interest since it gives a visual sense of how similar local explanations get clustered in the same concept. Thus, we report in Figure 10 the 2D plot of local explanations embedding for BAMultiShapes, Mutagenicity

Original: [Figure]
Revised: [Figure]

Revised: and HIN.

Revised: In addition to this

Original: [Figure]
Revised: [Figure]

Original: C C
Revised: we report a graphical materialization of five random samples for each concept, in Figures [Figure]

Original: C C C C 
Revised: C C C C C C

Original: O
Revised: O

Original: O O
Revised: O O

Original: H
Revised: H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: N N
Revised: N N

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C
Revised: C C

Original: O
Revised: O

Original: H H H H
Revised: H H H H

Original: H H H
Revised: H H H

Original: H
Revised: H

Original: H H H H
Revised: H H H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: N
Revised: N

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: C C C C
Revised: C C C C

Original: C
Revised: C

Original: C C
Revised: C C

Original: C
Revised: C

Original: O
Revised: O

Original: O O O
Revised: O O O

Original: H
Revised: H

Original: H
Revised: H

Original: HH
Revised: HH

Original: H
Revised: H

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: C C
Revised: C C

Original: C C C
Revised: C C C

Original: C C C
Revised: C C C

Original: O O O
Revised: O O O

Original: O
Revised: O

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: H H
Revised: H H

Original: H H N
Revised: H H N

Original: N
Revised: N

Original: Class
Revised: Class

Original: C
Revised: C

Original: C C
Revised: C C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C C
Revised: C C C

Original: O O
Revised: O O

Original: H
Revised: H

Original: H H H 
Revised: H H H H

Original: H HH
Revised: H H

Original: H H
Revised: H H

Original: N
Revised: N

Original: S
Revised: S

Original: C C
Revised: C C

Original: C
Revised: C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C
Revised: C C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: O O
Revised: O O

Original: O O O O
Revised: O O O O

Original: H
Revised: H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: N N
Revised: N N

Original: N
Revised: N

Original: C C
Revised: C C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: O O
Revised: O O

Original: O O
Revised: O O

Original: H H
Revised: H H

Original: H H H
Revised: H H H

Original: H N
Revised: H N

Original: N
Revised: N

Original: C C
Revised: C C

Original: C C
Revised: C C

Original: C C
Revised: C C

Original: C C
Revised: C C

Original: C C
Revised: C C

Original: H H H
Revised: H H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H H H
Revised: H H H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: Br Br
Revised: Br Br

Original: C C
Revised: C C

Original: C C
Revised: C C

Original: C
Revised: C

Original: C H
Revised: C H

Original: H H H H H H
Revised: H H H H H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H
Revised: H

Original: H H H H H H H H
Revised: H H H H H H H H

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: N
Revised: N

Original: N N
Revised: N N

Original: N
Revised: N

Original: N N N N
Revised: N N N N

Original: N P
Revised: N P

Original: P P
Revised: P P

Original: C C
Revised: C C

Original: C CC
Revised: C C

Original: C C C 
Revised: C C C C

Original: H
Revised: H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: Class
Revised: Class

Original: C
Revised: C

Original: C C C C
Revised: C C C C

Original: C
Revised: C

Original: C C
Revised: C C

Original: O O
Revised: O O

Original: Cl Cl
Revised: Cl Cl

Original: Cl
Revised: Cl

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: C C C
Revised: C C C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C
Revised: C C

Original: C
Revised: C

Original: H H H
Revised: H H H

Original: H
Revised: H

Original: H
Revised: H

Original: H
Revised: H

Original: H H H H
Revised: H H H H

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: CCC
Revised: CCC

Original: C
Revised: C

Original: C C
Revised: C C

Original: C
Revised: C

Original: C C C
Revised: C C C

Original: C C C
Revised: C C C

Original: CC
Revised: CC

Original: C C C
Revised: C C C

Original: C C
Revised: C C

Original: C C C
Revised: C C C

Original: CCC
Revised: CCC

Original: C
Revised: C

Original: C C C C C
Revised: C C C C C

Original: C
Revised: C

Original: O O
Revised: O O

Original: O
Revised: O

Original: O O O
Revised: O O O

Original: H H H H
Revised: H H HH

Original: H H H H
Revised: H H H H

Original: H
Revised: H

Original: HH HH
Revised: HH HH

Original: HH
Revised: HH

Original: H H H H H
Revised: H H H HH

Original: HH
Revised: HH

Original: HHHH
Revised: HHHH

Original: HH
Revised: HH

Original: H
Revised: H

Original: H H H H H
Revised: H H H H H

Original: H H H H H H
Revised: H H H H H H

Original: HHH
Revised: HHH

Original: H HH HHH
Revised: H HH HHH

Original: H
Revised: H

Original: N
Revised: N

Original: N N N
Revised: N N N

Original: S
Revised: S

Original: A
Revised: A

Original: N N
Revised: N N

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: A
Revised: A

Original: N
Revised: N

Original: N N
Revised: N N

Original: A
Revised: A

Original: Ego
Revised: Ego

Original: P
Revised: P

Original: N
Revised: N

Original: N N
Revised: N N

Original: P P
Revised: P P

Original: N
Revised: N

Original: P N
Revised: P N

Original: N
Revised: N

Original: A
Revised: A

Original: Ego
Revised: Ego

Original: M M
Revised: M M

Original: A
Revised: A

Original: P
Revised: P

Original: M N
Revised: M N

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: A
Revised: A

Original: N
Revised: N

Original: A
Revised: A

Original: N N
Revised: N N

Original: N
Revised: N

Original: Class
Revised: Class

Original: N N N
Revised: N N N

Original: A
Revised: A

Original: N
Revised: N

Original: A
Revised: A

Original: Ego
Revised: Ego

Original: N N
Revised: N N

Original: N
Revised: N

Original: N P
Revised: N P

Original: N
Revised: N

Original: P
Revised: P

Original: Ego
Revised: Ego

Original: N P
Revised: N P

Original: P
Revised: P

Original: P P
Revised: P P

Original: N N
Revised: N N

Original: N
Revised: N

Original: N N
Revised: N N

Original: A A
Revised: A A

Original: A
Revised: A

Original: N N
Revised: N N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: A
Revised: A

Original: Ego
Revised: Ego

Original: N N N
Revised: N N N

Original: P
Revised: P

Original: N P
Revised: N P

Original: N
Revised: N

Original: P M P
Revised: P M P

Original: P
Revised: P

Original: N N M
Revised: N N M

Original: M
Revised: M

Original: N
Revised: N

Original: P
Revised: P

Original: Ego
Revised: Ego

Original: N N
Revised: N N

Original: A
Revised: A

Original: N
Revised: N

Original: N M M
Revised: N M M

Original: P
Revised: P

Original: N
Revised: N

Original: A
Revised: A

Original: N
Revised: N

Original: M
Revised: M

Original: Ego
Revised: Ego

Original: A
Revised: A

Original: M
Revised: M

Original: N
Revised: N

Original: M M
Revised: M M

Original: Ego
Revised: Ego

Original: A
Revised: A

Original: M
Revised: M

Original: M M
Revised: M M

Original: N
Revised: N

Original: [Figure]
Revised: Class

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: P
Revised: P

Original: N M
Revised: N M

Original: N N 
Revised: N N N

Original: N N 
Revised: N N M

Original: N
Revised: Ego M P

Original: M
Revised: M

Original: Ego M P 
Revised: P N

Original: M
Revised: M

Original: P N M M
Revised: M

Original: M
Revised: M

Original: Ego
Revised: Ego

Original: M
Revised: M

Original: A
Revised: A

Original: N
Revised: N

Original: A. L OCAL E XPLANATIONS E MBEDDING
Revised: [Figure] p p local explanations embeddings

Original: For layout reasons, we did not report in the main paper the 2D embedding plot for each dataset.
Revised: NO2 Others 1. prototype assignments p0 p principal comp.0.

Original: However, we believe it is of great interest since it gives a visual sense of how similar local explanations get clustered in the same concept.
Revised: p p p p local explanations embeddings M N

Original: Thus, we report in Figure 10 the 2D plot of local explanations embedding for BAMultiShapes, Mutagenicity, and HIN.
Revised: NA PAT NP MN MP OTHERS 0. prototype assignments p0 p p2 p principal comp.

Original: [Figure]
Revised: [Figure]

Revised: r i n c i p a l c o m p .

Original: HIN
Revised: HIN

Original: In addition to this, we report a graphical materialization of five random samples for each concept, in

Original: Figures

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Others

Original: C C
Revised: C C

Original: C O
Revised: C O

Original: O O
Revised: O O

Original: S
Revised: S

Original: C O
Revised: C O

Original: O O
Revised: O O

Original: S
Revised: S

Original: C C
Revised: C C

Original: O O
Revised: O O

Original: O O
Revised: O O

Original: S
Revised: S

Original: P [Figure] NO
Revised: P 1 NO

Original: O O
Revised: O O

Original: N
Revised: N

Original: O O
Revised: O O

Original: N
Revised: N

Original: O O
Revised: O O

Original: N
Revised: N

Original: C O
Revised: C O

Original: O
Revised: O

Original: N
Revised: N

Original: O O
Revised: O O

Original: N
Revised: N

Original: [Figure]
Revised: [Figure]

Original: (+ Adm.)
Revised: (+ Adm.)

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: Ego N
Revised: Ego N

Original: N N
Revised: N N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: A
Revised: A

Original: P 1 Patients
Revised: P [Figure] Patients

Original: P
Revised: P

Original: Ego
Revised: Ego

Original: P
Revised: P

Original: Ego
Revised: Ego

Original: P
Revised: P

Original: Ego
Revised: Ego

Original: P
Revised: P

Original: P P
Revised: P P

Original: Ego Ego
Revised: Ego Ego

Original: P P
Revised: P P

Original:  Doctors (+ Nur./Pat.)
Revised: [Figure] Doctors (+ Nur./Pat.)

Original: D
Revised: D

Original: Ego Ego
Revised: Ego Ego

Original: D D
Revised: D D

Original: D Ego
Revised: D Ego

Original: D D
Revised: D D

Original: Ego Ego
Revised: Ego Ego

Original: D
Revised: D

Original: D P
Revised: D P

Original: [Figure]
Revised: [Figure]

Original: Nurses +Patients
Revised: Nurses +Patients

Original: P
Revised: P

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: N
Revised: N

Original: P
Revised: P

Original: Ego Ego
Revised: Ego Ego

Original: N
Revised: N

Original: P
Revised: P

Original: N
Revised: N

Original: Ego
Revised: Ego

Original: P
Revised: P

Revised: P N

Revised: P Ego

Original: P N
Revised: Here we will describe in mode detail the metrics briefly introduced in Section 4.3: • Fidelity measures the accuracy between the prediction of the E-LEN and the one of the GNN to explain. It is computed as the accuracy between the class predictions of the ELEN and the GNN f . • Formula Accuracy Barbiero et al.

Original: P Ego
Revised: (2021) represents how well the learned formulas can correctly predict the class labels. To compute this metric, we treat the final formulas as a classifier that given an input concept vector predicts the class corresponding to the clause evaluated to true. In the cases in which either no clause or more clauses of different classes are evaluated to be true, the sample is always considered as wrongly predicted. • Concept Purity is computed for every cluster independently and measures how good the embedding is at clustering the local explanations.

Revised: It was first proposed in Magister et al.

Revised: (2021) for evaluating concept representations by means of graph edit distance.

Revised: However, since the computation of the graph edit distance is expensive, in our work we adapted such metric to exploit the annotation of local explanations as described in Section 4.3.

Revised: Specifically, in our cases such annotation corresponds to the typology of the motif represented by the local explanation.

Revised: The computation of the metric can be summarized by:

Revised: [Equation]

Revised: where C i corresponds to the cluster having p i as prototype (i.e., the cluster containing every local explanation associated to prototype p i by the distance function d ( ., . )

Revised: described in Section 3).

Revised: count most frequent label ( C i ) instead returns the number of local explanations annotated with the most present label in cluster C i .

Revised: The Concept Purity results reported in Table 2 are computed by taking the mean and the standard deviation across the m clusters.

Revised: A. E NTROPY - BASED L OGIC E XPLAINED N ETWORKS

Revised: Concept-based classifiers (Koh et al., 2020) are a family of machine learning models predicting class memberships from the activation scores of k human-understandable categories, i.e., q : C (cid:55)→ Y , where C ⊆ [0 , 1] k .

Revised: Concept-based classifiers improve human understanding as their input and output spaces consist of interpretable symbols (Wu et al., 2018; Ghorbani et al., 2019; Koh et al., 2020).

Revised: Logic Explained Networks (LENs (Ciravegna et al., 2021a)) are concept-based neural models providing for each class i simple logic explanations

Revised: ϕ

Revised: i : ¯ C (cid:55)→ { 0 , 1 } for their predictions ¯ q i (¯ c ) ∈ { 0 , 1 } .

Revised: In particular LENs provide concept-based First-Order Logic (FOL) explanations for each classification task:

Revised: [Equation]

Revised: (8) where ϕ i is a concept-based formula in disjunctive normal form.

Revised: The E-LEN employed in our work corresponds to a LEN with an Entropy Layer Barbiero et al.

Revised: (2021) as first layer, which is the responsible for the extraction of the logic formulas.

Revised: As mentioned in Section 4.2, given that our architecture already promotes by construction a singleton activation of concepts, the entropybased regularization described in Barbiero et al.

Revised: (2021) promoting a parsimonious activation of the concepts, allowing the E-LEN to predict class memberships using few relevant concepts only, is removed.

Revised: A.5. E XTRACTION OF LOGIC FORMULAS

Revised: Considering the removal of the entropy-based regularization mentioned above, the process of formula extraction can be summarized as follows: given a classification task with r classes, and given a truth table T i for each of the r classes, the l -th row of the resulting table T i is obtained by concatenating together the l -th input concept activation vector ¯ c l with the respective prediction q

Revised: i (¯ c l ) :

Revised: [Equation]

Revised: Then, for every row l , where q i (¯ c l ) = 1 , concepts in ¯ c l are connected with the AND operator resulting in a logic clause where concepts that appear as false in ¯ c l are negated.

Revised: Finally, the final formulas in disjunctive normal form for table T i are obtained by connecting every clause with the

Revised: OR operator.

Revised: Further details are available in Barbiero et al.

Revised: A.5. E NHANCEMENTS OF GLGE XPLAINER TO THE E-LEN FRAMEWORK

Revised: To make clear the enhancements of our proposed GLGExplainer to the E-LEN framework, note that the original formulation of the E-LEN requires C to be known.

Revised: Indeed, the experiments carried out in

Revised: Barbiero et al.

Revised: (2021) were all assuming a known mapping from the input space to C .

Revised: In this case, however, since such mapping is not available, we aim at learning a set of human-understandable concepts.

Revised: The choice of learning such concepts from local explanations, rather than generating them via any graph-generation process, allowed our contribution to be faithful to the data domain contrary to previous works on Global Explainability for GNNs Yuan et al.

Revised: Since the E-LEN framework does not provide a principled way for dealing with the peculiarities of the graph domain, it cannot be directly applied on top of local explanations.

Revised: We thus devised the method proposed in

Revised: Section 3 in order to aggregate local explanations into clusters of similar subgraphs, with no further supervision than the one provided by Eq 3.

Revised: The experimental evaluation in Section 4.4 shows that the learned clusters have indeed a human-understandable meaning, and the final formulas correctly allow to assess the performances of the model to explain.

Revised: On the same vein, the framework proposed in Barbiero et al.

Revised: (2021) is aimed at creating an interpretable classifier which can generate logic explanations of its predictions, resulting in an interpretable-by-design architecture.

Revised: This is radically different from our scenario, where we provide explanations of an already trained model.

Revised: Finally, given the fact that we removed the entropy-based regularization from the original implementation of the Entropy Layer 3 , we are basically restricting the usage of it to just the principled process of extracting logic formulas from the truth table T , introduced in Barbiero et al.

Revised: A. D ISTANCE FUNCTION d ( ., . )

Revised: In Section 3 we presented Eq.

Revised: 2 as the distance function

Revised: d ( p i , e ) to compute a relative assignment value for the graph embedding e to prototype p i , expressed as a probability value thanks to the softmax.

Revised: For convenience, we report again the mathematical definition below:

Revised: [Equation]

Revised: Note that when d ( p i , e ) has a high activation value for a prototype p i , then the graph embedding e is very close to prototype p i in the embedding space, thus meaning that the input graph leading to e exhibit a similar high-level concept to what p i represents.

Revised: Overall, the output of the projection step described in Section 3 is a set V

Revised: = { v e , ∀ e ∈ E } where v e =

Revised: [ d ( p 1 , e ) , .., d ( p m , e )] is a vector containing a probabilistic assignment of graph embedding e to the m prototypes.