C ONCEPT - LEVEL D EBUGGING OF P ART -P ROTOTYPE N ETWORKS
A BSTRACT
Part-prototype Networks (ProtoPNets) are concept-based classiﬁers designed to achieve the same performance as black-box models without compromising transparency.
ProtoPNets compute predictions based on similarity to class-speciﬁc part-prototypes learned to recognize parts of training examples, making it easy to faithfully determine what examples are responsible for any target prediction and why.
However, like other models, they are prone to picking up confounders and shortcuts from the data, thus suffering from compromised prediction accuracy and limited generalization.
We propose ProtoPDebug, an effective concept-level debugger for ProtoPNets in which a human supervisor, guided by the model’s explanations, supplies feedback in the form of what part-prototypes must be forgotten or kept, and the model is ﬁne-tuned to align with this supervision.
Our experimental evaluation shows that ProtoPDebug outperforms state-of-the-art debuggers for a fraction of the annotation cost.
An online experiment with laypeople conﬁrms the simplicity of the feedback requested to the users and the effectiveness of the collected feedback for learning confounder-free part-prototypes.
ProtoPDebug is a promising tool for trustworthy interactive learning in critical applications, as suggested by a preliminary evaluation on a medical decision making task.
I NTRODUCTION
Part-Prototype Networks (ProtoPNets) are “gray-box” image classiﬁers that combine the transparency of case-based reasoning with the ﬂexibility of black-box neural networks (Chen et al., 2019).
They compute predictions by matching the input image with a set of learned part-prototypes – i.e., prototypes capturing task-salient elements of the training images, like objects or parts thereof – and then making a decision based on the part-prototype activations only.
What makes ProtoPNets appealing is that, despite performing comparably to more opaque predictors, they explain their own predictions in terms of relevant part-prototypes and of examples that these are sourced from.
These explanations are – by design – more faithful than those extracted by post-hoc approaches (Dombrowski et al., 2019; Teso, 2019; Lakkaraju & Bastani, 2020; Sixt et al., 2020) and can effectively help stakeholders to simulate and anticipate the model’s reasoning (Hase & Bansal, 2020).
Despite all these advantages, ProtoPNets are prone – like regular neural networks – to picking up confounders from the training data (e.g., class-correlated watermarks), thus suffering from compromised generalization and out-of-distribution performance (Lapuschkin et al., 2019; Geirhos et al., 2020).
This occurs even with well-known data sets, as we will show, and it is especially alarming as it can impact high-stakes applications like COVID-19 diagnosis (DeGrave et al., 2021) and scientiﬁc analysis (Schramowski et al., 2020).
We tackle this issue by introducing ProtoPDebug, a simple but effective interactive debugger for
ProtoPNets that leverages their case-based nature.
ProtoPDebug builds on three key observations: (i)
In ProtoPNets, confounders – for instance, textual meta-data in X-ray lung scans (DeGrave et al., 2021) and irrelevant patches of background sky or foliage (Xiao et al., 2020) – end up appearing as part-prototypes; (ii) Sufﬁciently expert and motivated users can easily indicate which part-prototypes are confounded by inspecting the model’s explanations; (iii) Concept-level feedback of this kind is context-independent, and as such it generalizes across instances.
The
In short, ProtoPDebug leverages the explanations naturally output by ProtoPNets to acquire concept-level feedback about confounded (and optionally high-quality) part-prototypes, as illustrated in Fig.
Then, it aligns the model using a novel pair of losses that penalize part-prototypes for behaving similarly to confounded concepts, while encouraging the model to remember highquality concepts, if any.
ProtoPDebug is ideally suited for human-in-the-loop explanation-based debugging (Kulesza et al., 2015; Teso & Kersting, 2019), and achieves substantial savings in terms of annotation cost compared to alternatives based on input-level feedback (Barnett et al., 2021).
In fact, in contrast to the per-pixel relevance masks used by other debugging strategies (Ross et al., 2017; Teso & Kersting, 2019; Plumb et al., 2020; Barnett et al., 2021), concept-level feedback automatically generalizes across instances, thus speeding up convergence and preventing relapse.
Our experiments show that ProtoPDebug is effective at correcting existing bugs and at preventing new ones on both synthetic and real-world data, and that it needs less corrective supervision to do so than state-of-the-art alternatives.
Contributions.
Summarizing, we:
(1) Highlight limitations of existing debuggers for black-box models and ProtoPNets; (2) Introduce ProtoPDebug, a simple but effective strategy for debugging
ProtoPNets that drives the model away from using confounded concepts and prevents forgetting well-behaved concepts; (3) Present an extensive empirical evaluation showcasing the potential of
ProtoPDebug on both synthetic and real-world data sets.
P ART -P ROTOTYPE N ETWORKS
ProtoPNets (Chen et al., 2019) classify images into one of v classes using a three-stage process comprising an embedding stage, a part-prototype stage, and an aggregation stage; see Fig.
1 (left).
Embedding stage: Let x be an image of shape w × h × d , where d is the number of channels.
The embedding stage passes x through a sequence of (usually pre-trained) convolutional and pooling layers with parameters φ , obtaining a latent representation z
= h ( x ) of shape w (cid:48) × h (cid:48) ×
d (cid:48) , where w (cid:48) <
w and h (cid:48) < h .
Let Q ( z ) be the set of 1 × 1 × d (cid:48) subtensors of z .
Each such subtensor q ∈ Q ( z ) encodes a ﬁlter in latent space and maps a rectangular region of the input image x .
Part-prototype stage: This stage memorizes and uses k partprototypes P = { p 1 , . .
, p k } .
Each p j is a tensor of shape 1 × 1 × d (cid:48) explicitly learned – as explained below – so as to capture salient visual concepts appearing in the training images, like heads or wings.
The activation of a part-prototype p on a part q ∈ Q ( z ) is given by a difference-of-logarithms function, deﬁned as (Chen et al., 2019): [Equation] p
[Equation] [Equation]
Here, (cid:107)·(cid:107) indicates the L 2 norm and (cid:15) > 0 is a small constant.
Alternatively, one can employ an exponential activation, deﬁned as (Hase & Bansal, 2020):
[Equation]
Both activation functions are bell-shaped and decrease monotonically with the distance between q and p , as illustrated in Fig.
The image-wide activation of p is obtained via max-pooling, i.e., act( p , z ) :
=
max q ∈Q ( z ) act( p , q ) .
Given an input image x , this stage outputs an activation vector that captures how much each part-prototype activates on the image:
[Equation]
The activation vector essentially encodes the image according to the concept vocabulary.
Each class y ∈ [ v ] := { 1 , . .
. , v } is assigned (cid:98) kv (cid:99) discriminative part-prototypes P y ⊆
P learned so as to strongly activate (only) on training examples of that class, as described below.
Aggregation stage: In this stage, the network computes the score of class y by aggregating the concept activations using a dense layer, i.e., (cid:80) j w yj a j ( x ) , where w y ∈ R k is the weight vector of class y , and applies a softmax function to obtain conditional probabilities p θ ( y | x ) .
The set of parameters appearing in the ProtoPNet will be denoted by θ .
Loss and training.
The network is ﬁt by minimizing a compound empirical loss over a data set D :
[Equation]
This comprises a cross-entropy loss (ﬁrst term) and two regularization terms that encourage the part-prototypes to cluster the training set in a discriminative manner:
[Equation]
Speciﬁcally, the part-prototypes are driven by the clustering loss (cid:96) cls to cover all examples of their associated class and by the separation loss (cid:96) sep not to activate on examples of other classes.
During training, the embedding and part-prototype layers are ﬁt jointly, so as to encourage the data to be embedded in a way that facilitates clustering.
At this time, the aggregation weights are ﬁxed: each weight w uj of p j ∈ P y is set to 1 if u = y and to − 0 .
5 otherwise.
The aggregation layer is ﬁt in a second step by solving a logistic regression problem.
ProtoPNets guarantee the part-prototypes to map to concrete cases by periodically projecting them onto the data.
Speciﬁcally, each p ∈ P y is replaced with the (embedding of the) closest image part from any training example of class y .
Explanations.
The architecture of ProtoPNets makes it straightforward to extract explanations highlighting, for each part-prototype p : (1) its relevance for the target decision ( x , y ) , given by the score w j a j ( x ) ; (2) its attribution map attr( p , x ) , obtained by measuring the activation of p on each part q of x , and then upscaling the resulting matrix to w × h using bilinear ﬁltering:
[Equation]
(3) the source example that it projects onto.
I NPUT - LEVEL D EBUGGING S TRATEGIES AND T HEIR L IMITATIONS
Explanations excel at exposing confounders picked up by models from data (Lapuschkin et al., 2019; Geirhos et al., 2020), hence constraining or supervising them can effectively dissuade the model from acquiring those confounders.
This observation lies at the heart of recent approaches for debugging machine learning models, see (Teso et al., 2022) for an overview.
A general recipe.
The general strategy is well illustrated by the right for the right reasons (RRR) loss (Ross et al., 2017), which aligns the attribution maps produced by a predictor to those supplied by a human annotator.
Let f : x (cid:55)→ y be a differentiable classiﬁer and IG ( f, ( x , y ))
∈
R d be the input gradient of f for a decision ( x , y ) .
This attribution algorithm (Baehrens et al., 2010;
Simonyan et al., 2014) assigns relevance to each x j for the given decision based on the magnitude of the gradient w.r.t.
x j of the predicted probability of class y .
The RRR loss penalizes f for associating non-zero relevance to known irrelevant input variables:
[Equation]
Here, m is the ground-truth attribution mask of example ( x , y ) , i.e., m j is 0 if the j -th pixel of x is irrelevant for predicting the ground-truth label and 1 otherwise.
Other recent approaches follow the same recipe (Rieger et al., 2020; Selvaraju et al., 2019; Shao et al., 2021; Viviano et al., 2021); see Lertvittayakumjorn & Toni (2021) and Friedrich et al.
for an in-depth overview.
Debugging ProtoPNets.
A natural strategy to debug a ProtoPNet with a confounded part-prototype p is to delete the latter and ﬁne-tune the remaining part-prototypes P (cid:48) :
= P \ { p } .
This solution is however very fragile, as nothing prevents the updated network from re-acquiring the same confounder using one of its remaining part-prototypes.
One alternative is to freeze the part-prototypes
P (cid:48) and ﬁne-tune the aggregation stage only, but this also problematic: in the presence of a strong confound, the model ends up relying chieﬂy on p , with the remaining part-prototypes P (cid:48) being low quality, not discriminative, and associated to low weights.
Fine-tuning the aggregation layer on P (cid:48) then typically yields poor prediction performance.
These issues prompted the development of the only existing debugger for ProtoPNets, IAIABL (Barnett et al., 2021).
It also ﬁts the above template, in that – rather than removing confounded part-prototypes – it penalizes those part-prototypes that activate on pixels annotated as irrelevant.
The IAIA-BL loss is deﬁned as:
[Equation]
where attr is the ProtoPNet’s attribution map (see Eq.
(7)) and (cid:12) indicates the element-wise product.
The ﬁrst inner summation is analogous to the RRR loss, while the second one encourages the partprototypes of other classes not to activate at all, thus reinforcing the separation loss in Eq.
Limitations.
A critical issue with these approaches is that they are restricted to pixel-level supervision, which is inherently local: an attribution map that distinguishes between object and background in a given image does not generally carry over to other images.
This entails that, in order to see any beneﬁts for more complex neural nets, a substantial number of examples must be annotated individually, as shown by our experiments (see Fig. 3).
This is especially troublesome as ground-truth attribution maps are not cheap to acquire, partly explaining why most data sets in the wild do not come with such annotations.
Put together, these issues make debugging with these approaches cumbersome, especially in humanin-the-loop applications where, in every debugging round, the annotator has to supply a nonnegligible number of pixel-level annotations.
C ONCEPT - LEVEL D EBUGGING WITH P ROTO PD EBUG
Our key observation is that in ProtoPNets confounders only inﬂuence the output if they are recognized by the part-prototypes, and that therefore they can be corrected for by leveraging concept-level supervision , which brings a number of beneﬁts.
We illustrate the basic intuition with an example:
Example: Consider the confounded bird classiﬁcation task in Fig.
1 (right).
Here, all training images of class Crested Auklet have been marked with a blue square “ (cid:4) ”, luring the network into relying heavily on this confound, as shown by the misclassiﬁed Laysian Albatro image.
By inspecting the learned part-prototypes – together with source images that they match strongly, for context – a sufﬁciently expert annotator can easily indicate the ones that activate on confounders, thus naturally providing concept-level feedback.
Concept-level supervision sports several advantages over pixel-level annotations:
It is cheap to collect from human annotators, using an appropriate UI, by showing part-prototype activations on selected images and acquiring click-based feedback.
It is very informative , as it distinguishes between content and context and thus generalizes across instances.
In our example, the blue square is a nuisance regardless of what image it appears in .
In this sense, one concept-level annotation is equivalent to several input-level annotations.
By generalizing beyond individual images, it speeds up convergence and prevents relapse.
The debugging loop.
Building on these observations, we develop ProtoPDebug, an effective and annotation efﬁcient debugger for ProtoPNets that directly leverages concept-level supervision, which we describe next.
The pseudo-code for ProtoPDebug is listed in Algorithm 1.
Algorithm f ProtoPNet on D . 1: initialize F ← ∅ , V ← ∅ 2: while True do 3: for p ∈ P do 4: for each ( x , y ) of the a training examples most activated by p do 5: if p looks confounded then 6: add cut-out x R to F 7: else if p looks high-quality then 8: add cut-out x R to V 9: if no confounders found then 10: break 11: ﬁne-tune f by minimizing (cid:96) ( θ ) + λ f (cid:96) for ( θ ) + λ r (cid:96) rem ( θ ) 12: return f
ProtoPDebug takes a ProtoPNet f and its training set D and runs for a variable number of debugging rounds.
In each round, it iterates over all learned part-prototypes p ∈ P , and for each of them retrieves the a training examples ( x , y ) that it activates the most on, including its source example.
It then asks the user to judge p by inspecting its attribution map on each selected example.
If the user indicates that a particular activation of p looks confounded, ProtoPDebug extracts a “cut-out” x R of the confounder from the source image x , deﬁned as the box (or boxes, in case of disconnected activation areas) containing 95% of the part-prototype activation.
It then embeds x R using h and adds it to a set of forbidden concepts F .
This set is organized into class-speciﬁc subsets F y , and the user can specify whether x R should be added to the forbidden concepts for a speciﬁc class y or to all of them (class-agnostic confounder).
Conversely, if p looks particularly high-quality to the user, ProtoPDebug embeds its cut-out and adds the result to a set of valid concepts V .
This is especially important when confounders are ubiquitous, making it hard for the ProtoPNet to identify (and remember) non-confounded prototypes (see
COVID experiments in Section 5.2).
Once all part-prototypes have been inspected, f is updated so as to steer it away from the forbidden concepts in F while alleviating forgetting of the valid concepts in V using the procedure described below.
Then, another debugging round begins.
The procedure terminates when no confounder is identiﬁed by the user.
Fine-tuning the network.
During ﬁne-tuning, we search for updated parameters θ (cid:48)
= { φ (cid:48) , P (cid:48) , w (cid:48) } that are as close as possible to θ – thus retaining all useful information that f has extracted from the data – while avoiding the bugs indicated by the annotator.
This can be formalized as a constrained distillation problem (Gou et al., 2021):
[Equation]
where d is an appropriate distance function between sets of parameters and by “consistent” we mean that the ProtoPNet parameterized by θ (cid:48) activates little or at all on the part-prototypes in f .
Since the order of part-prototypes is irrelevant, we deﬁne it as a permutation-invariant Euclidean distance:
[Equation]
Here, π simply reorders the part-prototypes of the buggy and updated models so as to maximize their alignment.
Recall that we are interested in correcting the concepts, so we focus on the middle term.
Notice that the logarithmic and exponential activation functions in Eqs.
(1) and (2) are both inversely proportional to (cid:107) p − p (cid:48)
(cid:107) 2 and achieve their maximum when the distance is zero, hence minimizing the distance is analogous to maximizing the activation.
This motivates us to introduce two new penalty terms:
[Equation]
The hard constraint in Eq.
(10) complicates optimization, so we replace it with a smoother penalty function, obtaining a relaxed formulation argmin θ (cid:48)
d ( θ, θ (cid:48) )
+ λ · (cid:96) for ( θ ) .
The forgetting loss (cid:96) for minimizes how much the part-prototypes of each class y ∈ [ v ] activate on the most activated concept [Figure] to be forgotten for that class, written F y .
Conversely, the remembering loss (cid:96) rem maximizes how much the part-prototypes activate on the least activated concept to be remembered for that class, denoted V y .
The overall loss used by ProtoPDebug for ﬁne-tuning the model is then a weighted combination of the ProtoPNet loss in Eq.
(4) and the two new losses, namely (cid:96) ( θ ) + λ f (cid:96) for ( θ ) + λ r (cid:96) rem ( θ ) , where λ f and λ r are hyper-parameters.
We analyze the relationship between Eqs.
(12) and (13) and other losses used in the literature in Appendix A.
Beneﬁts and limitations.
The key feature of ProtoPDebug is that it leverages concept-level supervision, which sports improved generalization across instances, cutting annotation costs and facilitating interactive debugging.
ProtoPDebug naturally accommodates concept-level feedback that applies to speciﬁc classes.
E.g., snow being useful for some classes (say, winter ) but irrelevant for others (like dog or wolf ) (Ribeiro et al., 2016).
However, ProtoPDebug makes it easy to penalize part-prototypes of all classes for activating on class-agnostic confounders like, e.g., image artifacts.
ProtoPDebug’s two losses bring additional, non-obvious beneﬁts.
The remembering loss helps to prevent catastrophic forgetting during sequential debugging sessions, while the forgetting loss prevents the model from re-learning the same confounder in the future.
One source of concern is that, if the concepts acquired by the model are not understandable, debugging may become challenging.
ProtoPNets already address this issue through the projection step, but following Koh et al.
; Marconato et al.
(2022), one could guide the network toward learning a set of desirable concepts by supplying additional concept-level supervision and appropriate regularization terms.
More generally, like other explanation-based methods, in sensitive prediction tasks
ProtoPDebug poses the risk of leaking private information.
Moreover, malicious annotators may supply adversarial supervision, corrupting the model.
These issues can be avoided by restricting access to ProtoPDebug to trusted annotators only.
E MPIRICAL A NALYSIS
In this section, we report results showing how concept-level supervision enables ProtoPDebug to debug ProtoPNets better than the state-of-the-art debugger IAIA-BL and using less and cheaper supervision, and that it is effective in both synthetic and real-world debugging tasks.
ProtoPDebug and IAIA-BL were implemented on top of ProtoPNets (Chen et al., 2019) using the Adam optimizer (Kingma & Ba, 2014).
5. C ONCEPT - LEVEL VS INSTANCE - LEVEL DEBUGGING
We ﬁrst compare the effectiveness of concept-level and instance-level supervision in a controlled scenario where the role of the confounders is substantial and precisely deﬁned at design stage.
To this end, we modiﬁed the CUB200 data set (Wah et al., 2011), which has been extensively used for evaluating ProtoPNets (Chen et al., 2019; Hase & Bansal, 2020; Rymarczyk et al., 2021; Hoffmann et al., 2021; Nauta et al., 2021).
This data set contains images of 200 bird species in natural environments, with approximately 40 train and 20 test images per class.
We selected the ﬁrst ﬁve classes, and artiﬁcially injected simple confounders in the training images for three of them.
The confounder is a colored square of ﬁxed size, and it is placed at a random position.
The color of the square is different for different classes, but it is absent from the test set, and thus acts as a perfect confounder for these three classes.
The resulting synthetic dataset is denoted CUB5 box .
Competitors and implementation details.
We compare ProtoPDebug, with supervision on a single instance per confounder , against the following competitors: 1) vanilla ProtoPNets; 2) a vanilla ProtoPNet trained on a clean version of the dataset without the confounders, denoted ProtoPNets clean ; 3) IAIA-BL X , the IAIA-BL model ﬁt using ground-truth attribution maps on X % of the training examples, for X ∈ { 5 , 20 , 100 } , and IAIA-BL n = 3 , with one attribution mask per confound (same number given to ProtoPDebug).
This setting is not interactive, but supervision is made available at the beginning of training.
The embedding layers were implemented using a pre-trained VGG-16, allocating two prototypes for each class.
The values of λ f and a were set to 100 and 5 , respectively, and λ iaia to 0 .
001 , as in the original paper (Barnett et al., 2021); experiments with different values of λ iaia gave inconsistent results.
In this setting, all supervision is made available to all methods from the beginning, and the corrective feedback fed to the forgetting loss is class-speciﬁc.
Results .
Fig.
3 reports the experimental results on all methods averaged over 15 runs.
The left and middle plots show the training set macro F 1 and cross entropy, respectively.
After 15 epochs, all methods manage to achieve close to perfect F 1 and similar cross entropy.
The right plot shows macro F 1 on the test set.
As expected, the impact of the confounders is rather disruptive, as can been seen from the difference between ProtoPNets and ProtoPNets clean .
Indeed, ProtoPNets end up learning confounders whenever present.
Instance-level debugging seems unable to completely ﬁx the problem.
With the same amount of supervision as ProtoPDebug, IAIA-BL n = 3 does not manage to improve over the ProtoPNets baseline.
Increasing the amount of supervision does improve its performance, but even with supervision on all examples IAIA-BL still fails to match the confound-free baseline ProtoPNets clean (IAIA-BL 100 curve).
On the other hand, ProtoPDebug succeeds in avoiding to be fooled by confounders, reaching the performance of ProtoPNets clean despite only receiving a fraction of the supervision of the IAIA-BL alternatives.
Note however that
ProtoPDebug does occasionally select a natural confounder (the water), that is not listed in the set of forbidden ones.
In the following section we show how to deal with unknown confounders via the full interactive process of ProtoPDebug
. 5. P ROTO PD EBUG IN THE REAL WORLD
CUB5 nat data set.
Next, we evaluate ProtoPDebug in a real world setting in which confounders occur naturally in the data as, e.g., background patches of sky or sea, and emerge at different stages of training, possibly as a consequence of previous user corrections.
To maximize the impact of the natural confounders, we selected the 20 CUB200 classes with largest test F 1 difference when training
ProtoPNets only on birds (removing background) or on entire images (i.e., bird + background), and testing them on images where the background has been shufﬂed among classes.
Then, we applied
ProtoPDebug focusing on the ﬁve most confounded classes out of these 20.
We call this dataset
CUB5 nat .
All architectures and hyperparameters are as before.
Experimental Protocol.
To evaluate the usability and effectiveness of ProtoPDebug with real users, we ran an online experiment, through Proliﬁc Academic (proliﬁc.com), one of the most reliable crowdsourcing platforms for behavioral research.
Participants were laypeople, all native speakers of
English.
To make the experimental stimuli as clear as possible, separate patches for the activation area of the prototype on an image were automatically extracted from the 5 most-activate images ( a = 5 ).
See Appendix H for the detailed procedure.
For each debugging round, a new group ofparticipants was asked to inspect the images and to determine whether the highlighted patch covered some part of the bird or whether, instead, it covered exclusively (or very nearly so) the background.
A patch was ﬂagged as a confounder when 80% of the users indicated it as covering the background.
A prototype was accepted when no confounder was identiﬁed in its most activated image.
Regardless of this, confounders in less activated images of the prototype were still collected as they might serve as feedback for other prototypes (class-agnostic confounders) in possible following iteration(s).
The procedure ended when all prototypes were accepted.
Results.
The average evaluation time for each image was 6 seconds.
Across all images, participants’ inter-annotation agreement was 96%, conﬁrming the simplicity of the users’ feedback required by
ProtoPDebug.
Fig.
4 shows the part-prototypes progressively learned by ProtoPDebug.
In the ﬁrst iteration, when no corrective feedback has been provided yet, the model learns confounders for most classes: the branches for the second and the last classes, the sky for the third one.
Upon receiving user feedback, ProtoPDebug does manage to avoid learning the same confounders again in most cases, but it needs to face novel confounders that also correlate with the class (i.e., the sea for the
[Figure]
(cid:51) (cid:51) (cid:51) (cid:55) (cid:51) (cid:51) (cid:51) (cid:55) (cid:51) (cid:51) (cid:51) (cid:51) (cid:55) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:51) (cid:55) (cid:51) (cid:51) (cid:51) (cid:51)
ProtoPNet
C LASS
{+F 1 AP 1 AP 0.48 0.75 0.750.38 0.73 0.480.
+}
Avg.
ProtoPDebug
C LASS
F 1 AP 1 AP 0.54 0.84 0.840.64 0.94 0.950.93 0.89 0.890.52 0.95 0.950.82 0.89 0.
Avg.
Fig.
4 reports the test set performance of ProtoPNet compared to ProtoPDebug in terms of F 1 and interpretability metric.
The latter has been introduced in (Barnett et al., 2021, Eq. 6) to quantify the portion of pixels that are activated by a prototype.
We slightly modiﬁed it to consider also their activation value (see
Suppl.
for the deﬁnition).
Results show that ProtoPDebug improves over ProtoPNets in all classes (with an increase in F 1 of up to 0.26 for classes 6 and 8), and that it manages to learn substantially more interpretable prototypes (i.e., it is right for the right reason (Schramowski et al., 2020)), with a 22% improvement in activation precision on average.
COVID data set.
The potential of artiﬁcial intelligence and especially deep learning in medicine is huge (EJ., 2019).
On the other hand, the medical domain is known to be badly affected by the presence of confounders (Jager et al., 2008; Smith & Nichols, 2018).
In order to evaluate the potential of
ProtoPDebug to help addressing this issue, we tested it on a challenging real-world problem from the medical imaging domain.
The task is to recognize COVID-19 from chest radiographies.
As shown in DeGrave et al.
(2021), a classiﬁer trained on this dataset heavily relies on confounders that correlate with the presence or absence of COVID.
These confounders come from the image acquisition procedure or annotations on the image.
We trained and tested ProtoPDebug on the same datasets used in DeGrave et al.
(2021) using the data pipeline source code (cod, 2021a).
To simplify the identiﬁcation of confounders, we focused on a binary classiﬁcation task, discriminating COVID-positive images from images without any pathology.
Given the extremely high inter-annotator agreement of the CUB5 nat experiment, this experiment was conducted internally in our lab.
We leave a crowdsourced evaluation to an extended version of this work.
Results .
Fig.
5 reports the (non-zero activation) prototypes of ProtoPDebug at different correction rounds.
As for Fig.
4 (left), the left-most column corresponds to the prototypes learned by ProtoPNets.
Note that for each prototype, the supervision is given to the 10 most-activated images ( a = 10 ).
Penalized confounders have been extracted from images on which the prototype has non-zero activation, because they inﬂuence the classiﬁcation. However, the patches of the images to remember are extracted even if the activation is zero, in order to force the prototype to increase
[Figure]
(cid:55) (cid:55) (cid:55) (cid:55) (cid:55) (cid:55) (cid:51) (cid:51)
R ELATED W ORK
Input-level debuggers.
ProtoPDebug is inspired by approaches for explanatory debugging (Kulesza et al., 2015) and explanatory interactive learning (Teso & Kersting, 2019;
Schramowski et al., 2020), which inject explanations into interactive learning, enabling a human-in-the-loop to identify bugs in the model’s reasoning and ﬁx them by supplying corrective feedback.
They leverage input attributions (Teso & Kersting, 2019;
Selvaraju et al., 2019; Lertvittayakumjorn et al., 2020;
Schramowski et al., 2020), example attributions (Teso et al., 2021; Zylberajch et al., 2021), and rules (Popordanoska et al., 2020), but are not designed for concept-based models nor leverage concept-level explanations.
IAIA-BL (Barnett et al., 2021) adapts these strategies to PPNets by penalizing partprototypes that activate on irrelevant regions of the input, but it is restricted to instance-level feedback, which is neiter as cheap to obtain nor as effective as concept-level feedback, as shown by our experiments.
For an overview of interactive debugging strategies, see Teso et al.
Concept-level debuggers.
Stammer et al.
(2021) debug neuro-symbolic models by enforcing logical constraints on the concepts attended to by a network’s slot attention module (Locatello et al., 2020), but requires the concept vocabulary to be given and ﬁxed.
ProtoPDebug has no such requirement, and the idea of injecting prior knowledge (orthogonal to our contributions) could be fruitfully integrated with it.
ProSeNets (Ming et al., 2019) learn (full) prototypes in the embedding space given by a sequence encoder and enable users to steer the model by adding, removing and manipulating the learned prototypes.
The update step however requires input-level supervision and adapts the embedding layers only.
iProto-TREX (Schramowski et al., 2021) combines ProtoPNets with transformers, and learns part-prototypes capturing task-relevant text snippets akin to rationales (Zaidan et al., 2007).
It supports dynamically removing and hot-swapping bad part-prototypes, but it lacks a forgetting loss and hence is prone to relapse.
Applying ProtoPDebug to iProto-TREX is straightforward and would ﬁx these issues.
Bontempelli et al.
(2021) proposed a classiﬁcation of debugging strategies for concept-based models, but presented no experimental evaluation.
Lage & Doshi-Velez (2020) acquire concepts by eliciting concept-attribute dependency information with questions like “does the concept depression depend on feature lorazepam ?”.
This approach can prevent confounding, but it is restricted to white-box models and interaction with domain experts.
FIND (Lertvittayakumjorn et al., 2020) offers similar functionality for deep NLP models, but it relies on disabling concepts only.
Other concept-based models.
There exist several concept-based models (CBMs) besides ProtoPNets, including self-explainable neural networks (Alvarez-Melis & Jaakkola, 2018), concept bottleneck models (Koh et al., 2020; Losch et al., 2019), and concept whitening (Chen et al., 2020).
Like
ProtoPNets, these models stack a simulatable classiﬁer on top of (non-prototypical) interpretable concepts.
Closer to our setting, prototype classiﬁcation networks (Li et al., 2018) and deep embedded prototype networks (Davoudi & Komeili, 2021) make predictions based on embedding-space prototypes of full training examples (rather than parts thereof).
Our work analyzes ProtoPNets as they perform comparably to other CBMs while sporting improved interpretability (Chen et al., 2019; Hase & Bansal, 2020).
Several extensions of ProtoPNets have also been proposed (Hase et al., 2019; Rymarczyk et al., 2021; Nauta et al., 2021; Kraft et al., 2021).
ProtoPDebug naturally applies to these variants, and could be extended to other CBMs by adapting the source example selection and update steps.
An evaluation on these extensions is left to future work.
Ethics statement ProtoPDebug is conceived for leveraging interactive human feedback to improve the trustworthiness and reliability of machine predictions, which is especially important for critical applications like medical decision making.
On the other hand, adversarial annotators might supply corrupted supervisions to alter the model and drive its predictions towards malignant goals.
For these reasons, access to the model and to the explanations should be restricted to authorized annotators only, as standard in safety-critical applications.
Concerning the online experiment with laypeople, all participants were at least 18 years old and provided informed consent prior to participating in the research activities.
Reproducibility statement All experiments were implemented in Python 3 using Pytorch (Paszke et al., 2019) and run on a machine with two Quadro RTX 5000 GPUs. Each run requires in-between 5 and 30 minutes. We implemented ProtoPDebug and IAIA-BL on top of ProtoPNets and we used the source code of the respective authors of the two methods (cod, 2019; 2021b). The full experimental setup is published on GitHub at https://github.com/abonte/protopdebug and in the Supplementary Material together with additional implementation details.
The repository contains the conﬁguration ﬁles reporting the hyper-parameters of all experiments, and the procedures to process the data and reproduce the experiments.
The implementation details are also available in Appendix C, while Appendix D informs about the data sets, which are all made freely available by their respective authors.
For CUB200 (Wah et al., 2011) and image masks (Farrell, 2022), the image pre-processing reuses the code of the ProtoPNets authors (cod, 2019).
The COVID images are processed using (cod, 2021a).
The images are downloaded from GitHub-COVID repository (Cohen et al., 2020), ChestX-ray14 repository (Wang et al., 2017), PadChest (Bustos et al., 2020) and BIMCV-COVID19+ (Vay ´ a et al., 2020).
Appendix H provides the experimental protocol for the online experiment with real users.
A CKNOWLEDGMENTS
This research of FG has received funding from the European Union’s Horizon 2020 FET Proactive project “WeNet - The Internet of us”, grant agreement No. 823783,
The research of AB was supported by the “DELPhi - DiscovEring Life Patterns” project funded by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) 2017 – DD n. 1062 del 31.05.2019.
The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
A R ELATIONSHIP TO OTHER LOSSES
The forgetting loss in Eq.
(12) can be viewed as a lifted version of the separation loss in Eq.
(6): the former runs over classes and leverages per-class concept feedback, while the latter runs over examples.
The same analogy holds for the remembering and clustering losses, cf.
Eqs.
(5) and (13)
The other difference is that the instance-level losses are proportional to the negative squared distance −(cid:107) p − q (cid:107) 2 between part-prototypes and parts and the concept-level ones to the activation instead.
However, existing activation functions are monotonically decreasing in the squared distance, meaning that they only really differ in fall-off speed, as can be seen in Fig.
The IAIA-BL loss can also be viewed as a (lower bound of the) separation loss.
Indeed:
[Equation]
where m q ≥ 0 is the “part-level” attribution mask for part q , obtained by down-scaling the inputlevel attribution mask of all pixels in the receptive ﬁeld of q .
This formulation highlights similarities and differences with our forgetting loss: the latter imposes similar constraints as the IAIA-BL penalty, but ( i ) it does not require full mask supervision, and ( ii ) it can easily accommodate conceptlevel feedback that targets entire sets of examples, as explained above.
B A CTIVATION P RECISION
In order to measure the quality of the explanations produced by the various models, we adapted the activation precision (AP) metric from the original IAIA-BL experiments (Barnett et al., 2021, Eq.
Assuming that the attribution mask attr output by ProtoPNets is ﬂattened to a vector with the same shape as m , the original formulation of AP can be written as:
[Equation]
Here, T τ is a threshold function that takes an attribution map (i.e., a real-valued matrix), computes the (100 − τ ) percentile of the attribution values, and sets the elements above this threshold to 1 and the rest to 0 .
Intuitively, the AP counts how many pixels predicted as relevant by the network are actually relevant according to the ground-truth attribution mask m .
This deﬁnition however ignores differences in activation across the most relevant pixels, which can be quite substantial, penalizing debugging strategies that manage to correctly shift most of the network’s explanations over the truly relevant regions.
To ﬁx this, we modiﬁed T τ not to discard this information, as follows:
[Equation]
In our experiments, we set τ = 5% .
The same value is used by IAIA-BL and ProtoPNets for visualizing the model’s explanations and is therefore what the user would see during interaction.
C I MPLEMENTATION DETAILS
Architectures.
We use the same ProtoPNet architecture for all competitors:
• Embedding step : For the CUB experiments, the embedding layers were taken from a pre-trained VGG-16, as in (Chen et al., 2019), and for COVID from a VGG-19 (Simonyan & Zisserman, 2014).
• Adaptation layers : As in (Chen et al., 2019), two 1 × 1 convolutional layers follow the embedding layers, and use ReLu and sigmoid activation functions respectively.
The output dimension is 7 × 7 .
• Part-prototype stage : All models allocate exactly two part-prototypes per class, as allocating more did not bring any beneﬁts.
The prototypes are tensors of shape 1 × 1 × 128 .
To stabilize the clustering and separation losses, we implemented two “robust” variants proposed by
Barnett et al.
(2021), which are obtained by modifying the regular losses in Eqs.
5 and 6 of the main text to consider the average distance to the closest 3 part-prototypes, rather than the distance to the closest one only.
These two changes improved the performance of all competitors.
Training.
We froze the embedding layer so as to drive the model toward updating the partprototypes.
Similarly to the ProtoPNet implementation, the learning rate of the prototype layer was scaled by 0 .
15 every 4 epochs.
The training batch size is set to 20 for the experiments on CUB5 box and COVID data sets, and 128 on CUB5 nat .
We noticed that, in confounded tasks, projecting the part-prototypes onto the nearest latent training patch tends the prototypes closer to confounders, encouraging the model to rely on it.
This strongly biases the results against the ProtoPNet baseline.
To avoid this problem, in our experiments we disabled the projection step.
Hyper-parameters.
The width of the activation function (Eq. 1 in the main paper) used in (cid:96) for was set to (cid:15)
= 10 − 8 in all experiments.
For CUB5 box and CUB5 nat , the weights of the different losses were set to λ cls = 0 .
5 , λ sep
=
08 , λ for = 100 .
In this experiment, the remembering loss was not necessary and therefore it was disabled by setting λ rem = 0 .
The value of λ iaia = 0 .
001 was selected from { 0 . 001 , 0 . 01 , 1 . 0 , 100 } so as to optimize the F 1 test set performance, averaged over three runs.
The number of retrieved training examples shown to the user is 5 ( a = 5 ).
For the COVID data set, λ cls , λ sep , λ for and λ rem were set to 0 .
08 , 200 , and 0 respectively for the ﬁrst two debugging rounds.
In the last round, λ cls , λ sep were decreased to 25% of the above values and λ rem increased to 0 .
01 , to boost the effect of (cid:96) for and (cid:96) rem .
This shifts the focus of training from avoiding very bad part-prototypes, to adjusting the partly wrong prototypes and increasing the activation on the relevant patches.
The value of a is set to 10.
D D ATA STATISTICS AND PREPARATION
Table 1 reports statistics for all data sets used in our experiments, including the number of training and test examples, training examples used for the visualization step, and classes.
The visualization data set contains the original images, i.e., before the augmentation step, and is used to visualize the images on which the prototypes activate most during the debugging loop.
All images were resized to 224 × 224 × 3 , as done in ProtoPNet (Chen et al., 2019), IAIA-BL (Barnett et al., 2021) and for the COVID-19 data in (DeGrave et al., 2021).
[Table]
D. CUB DATA SETS
Classes in the CUB200 data set include only 30 training examples each.
To improve the performance of ProtoPNets, as done in (Chen et al., 2019), we augmented the training images using random rotations, skewing, and shearing, resulting in 900 augmented examples per class.
CUB5 box .
The ﬁve CUB200 classes that were make up the CUB5 box data set are listed Fig.(left).
Synthetic confounders (i.e., colored boxes) have been added to training images of the ﬁrst, second and last class.
No confounder was added to the other two classes.
Fig.
6 (left) shows one of the three colored boxes.
An example image confounded with a green box “ (cid:4) ” is shown in Fig.(right) along with the corresponding ground-truth attribution mask m used for IAIA-BL.
[Figure]
CUB5 nat .
We created a variation of CUB200, namely CUB5 nat , to introduce a confounder that occurs naturally, e.g., sky or trees patches in the background, and are learned during the training phase.
We followed these steps to select the twenty classes reported in Table 2 and used for the experiments:
We modiﬁed the test set images of all the 200 classes of CUB200 by placing the bird of one class on the background of another one.
Since the outline of the bird removed from the background may not match the pasted one, the uncovered pixels are set to random values.
Figure Fig.
8 shows the background swap for one image and the corresponding groundtruth mask.
This background shift causes a drop in the performance of the model that relies on the background to classify a bird.
We trained ProtoPNet on two variations of the training set : in the ﬁrst, by removing the background, the images contain only the birds, and in the second, both the bird and the background.
We selected the twenty classes with the maximum gap in terms of F 1 on the modiﬁed test set, out of the 200 classes of CUB200.
Thus, the background appears to be a confounder for these twenty classes.
The right most images in Fig.
6 highlight patches of the background learned by the model.
We repeated the step above but only on the twenty classes, and we selected the ﬁve classes with the maximum gap.
In the experiments, we provided supervision only on these ﬁve classes.
Since the experiments are run on this subset of twenty classes, the separation loss might force the model to learn different prototypes with respect to the ones learned when trained on 200 classes.
Hence, the prototypes activate on different pixels and the classes with the largest gap are different.
This additional ranking of the classes allows us to debug the most problematic ones and to show the potential performance improvement on them.
For this reason, the results reported in the main paper are computed on only these ﬁve classes. [Table]
D. COVID DATA SET
The training set consists of a subset of images retrieved from the GitHub-COVID repository (Cohen et al., 2020) and from the ChestX-ray14 repository (Wang et al., 2017).
The number of COVIDnegative and COVID-positive radiographs is 7390 and 250, respectively.
The test set is the union of
PadChest (Bustos et al., 2020) and BIMCV-COVID19+ (Vay ´ a et al., 2020) datasets, totallingnegative and 597 positive images.
In (DeGrave et al., 2021), the classiﬁer is trained on 15 classes, i.e., COVID and other 14 pathologies.
In (DeGrave et al., 2021), the classiﬁer is trained on a multilabel data set, in which each scan is associated with multiple pathologies, and the evaluation is then performed on COVID positive vs. other pathologies.
To simplify the identiﬁcation and thus the supervision given to the machine, we generated a binary classiﬁcation problem containing COVIDpositive images and images without any pathology.
In this setting, supervision is given only on obvious confounders like areas outside the body proﬁle or on arms and elbows.
A domain expert can provide detailed supervision on which parts of the lungs the machine must look at.
E A DDITIONAL RESULTS
Additional part-prototypes for COVID.
Fig.
9 reports all prototypes for the experiment on the
COVID data set.
Confusion matrices.
The confusion matrix on the left side of Fig.
10 shows that ProtoPNet overpredicts the COVID-positive class, whereas the prediction of ProtoPDebug after three rounds is more balanced since the confounders have been debugged.
A domain expert can give additional supervision on which parts of the lungs are relevant for the classiﬁcation task, further improving the performance.
F C OMPARISON TO R EMOVING - AND -F INETUNING
One alternative strategy for debugging ProtoPNets consists of removing confounded part-prototypes (or ﬁxing their weight to zero) and then ﬁne-tuning the aggregation layer.
Here, we highlight some
[Figure]
[Figure]
0 2 4 6 8 1014 0.7 0.8 0.9 1.0 train f        14 0.25 0.30 0.35 0.40 0.45 0.
test f1 0 2 4 6 8 1014 0.4 0.
0.8 1.0 train f1_per_class 0       14 0.0 0.1 0.2 0.3 0.4 0.5 0.
test f1_per_class 0 1 0 2 4 6 8 1014 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 train cross_ent       14 0.04 0.02 0.00 0.02 0.
test cross_ent 0 2 4 6 8 1014 0.04 0.02 0.00 0.02 0.04 train forgetting_loss       14 0.04 0.02 0.00 0.02 0.
test forgetting_loss 0 2 4 6 8 1014 0.04 0.02 0.00 0.02 0.
[Table]
remembering
[Table]
loss
[Table] [Table]
train cluster
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
0 train rc
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
980 train pr
[Table] [Table]
[Table] [Table]
[Table]
000 train accu
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
00 train weights
[Table]
10 1 10 2 train
[Table]
total_ [Table] [Table] [Table]
total_add_on_layer_grad layer
[Table]
0.0030 train learning_rate_by_epochs
[Figure]
test learning by 0 2 4 6 8 10 106×102×10 2 train proto_norm
[Figure]
0 1 Predicted label
T r u e l a b e l 73791train conf_matrix
[Figure]
[Figure]
[Figure]
train f  1 0.35 0.40 0.45 0.
test f11 0.88 0.90 0.92 0.94 0.96 0.98 1.
train f1_per_class 0 1 0.2 0.3 0.4 0.5 0.
test f1_per_class 0 11 0.06 0.08 0.10 0.12 0.14 0.16 0.
train cross_ent 1 0.04 0.02 0.00 0.02 0.
test cross_ent1 3.4 3.5 3.6 3.7 3.8 [-3.
-]3.9 4.
[Table]
[Table]
[Table]
total proto_
[Table]
[Table] [Figure]
[Table]
[Figure]
test learning_rate_by_epochs
[Figure]
T r u e l a b e l
[Figure]
limitations of this strategy that ProtoPDebug does not suffer from.
In all experiments we use the same setup as in the experiments reported in the main text.
First, this strategy cannot be used when all prototypes are confounded.
This can occur because, in
ProtoPNets, there is no term encouraging the network to learn distinct part-prototypes. In practice, it can happen that all part-prototypes learn to recognize the same confound. This is illustrated by the two left images in Fig. 11: here a ProtoPNet trained on the CUB5 box data set has learned the same confounder (the colored box) with both of the part-prototypes allocated to the target class.
In this situation, the above strategy cannot be applied as there are no unconfounded concepts to rely on for prediction.
What happens if only some part-prototypes are confounded?
In order to answer this question, we assume that the ProtoPNet loss is augmented with a diversiﬁcation term (cid:96) div of the form:
[Equation]
(16) where λ div is a hyper-parameter.
This term encourages the different part-prototypes of each class y to be different from each other, and is useful to prevent them to all acquire the same confound.
Now, even if only some part-prototypes are confounded, there is no guarantee that the unconfounded ones capture meaningful information.
The opposite tends to be true: when a strong confounder is present, the network can (and typically does) achieve high accuracy on the training set by learning to recognize the confounder and allocating all weight to it, and has little incentive to learn other discriminative, high-quality part-prototypes.
We validated this empirically by training a ProtoPNet model on the CUB5 box data set using the diversiﬁcation term above, setting λ div to 0 .
The results are shown in Fig.
11 (right): the two activation maps show that one part-prototype activates on the confound, while the other captures irrelevant information and that therefore it would perform very poorly when used for classiﬁcation.
G A BLATING THE REMEMBERING LOSS
We evaluate the contribution of the remembering loss in last debugging rounds of the COVID experiment.
The remembering loss avoids performance drop over time.
We re-run the last two debugging rounds of the experiment in Figure 5 by setting λ r to zero.
In terms of F 1 , the performance in the third round goes from 0.55 with the remembering loss to 0.44 without.
In the fourth round, performance decrease from 0.54 to 0.4.
H U SER EXPERIMENT
We evaluated the usability of ProtoPDebug in a sequence of debugging round with real users.
The participants were asked to review patches of prototype activation on the training images.
The patch extraction procedure starts by selecting the ﬁve most activated images for each prototype.
For each activation map, we generate a binary image in which pixels are set to one if their activation value is [Table] at least 95%-percentile of all activation (as in (Chen et al., 2019)), zero otherwise.
Then, we extract the contours representing the border of the activation patches.
We discard the patches that are too small, difﬁcult to be spotted by the participants and not capturing enough information.
We set the threshold to patches having less than 200 pixels area.
Finally, we overlay the original images with the extracted patches.
We ran the online experiment through Proliﬁc Academic.
For each debugging round experiment, ten participants, laypeople and English native speakers, inspected the images generated as described above in a sequence of questions like Fig.
Each experiment took between 9 and 15 minutes according to the number of patches to review.
Participants received an hourly fee for their participation.
On top of this amount, we gave an additional bonus based on the agreement with the other participants.
We randomly selected ﬁve images, and we awarded the users whose evaluations agree with at least 80% of others’ evaluations for all ﬁve images.
We provided the following instruction to the user:
You will now be presented with a series of images of birds in various natural settings.
Each image has a highlighted overlay that goes from dark red (the highlight focus) to light green (the highlight margin).
Your task will be to carefully inspect each image and then to determine whether the highlighted overlay covers some part of the bird or whether, instead, it covers exclusively (or very nearly so) the background.
Table 3 exposes statistics about user feedback over the debugging rounds.
The number of patches on which the prototypes activate almost exclusively on the backgrounds decreases by giving more supervision to the model.
The inter-annotation agreement is 96% and shows that the ProtoPDebug feedback required from the supervisor is simple and effective.