Paper: Improving Prototypical Part Networks with Reward Reweighing, Reselection, and Retraining (ICLR-2024)

Reviewer comment (Evaluation):
There is no evaluation of the learned reward function.

Author response:
$\textbf{No evaluation of the learned reward function:}$ 
We mentioned in 5.2 that our reward model achieves 90.09\% test accuracy on the synthetic dataset (the held-out test set has about 13k pairs of heatmaps), which means it could effectively predict the relative order of prototype quality and thus model the collected human preferences reliably.

Reviewer comment (Theory):
While the proposed R3-ProtoPNet does empirically improve the prediction accuracy over the original ProtoPNet, there is little theoretical insight as to what makes R3-ProtoPNet work.

Author response:
$\textbf{Why our method empirically improves the performance:}$
In the original ProtoPNet, the prototypes are generated by the model itself only based on supervised predictive signal, and the interpretable reasoning chain is maintained by the "push" operation, which sets the prototypes to be the closest input image patch. However, one issue with this process is that once initialized the prototypes cannot change dramatically, so the bad-initialized (possibly due to some predictive shortcuts) ones will always remain bad - we consider this as the main cause of the spurious features in the ProtoPNet. In our R3-ProtoPNet, with the help of an external reward model, reselection allows the global movement/reinitialization of the prototypes, while reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. more overlap, more characteristic features, etc.).

Reviewer comment (Experiments):
Is it necessary to have both reward reweighing and prototype reselection? More specifically, while prototype reselection is more necessary (it is a way to move away from badly learned/low-reward prototypes), reward reweighing seems optional to me. An ablation study on this will be helpful.

Author response:
$\textbf{Reward reweighing seems optional:}$
As stated above, reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. moving an originally partially overlapped prototype more toward the body), and we did find both interpretability and accruacy gains as the result of reward reweighing. Please refer to 1) our updated performance report tables which will break the R2 step into reselection and reward reweighing as well as 2) an ablation study added in the appendix. Thanks for the suggestion!

Reviewer comment (Presentation):
There are limited visualizations. While the authors did include visualizations of prototypes (in the appendix), they did not include enough examples of how prototypes are used in R3-ProtoPNet, the closest prototypes to a given image from a trained R3-ProtoPNet, and the closest image patches to a given prototype learned by an R3-ProtoPNet. These are needed to convince readers that an R3-ProtoPNet uses high-quality prototypes in reasoning, and the learned prototypes are semantically meaningful.

Author response:
$\textbf{More illustrative visualizations:}$
Actually We included the closest image patches to each prototype and the prototypes corresponding to each image during different stages in the appendix. That being said, we will consider adding more illustrative visualizations in our modified paper and also revising the captions, and thanks for the feedback.

Reviewer comment (Theory):
An R3-ProtoPNet can be thought of as initializing a ProtoPNet in a smart way (by reward reweighing and prototype reselection), and then training it again. As mentioned earlier, is there any explanation for why this empirically improves the prediction performance?

Author response:
$\textbf{Why our method empirically improves the performance:}$
In the original ProtoPNet, the prototypes are generated by the model itself only based on supervised predictive signal, and the interpretable reasoning chain is maintained by the "push" operation, which sets the prototypes to be the closest input image patch. However, one issue with this process is that once initialized the prototypes cannot change dramatically, so the bad-initialized (possibly due to some predictive shortcuts) ones will always remain bad - we consider this as the main cause of the spurious features in the ProtoPNet. In our R3-ProtoPNet, with the help of an external reward model, reselection allows the global movement/reinitialization of the prototypes, while reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. more overlap, more characteristic features, etc.).

Reviewer comment (Experiments):
As mentioned before, is it necessary to have both reward reweighing and prototype reselection? Please include an ablation study on this.

Author response:
$\textbf{Reward reweighing seems optional:}$
As stated above, reweighing allows local movement of the prototypes to be more aligned with human preferences (e.g. moving an originally partially overlapped prototype more toward the body), and we did find both interpretability and accruacy gains as the result of reward reweighing. Please refer to 1) our updated performance report tables which will break the R2 step into reselection and reward reweighing as well as 2) an ablation study added in the appendix. Thanks for the suggestion!

Reviewer comment (Experiments):
[Q1] One issue with the experiments is that they consider a single data set (CUB-200). Bontempelli et al. (cited by the authors) do evaluate their approach on three data sets (CUB-200, a synthetic data set, and an X-ray data set). The choice of focusing on CUB-200 only is not exactly justified.

Author response:
$\textbf{Only one single bird dataset is used: }$  
Thanks for pointing this out! We chose the CUB bird dataset because it's the most commonly used dataset in related works and it could elicit very natural human preferences. We are currently working on evaluating our R3 framework on other datasets such as the Stanford Cars dataset. We expect similar performance gain, but the full results on all base architectures might not come out by the end of the discussion period. We will definitely include those results later.

Reviewer comment (Experiments):
[Q2] It is also not clear why R3-PPNet was not compared to the work of Bontempelli et al. -- it should be trivial to convert ratings into binary labels (say, ratings below 3 could be converted to a "bad" label, and 4 and above to "good").

Author response:
$\textbf{Why R3-PPNet was not compared to the work of Bontempelli et al.: }$  
One major distinction we'd like to point out is that R3-ProtoPNet is an offline debugging framework: it only requires one batch of human feedback to train the reward model, and this happens before we actually debug the ProtoPNet. On the other hand, the ProtoDebug framework proposed by Bontempelli et al. requires iterative online human feedback on the exact models and prototypes to be updated. Furthermore, to the best of our knowledge, ProtoDebug is not designed for large-scale prototype debugging: the required human labor would scale up with the number of prototypes (their collected feedback and evaluation was on 5 out of 200 classes while we evaluated on all 200 classes); by contrast our method only needs a fixed number of human annotations to train the reward model. Given the reasons above, we don't think the two frameworks are quite comparable.

Reviewer comment (Theory):
[Q4] It seems to me R3-PPNet is designed for passive learning only. The reason is that, if I understand correctly, the reward function (which depends on the learned model) becomes obsolete after fine-tuning/debugging the model. If the debugged model is still buggy, the old reward function cannot be used again. Is this correct?

Author response:
$\textbf{The reward function becomes obsolete after fine-tuning/debugging the model: }$  
Yes, this is largely correct. And if some prototypes are still buggy after the R3 update, it's most likely that it's not detected/properly corrected by the learned reward model, and such cases do happen due to the noise in the entire human-in-the-loop module. Conventional ways to robustify the reward model include comprehensive selection of the heatmaps (so that most typical errors are included) to be rated and increasing the number of raters (which we did). One interesting future direction would be having a ProtoPNet updated by multiple learned reward models sequentially, each capturing one single aspect of human preference (e.g. overlap, body part consistency, etc.), so that each reward model would be less noisy because of the single learning objective.

Reviewer comment (Evaluation):
While it's great that your scope is clear, it ends up feeling a bit like a lab report, where your assignment was to apply RLHF to ProtoPNet and report the results. This isn't bad per-se; science for the sake of doing it can be great, but ideally for a scholarly paper I would want to see some more interpretation, insight, and high-level thinking.

Author response:
$\textbf{The paper has limited scope:}$
While we agree that the paper is focused on applying RLHF to ProtoPNet, we respectfully disagree with the reviewer that the paper has limited scope. Many interpretability or explainability methods in deep learning are designed for a particular architecture, data modality, or problem in mind. Part of the difficulty of interpretable deep learning is designing or utilizing interpretable methods for particular problems. This process usually involves intense and time-consuming exploration of the data and understanding of the problem. Combining RLHF with existing interpretability methods, while certainly not eliminating the need for a deep understanding of the problem at hand, has the potential to simplify the modeling process, allowing researchers and practitioners to spend less time designing and building complex methods, and more time interrogating the reasoning and limits of their models. We believe that R3-ProtoPNet is one such example of how RLHF and adjacent ideas can allow for improvements to interpretable deep learning more generally.

An example of this is that ProtoPNet fails to provide consistent prototype labels across images, where a single prototype can still map to multiple parts of a bird across images. Instead of designing a hand-crafted regularization function to promote cross-image consistency, we can now instead consider how to best elicit and model human feedback on cross-image consistency of prototypes, which is something human viewers can do with ease. 

We will revise our paper to include a discussion of the general contribution to RLHF and interpretability.

Reviewer comment (Evaluation):
It would be ideal to have an evaluation done by expert human evaluators (whatever that group might be depending on the intended goal of the paper, if any) -- ie. most likely, ornithologists or birders. Given that the focus is on interpretability, how useful are the prototypes actually, to humans who might use them? Do they find the same characteristic features people use for ID? Any novel and interesting ones?

Author response:
$\textbf{Human expert evaluation needed:}$
The CUB-200-2011 dataset is a commonly used benchmark for evaluating ProtoPNets due to the ease with which an amateur can recognize how a method makes classifies a bird, such as focusing on the background or the bird itself, on the bird's wings or head--or some uninterpretable combination of the two. We agree with the reviewer that the utility of such a bird classification method to an actual expert is unclear, given that no expert evaluation is used to justify this method. We hope to provide such an exploration in future work.

Reviewer comment (Experiments):
The "parts" highlighted by the original or your modified version tend to be very soft-edged and blobby. In some sense I would "like" to see prototypes that crisply highlight e.g. tails, feet, beaks, etc. Do you think this is desirable and/or possible? Why or why not? For example, the pixel-level segmentation maps used in AP seem like they could provide very good supervisory signal for this.

Author response:
$\textbf{Prototypes are sometimes soft-edged and blobby:}$
In ProtoPNet, a heatmap is generated by calculating the similarity of the upsampled prototype and each of the 49 patches (each of size 32x32) of an original image, so if the textures of several neighboring patches are similar, then the highly activated region would inevitably be larger/blobby. In fact we would consider this as more of a visualization problem. As we can see in the visualized examples in the appendix, many prototypes are able to clearly focus on wings, tails, etc. We hypothesize that some prototypes don't focus on specific body parts because sometimes more subtle visual information such as characteristic edges and texture may provide better predictive features.

Reviewer comment (Theory):
There could be a more nuanced discussion of what makes a feature spurious or not. For example, it's repeatedly stated that the background is spurious, but habitat (especially if it includes something as characteristic as a nest) can often be incredibly useful for identification. Even the presence of sky vs. grass or sea or something could be informative (e.g. woodland thrushes would virtually never be photographed against open sky, unlike a swallow). At a more general level, the shape/contour of tailfeathers or the bird's overall silhouette (i.e. the edge pixels which would include some background) are also often characteristic.

Author response:
$\textbf{A more nuanced discussion of what makes a feature spurious or not:}$
We thank the reviewer for pointing this out. Placing attention on the background is commonly considered 'spurious' information in interpretable deep learning methods for computer vision tasks, with the original ProtoPNet paper doing so in Section 2.6 (Chen et al. 2019). Our understanding is that the reason for this trend is a desire for image classification systems to make decisions not on correlative features but causal ones. While the background of the image being a blue sky raises the probability of the bird in the image being a swallow and not a woodland thrush, the presence of the blue sky does not make the swallow a swallow. We believe this focus on wanting to learn causal and not correlative features comes from the want for interpretable methods like ProtoPNet to generalize across image classification tasks.

Reviewer comment (Novelty):
The main stated high-level goal of the work is to improve interpretability — for whom, and for what purpose? If everything worked "perfectly", what would this system be able to do or be used for? Or if it doesn't have a goal in mind (which again, is totally fine for science), what do we understand better about RLHF or Prototype nets or deep nets in general as a result of this work?

Author response:
$\textbf{The application of improving interpretability and high-level understanding about RLHF, ProtoPNets, or general neural networks: }$
As we have discussed, the use of the CUB-200-2011 dataset is largely a demonstration of how one could use ProtoPNet and R3-ProtoPNet to perform image classification according to a particular reasoning framework. A limitation of this work is that we only propose a hypothetical application, but do not provide the reader with a particular application where we work with domain experts to solve a particular problem. The questions of who this is specifically for, and what specific purpose is this for, are left unanswered by our work. We hope to remedy this limitation in future work. 

That said, as discussed earlier, we believe our method has general applicability to interpretable deep learning. As we demonstrate, RLHF or related ideas have the potential to simplify the process for designing and training interpretable deep methods, with possible follow-ups extending beyond the architecture of ProtoPNet.

We again thank the reviewer for these helpful comments!