From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper considers a new problem setting for deep anomaly detection in the image domain: how to ignore false-positives from a real-world perspective (i.e., anomalies due to variations in nuisance attributes). Each image is labeled with the corresponding nuisance attributes. It proposes a principled method to solve this problem which is inspired by prior work on domain-supervised disentanglement. The paper demonstrates good experimental results compared to baseline methods.

Overall, the paper is well-written and easy to follow. The author response has addressed most of the critical comments from reviewers and the revised paper is much improved.

Two of the reviewers' who gave low scores  and didn't participate in the discussion after submitting their reviews. I will list their critical review comments here:
1. Nuisance labels 
2. Applicability to other data types
3. Criticism based on the understanding that anomalies are scored based on reconstruction.

Based on my reading of the paper, author response to review comments, and letter to AC, I'm positive about the paper. The paper also does a great job in explicitly discussing the limitations of the proposed approach. The author response addresses the above comments as follows: 
- #1 is addressed in Section 1 and 4.1. 
- #3 is a factual error -- method uses nearest neighbor anomaly scoring
- #2 is valid, but I don't think it is a show-stopper. I suggest the authors to add a discussion about applicability to different data types. If no experiments beyond image domain are provided, I recommend scoping the paper and title to image domain.

Overall, I recommend accepting the paper. The main reason behind my decision is that the paper will likely inspire the community to study this new anomaly detection problem setting and come up with better solutions -- the paper proposed to make the resources available to further catalyze research on this problem.
justification_for_why_not_higher_score: Lack of technical novelty.
justification_for_why_not_lower_score: New problem setting which will likely inspire the community to study this problem.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Thank you for responding to our rebuttal!
comment: Dear Reviewer W97W,

Thank you for your positive opinion of the technical content and for leaning towards acceptance.

We agree that the claims can be made more precise. OpenReview does not allow us to upload a revision of the paper at this stage. However, in the final version we promise to clearly state that:

i) While making progress on achieving more invariant representations than previous methods, our representations are still not fully invariant and there is still much work to be done.

ii) The datasets that we use are simple and operating on real-world problems is a challenge for future work.

We will extend the introduction and limitations sections (Sec.1,7) with the points raised in the review and our rebuttal.

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_W97W
title: Happy to discuss further!
comment: I agree that the paper has its limitations, but I think it has enough value that it doesn't merit a "strong reject". However, I'm not deep into this particular research area, so I'd welcome more insight into your evaluation.

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_W97W
title: Thank you for the responses!
comment: Thank you for the responses. At the current stage I think the technical content of the paper is fine and I still lean towards accepting the paper. 

My primary remaining concerns relate to presentation. As discussed above, I think some of the claims in the paper need to be toned down to match the results. For instance:
* Saying that a method "achieves invariance" seems like an overstatement for all of these methods (even if prior papers do it), and would likely mislead a reader unfamiliar with this literature. 
* Saying this is useful for "real-world" problems seems like an overstatement if all the testing is done on toy datasets (even if prior papers do it). It's fine if this is a first step that is not yet practical, but we should make sure our readers don't get the wrong impression. In addition, it would be helpful to the "real-world" reader to clarify the limitations discussed above (e.g. categorical nuisance factors, misspecified nuisance factors, missing nuisance factors) in the text.

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: General Comment:
comment: We thank all the reviewers for their time and effort. We are glad that the reviewers found our solution elegant and creative *(9jBA)*, the idea novel, the experimental analysis is interesting *(LAMp)*, and our paper well written and thorough *(W97W)*.

Please find specific answers below and correction in the revised manuscript.

Note that we slightly exceed the page limit in the revised manuscript, but this is due to struck-through text that will not be included in the final version.

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Response to Reviewer W97W (Part 2): 
comment: ***“The paper assumes that nuisance factors are categorical. Is this a significant limitation or is there an easy way around it?”*** 
Continuous attributes can be encoded by quantization or fourier encoding  [4]. Actually, two of our experiments (Cars3D and SmallNorb) already use quantized continuous attributes (azimuth angles which define the pose of a vehicle).

***“what if the user is wrong about a feature being a nuisance factor?”*** 
We aim to ignore the user-specified nuisance attributes. If the user is wrong about an attribute being nuisance, i.e. this attribute is actually relevant for detecting anomalies, our algorithm will not be able to identify these anomalies. In many cases a nuisance attribute is easy to identify (pose, ethnicity, etc.).

***“ If the user does not enumerate every possible nuisance factor, do they pay a price?”*** 
Generally, unidentified nuisance attributes could indeed hinder the anomaly detection capabilities. Our aim is to remove identified nuisance attributes from the dataset, which is a non-trivial task in images by itself. The disentanglement approach we use still works in the presence of some nuisance variation, but very severe unexpected domain shifts might be more challenging.



[1] Kahana, Jonathan, and Yedid Hoshen. "A Contrastive Objective for Learning Disentangled Representations." in ECCV, 2022.

[2] Aviv Gabbay and Yedid Hoshen. “Demystifying inter-class disentanglement”. In ICLR, 2020.

[3] Locatello, Francesco, et al. "Disentangling factors of variation using few labels." arXiv preprint arXiv:1905.01258 (2019).

[4] Li, Yang, et al. "Learnable fourier features for multi-dimensional spatial positional encoding." Advances in Neural Information Processing Systems 34 (2021): 15816-15829.

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Response to Reviewer W97W (Part 1): 
comment: ***“The proposed method is within the margin of error of a competitor for AD-Score”***
Our main competitor on the SmallNorb dataset (Tab. 2) significantly underperforms on Cars3D (Tab.1) and even more significantly on “Edges2Shoes” (Tab.3).

***“No error bounds”***
We thank the reviewer for the suggestion. We provide error bounds for all datasets (also added to the revised Appendix):
|     | SmallNorb |          |          | Cars3D   |          |          | Edges2Shoes |          |          |
|-----|-----------|----------|----------|----------|----------|----------|-------------|----------|----------|
|     | AD Score  | PA Score | RA Score | AD Score | PA Score | RA Score | AD Score    | PA Score | RA Score |
| Avg |     0.952 |    0.554 |    0.947 |    0.983 |    0.509 |    0.975 |       0.793 |    0.687 |    0.745 |
| Std |     0.008 |    0.016 |    0.008 |    0.002 |    0.011 |    0.001 |       0.014 |    0.019 |    0.022 |

***“ worse than a competitor for PA-Score (this is true for 2/3 datasets)”***
All the methods that outperformed us in terms of the PA score provide no information and are agnostic to pseudo-anomalies and true-anomalies alike (significantly worse AD-Score). The PA score alone can simply be optimized by a random anomaly score - always getting ROCAUC of 0.5. To measure the ability of a method to distinguish true anomalies from pseudo anomalies directly we proposed the RA score, where we significantly outperform all baselines. (Also clarified in the revised version, page 9)

***“There is no ablation study for the components of the proposed method.”***
We thank the reviewer for noting this. First, SimCLR is effectively an ablation of our method. SimCLR trains a single contrastive loss, similar to Eq.4, but when treating the entire training set as coming from the same single label (ii) Second, No Rec presents our method without using the reconstruction loss (Eq.5). Similarly to [1], it works well in some cases, but is not stable across datasets. Especially, in dataset such as Edges2Shoes where invariance is very hard to achieve because the domain shifts are dominant. We provide the additional ablation here (and in the revised Appendix):

|        | SmallNorb |          |          | Cars3D   |          |          | Edges2Shoes |          |          |
|--------|-----------|----------|----------|----------|----------|----------|-------------|----------|----------|
|        | AD Score  | PA Score | RA Score | AD Score | PA Score | RA Score | AD Score    | PA Score | RA Score |
| SimCLR | 0.805     | 0.728    | 0.638    | 0.780    | 0.519    | 0.741    | 0.567       | 0.642    | 0.510    |
| No Rec |     0.779 |    0.516 |    0.783 |    0.977 |    0.511 |    0.975 |        0.55 |    0.706 |    0.466 |
| Ours   |     0.953 |    0.581 |    0.943 |    0.985 |    0.506 |    0.980 |       0.781 |    0.711 |    0.719 |

***“I was surprised by how ineffective all methods seem to be at achieving invariance”***
Datasets such as Edges2Shoes have strong correlation between the style (e.g. texture) and the domain (i.e. edges don’t have color or textured surfaces). Learning a representation which is both informative but does not contain domain information is non-trivial. 

***"These seem to be toy datasets. If NAGAD is meant to be a well-motivated practical problem, shouldn't there be more challenging real-world datasets where this problem shows up?"***
As we mentioned in the response above, disentanglement is non-trivial even for fairly simple cases. We therefore presented results for relatively controlled datasets, and showed that our method can be successful there. We agree that providing results on in-the-wild datasets would be helpful for future research, and for improving representation disentanglement methods, in cases where current methods still obtain preliminary results. We will provide such a benchmark in the final version. 

***“The paper asserts that the (more realistic) case of multiple attributes can be handled ‘very naturally’”***
Our statement referred to bottleneck methods such as LORD[2], Locatello et al. [3] which provide disentanglement capabilities based on representations bottlenecks. Codes describing different factors could be easily concatenated to describe multiple attributes. For our contrastive disentanglement approach, this indeed is more involved. Yet, as some SoTA disentanglement methods can already deal with multi-attribute settings we do not perceive it as a major limitation for our setting. This is clarified in the revised manuscript (page 9).

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Response to Reviewer 1pDc:
comment: ***"“Some of the claims in the paper are overreaching: “Taken together in many cases labels for nuisance attributes they can be achieved at virtually no cost.”
 [Section 4.1] For the approach, the users have to provide additional labeled data. It is not clear if such labels are easily available apart from synthetic datasets.”"***
We suggested that zero-shot models, in particular CLIP, can be used for obtaining these labels automatically. CLIP is actually better on real than synthetic datasets. The labels in Sec.4.1 were obtained by simply prompting a pretrained CLIP-based classifier. We also discuss other labeling sources in Sec.1.

***“Anomaly detection applies to various types of datasets. These authors solve it in a specific setting. Does it apply other types of images or to all images or other types of data as well?”*** 
We demonstrate our approach on three different datasets. We expected a similar approach, with small modifications to the augmentations and perceptual loss, should work in other cases where disentanglement approaches are successful. This includes modalities such as audio signals [1] or text [2].

***“The technical solutions, given the assumptions, are widely known which limit the novelty of the paper.”*** 
While we rely on published technical solutions [3], they are far from being widely known, especially in the anomaly detection community:  (i) Using disentanglement approaches is non-standard in the field of anomaly detection. (ii) As far as we know, this is the first work to apply contrastive disentanglement for a downstream application.

[1] Abeßer, Jakob, and Meinard Müller. "Towards Audio Domain Adaptation for Acoustic Scene Classification using Disentanglement Learning." arXiv preprint arXiv:2110.13586 (2021).

[2] Cheng, Pengyu, et al. "Improving disentangled text representation learning with information-theoretic guidance." arXiv preprint arXiv:2006.00693 (2020).

[3] Kahana, Jonathan, and Yedid Hoshen. "A Contrastive Objective for Learning Disentangled Representations." arXiv preprint arXiv:2203.11284 (2022).

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Response to Reviewer LAMp:
comment: ***“if they believe that the method is applicable in other settings (straight tabular data), then they should include discussions about why this would be better than simply removing the attribute.***
It is true that in tabular data, one can often simply remove the nuisance feature from the anomaly detector by simply removing the column if the nuisance attribute is perfectly correlated with some column and completely uncorrelated with all others.  The goal of our method is to deal with cases where disentangling factors from the representation is more complicated. We expect our method to also apply, with modification to the augmentation and perceptual loss, to speech and audio. We clarified it in the revision (Title, abstract, page 2).

***“it is hard to know which approaches are they referring to. Also, I do not think traditional anomaly detection methods require knowledge of attributes to identify anomalies. They possibly require normally labeled data (and maybe some labeled anomalies).”***
Thank you for this question, this requires further clarification. We use standard scientific terms commonly used in the literature, but should have defined them. 

The standard anomaly detection setting (also called one-class classification) indeed requires only normal data. We mentioned other two settings that use more supervision: 

Out-of-distribution detection [1,2,3] -  each sample is labeled with the class - which we denote as the relevant attribute (e.g., object category). Anomalies are unseen values of the class,

Weakly supervised anomaly detection [4,5,6] - where limited or partial labels exist for both normal and anomalous data.
We added this to the revised manuscript (page 2).

***“The authors have not talked much about the real world scenarios where this problem formulation might be applicable.”***
While we provided an illustrative example in the manuscript (page 2), the setting itself is applicable in many other realistic anomaly detection scenarios. Current state-of-the-art methods detect anomalies using pretrained features, which typically focus on the main object in the image. But in many cases we would like to discover anomalies in other image attributes, e.g., activity, scene background, or any other non-object features. Our approach is applicable in all these cases.

***"The level of innovation in the method is limited, given that the method builds on past work on contrastive disentanglement."***
While we indeed rely on previous disentanglement methods we would like to stress that: (i) to the best of our knowledge, this is the first paper to suggest an application of the used contrastive disentanglement method. (ii) the use of disentanglement methods in anomaly detection is non-standard. We also believe that the work has merits in experimental analysis and as a novel setting.

All minor comments were addressed in the manuscript in pages 1,2.

[1] Salehi, Mohammadreza, et al. "A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges." arXiv preprint arXiv:2110.14051 (2021).

[2] Hendrycks, Dan, and Kevin Gimpel. "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136 (2016).

[3] Hendrycks, Dan, et al. "Pretrained transformers improve out-of-distribution robustness." arXiv preprint arXiv:2004.06100 (2020).

[4] Cozzolino, Davide, et al. "Forensictransfer: Weakly-supervised domain adaptation for forgery detection." arXiv preprint arXiv:1812.02510 (2018).

[5] Gianchandani, Urvi, Praveen Tirupattur, and Mubarak Shah. "Weakly-supervised spatiotemporal anomaly detection." University of Central Florida Center for Research in Computer Vision REU (2019).

[6] Deecke, Lucas, et al. "Transfer-based semantic anomaly detection." International Conference on Machine Learning. PMLR, 2021.

From: ICLR.cc/2023/Conference/Paper3029/Authors
title: Response to Reviewer 9jBA:
comment: ***“example of a nuisance feature in the motivation was not applicable to the approach”.***  
We revised the manuscript to contain an explicit example more similar to our experimental setting (Revised in page 1).

***“It would be nice to see with and without Red PANDA analysis of the role the nuisance feature's role in the reconstruction.”*** - 
We would like to stress that our anomaly scoring method is **not** a reconstruction based autoencoder method. Rather, we use reconstruction as one of the losses for the adaptation of a density estimation based method (similarly to [1]). This is now clarified in the manuscript (Revised in page 2,6).

***“no fault of the authors, is that autoencoders, although popular in the literature, have not been found to be worth the training effort in industry”***
We strongly agree with the claim that auto-encoders do not provide strong results on images in most settings. For this reason, both our method and our (state-of-the-art) baselines do not use auto-encoders for anomaly scoring. We therefore do not believe it is a weakness of our paper.

***“we probably have enough information to create a simple heuristic to deal with these edge cases.”***
Not necessarily. During the research towards this work we tried numerous approaches to deal with settings including: using CDS[2] and trying to remove attributes with linear methods (using CLIP[3]), but neither of these provided a solution. (other reviewers agree: “method is reasonable, based on a sound strategy proposed in another paper. “ LAMp). While a simple heuristic solution might be possible in some cases, it may be very problem specific. Simple heuristics aimed at ignoring specific attributes (e.g., color) may be tailored for specific domains, while our method aims at the general case of nuisance attributes. Moreover, most simple heuristics in anomaly detection use some knowledge on the attributes of the anomalies, while both our method and baselines do not. (Clarified in page 2 in the revised manuscript).

***“I would like to see a more nuanced example of a nuisance feature. A more real-world example would be features which affect the distribution of the data which can mask anomalies when the nuisance feature is in certain value ranges.”***
We thank the reviewer for the suggestion. As suggested, we performed an additional experiment where we select pseudo-anomalies from a range of values, instead of considering merely one viewing angle for each class as a pseudo-anomaly. Here, for a set of 24 random classes we randomly select a range of 36 degrees in azimuth, which we remove from the train set, and treat as pseudo anomalies. We report the results for the SmallNorb dataset:

| SmallNorb Ranges | AD-Score | PA-Score | RA-Score |
|------------------|----------|----------|----------|
| SimCLR           |    0.797 |    0.743 |    0.633 |
| DN2              |    0.908 |    0.819 |    0.768 |
| MeanShifted      |    0.949 |    0.872 |    0.812 |
| DCODR            |    0.949 |    0.557 |    0.942 |

***"The paper was well-written. However, it is not clear how the approach is strung together end-to-end from data set to addition of training signal in the form of identification of nuisance signal."***
We would like please to refer the reviewer to the algorithm box found in the Appendix (page 16 and referred to in page 5). We will move it to the main text. For convenience, we provide it here as well:

**Result**: An anomaly score $S_t$ for each test image.

**Input**: A classifier for nuisance attributes $C$, train data consisting of normal samples $D_{train}$, a pre-trained encoder $f$, and a random augmentation function $A$ (See App. 9.1.1). $sim$ is cosine similarity.

1. Preprocessing:
* We extract the label for the nuisance attribute for all training data $x_i \in D_{train}$ as $n_i = C(x_i) $.
2. Training:
* Until convergence:
  * $L_{con} = \log (\sum_{x_i, x_j }^{} {1}_{(n_i = n_j)}e^{sim((f(x_i), f(x_j))})$ 
    * $x_i, x_j \in D_{train}$ are arbitrary normal samples
  * $L_{rec} = \sum_{x_i }^{}\ell_{perceptual}( x_i, G (f(x_i), n_i))$
  *  $L_{aug} = \sum_{x_i }^{}-sim(f(A_1(x_i),A_2(x_i)))$
  *  $F, G = argmin_{F, G}(L_{rec} + L_{con} + L_{aug})$ # SGD
* Use the fixed encoder $f$ to compute representations for all training samples:
  * $T =$ {$ f(x) | x \in D_{train} $}
3. Anomaly scoring for test sample $x_t$:
  * $f_t = f(x_t)$ # Extract test sample representation
  * $ S(x_t) \leftarrow$ $k$NN_Distance($f_t$, $T$)

[1] Reiss, Tal, et al. "Panda: Adapting pretrained features for anomaly detection and segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[2] Kim, Donghyun, et al. "Cds: Cross-domain self-supervised pre-training." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_9jBA
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose a method for anomaly detection in the presence of nuisance features.  In their motivation they describe nuisance features as those which give the appearance of anomaly but are not anomalous.  These would be features which are seen in nominal situations as only having a narrow range of values but cannot be realistically expected to be constrained to that range outside of training.  The method requires identification of the nuisance feature and uses a modified loss function with an autoencoder-style network.  The modified loss function is designed to remove the nuisance feature from the reconstruction and to force the reconstructed images to have distributions unaffected by the nuisance feature.
strength_and_weaknesses: The proposed solution to the identified problem is elegant and creative.  Nuisance features are certainly a problem in real-world anomaly detection applications.

The example of a nuisance feature in the motivation was not applicable to the approach in that they describe a very high level feature but the example

The authors are explicit about the strengths and weaknesses of their approach and discuss how the identified limitations can be mitigated.

The results strongly point to the usefulness of this approach.  However, they are missing a next-level analysis of the results.  It would be nice to see with and without Red PANDA analysis of the role the nuisance feature's role in the reconstruction.

A couple weaknesses: 
* no fault of the authors, is that autoencoders, although popular in the literature, have not been found to be worth the training effort in industry
* if we have the information needed to provide the training set needed for this approach, then we probably have enough information to create a simple heuristic to deal with these edge cases.
* I would like to see a more nuanced example of a nuisance feature.  A more real-world example would be features which affect the distribution of the data which can mask anomalies when the nuisance feature is in certain value ranges.
clarity,_quality,_novelty_and_reproducibility: The paper was well-written.  However, it is not clear how the approach is strung together end-to-end from data set to addition of training signal in the form of identification of nuisance signal.

With the exception of the above, the experiment appears to be reproducible but I was unable to take a deeper look at the synthetic data set the authors used.
summary_of_the_review: A well-written paper featuring an interesting approach.  The approach aims to solve a significant issue with anomaly detection.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_LAMp
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper proposes a novel setting for anomaly detection where the visual data also has a set of attributes that describe the environmental parameters. The proposed method focuses on a setting where the user can define one or more nuisance attributes such that the algorithm should ignore the impact of nuisance attributes when identifying anomalies. This is done by first learning a representation of the data that does not consider the nuisance attribute using a contrastive disentanglement algorithm. The second step is to find anomalies by using a nearest neighbor strategy on the representation data. Results are shown on benchmark data sets and the method is shown to outperform other strategies for detecting anomalies which do not consider the nuisance attribute.
strength_and_weaknesses: Strengths:
+ The paper formulates a novel anomaly detection problem which might have potential applications when dealing with visual data.
+ The idea of using the knowledge of nuisance attributes and incorporating it in the representation phase is based on past idea, but using it for anomaly detection is novel.
+ The experimental analysis is interesting.

Weaknesses:
- The approach is meant for a particular type of data sets, for example, images, where the analysis is done on the observations (images) while the nuisance attributes are part of the environmental conditions in which the data was recorded. The nuisance attributes are not encoded in the data as independent features,  otherwise one could just remove them from the analysis. However, the title and abstract does not make that clear. I would request authors to clarify that. Or, if they believe that the method is applicable in other settings (straight tabular data), then they should include discussions about why this would be better than simply removing the attribute.

- The authors distinguish their methods from past work as this - "Our approach differs from previous approaches to this problem. Previous approaches require knowl- edge of an attribute that should be used to identify the anomalies. Out of distribution detection assumes that anomalies belong to a novel class, outside the fixed set of labelled classes. The weakly supervised setting assumes future anomalies will be similar to a few labelled anomalous samples available during training." There are no references here so it is hard to know which approaches are they referring to. Also, I do not think traditional anomaly detection methods require knowledge of attributes to identify anomalies. They possibly require normally labeled data (and maybe some labeled anomalies). Could the authors clarify on this?

- The authors have not talked much about the real world scenarios where this problem formulation might be applicable.

- The level of innovation in the method is limited, given that the method builds on past work on contrastive disentanglement. 

Some minor comments:
- What is an operator? It was unclear in the abstract if the authors are referring to a human operator or a mathematical operator.
- Is there a citation for this statement - "A major source for such biases is the presence of additional, nuisance factors."?
- I did not understand the example: "For example, assume all buses in the training data were captured in a certain pose since they were driving in a bus lane. It is likely that a bus driving lawfully in a regular lane will be viewed from a different angle than the buses in the normal data. Thus, it will be regarded as an anomaly by current standards." Are you saying that it is possible that a bus driving lawfully in a regular lane might be viewed from a different angle? I am not well-versed with this domain (and I am guessing many readers won't be as well), so I would request the authors to clarify this.
- I was confused by this example: "For example, an encoder guided to be invariant to driving lane, would be trained to contrast images of females against other images of females, but not against images of males (and vice versa)." Why are we suddenly talking about females and males in the driving context?
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The quality is high. There are some places where more details could be beneficial. 

The paper has enough details about the method, parameter choices, data, to reproduce the results. 
summary_of_the_review: The paper has an interesting problem formulation for anomaly detection that I have not encountered before. While there is academic novelty in that, the authors do not quite discuss the practical relevance of the problem setting. There method is reasonable, based on a sound strategy proposed in another paper. There are some issues with the descriptions that could be fixed by a careful rewrite.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_1pDc
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper addresses the problem of guiding anomaly detection techniques to ignoring distracting aspects. The distractions or nuisance factors are labeled and the disentanglement is done through contrastive loss. Results on Cars3D, SmallNORDB, and Edges2Shoes are shows that the approach outperforms SimCLR, CSI, DN2, and MeanShifted in terms of AD-Score, PA-Score, and RA-Score. The authors also make the dataset available.
strength_and_weaknesses: (+) The authors address a core problem and make datasets available.

(+) The technique outperforms unsupervised and self-supervised baselines.


(-) Some of the claims in the paper are overreaching: “Taken together in many cases labels for nuisance attributes they can be achieved at virtually no cost.” [Section 4.1] For the approach, the users have to provide additional labeled data. It is not clear if such labels are easily available apart from synthetic datasets.

(-) Anomaly detection applies to various types of datasets. This authors solve it in a specific setting. Does it apply other types of images or to all images or other types of data as well?

clarity,_quality,_novelty_and_reproducibility: The paper is clear in terms of technical content and can be reproduced in the current form. Technical novelty is limited once the assumption that nuisance features are part of the input is made as the paper uses well-known techniques after that.
summary_of_the_review: The paper addresses an important problem. Problem formulation being influenced by nuisance factors is very relevant.

The assumptions that users can name nuisance factors and the nuisance factors can be readily labeled are very strong assumptions which are not validated beyond the synthetic data sets explored in the paper. The technical solutions, given the assumptions, are widely known which limit the novelty of the paper.


correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper3029/Reviewer_W97W
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper concerns anomaly detection. In particular, it considers the fact that the definition of an "anomaly" depends on the use case. The goal is to flag "real anomalies" (unusual variation in features we care about) while ignoring "pseudo-anomalies" (unusual variation in nuisance features). Nuisance features are labeled in each image. The paper names this setting "negative attribute guided anomaly detection" (NAGAD). The paper benchmarks a four baseline methods on NAGAD (3 pre-existing datasets). They also provide their own method, inspired by the domain-supervised disentanglement literature. The new approach has a few components:
* ImageNet-pretraining for the image encoder.
* A SimCLR-style contrastive loss applied independently on each collection of images with the same value for the nuisance factor.  This is meant to eliminate information about nuisance factors.
* A reconstruction loss applied to the output of an image generator. The generator takes as input (i) the representation and (ii) the value of the nuisance variable. 
strength_and_weaknesses: # Strengths
* Anomaly detection is an interesting and important problem.
* The paper is generally well-written.
* The idea that it is more reasonable to specify nuisance factors a priori than it is to specify the features where you expect anomalies makes a lot of sense from a practical point of view, so the NAGAD problem setting seems well-motivated. The metrics used for benchmarking also seem sensible. 
* The experiments are reasonably thorough, with a good number of recent methods tested across several datasets. 

# Weaknesses
* When we consider Table 7 alongside Table 2, it's not obvious that the proposed method is a significant improvement. The proposed method is within the margin of error of a competitor for AD-Score, worse than a competitor for PA-Score (this is true for 2/3 datasets), and better than the competitors for RA-Score. Note that we don't have error bars for other datasets, only for the How should we understand these results?  
* There is no ablation study for the components of the proposed method. 
* These seem to be toy datasets. If NAGAD is meant to be a well-motivated practical problem, shouldn't there be more challenging real-world datasets where this problem shows up? Section 7 mentions this point, but dismisses it on the grounds that the current methods would not do well. Even if current methods wouldn't work great, it seems worthwhile to ground the study of NAGAD in realistic data. 
On the topic of practicality, there are a few points relevant to real-world uses that are not addressed:
* The paper asserts that the (more realistic) case of multiple attributes can be handled "very naturally" - I would like a more detailed explanation. 
* The paper assumes that nuisance factors are categorical. Is this a significant limitation or is there an easy way around it? 
* The paper assumes that nuisance factors are always identified correctly - what if the user is wrong about a feature being a nuisance factor? 
* If the user does not enumerate every possible nuisance factor, do they pay a price because the representation is forced to preserve everything else (including all unspecified nuisance factors)?


clarity,_quality,_novelty_and_reproducibility: # Clarity & Quality
* Some of the writing could be made clearer. For instance, I found the paragraph that begins "This objective encourages the encoder to map..." to be confusing due to the "hand-wavey" mathematical statements. I would either reference [Wang & Isola 2020] and leave the details to them, or be more precise about the statements and why they matter to the reader. Also, $F$ seems to never be used again. 
* I was surprised by how ineffective all methods seem to be at achieving invariance (Table 8). Particularly for Edges2Shoes! It doesn't really seem appropriate for any method here to claim to "learn a representation that is invariant to nuisance factors" if these are the empirical results. Am I reading this incorrectly? Please offer some insight. 

# Novelty
* To the best of my knowledge the proposed approach and the NAGAD setting are both novel. 

# Reproducibility
* Are baseline numbers copied and pasted or were they re-trained for the purposes of this paper? 
* Are all methods using the same backbone? 
* What was the protocol for hyperparameter tuning? Was it different for the proposed method and the baselines? 
summary_of_the_review: Overall I like this paper and I enjoyed reading it, but I have some questions about the details that I would like to see addressed. There also seems to be a notable disconnect between the stated real-world motivation for the paper and the work in the paper (which is focused on a simplified case with toy problems). Regardless, I find the paper to be valuable and interesting and I would ultimately like to see it accepted if these issues can be resolved. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold