From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to comment
comment: Thank you for your comment.

* 'a working input' just refers to the dataset transformation required to compute a certain distance. For example a dataset can be categorical or textual, which requires different preprocessing. 

* ICA is used as a preprocessing technique here, as datasets from ADBench(https://github.com/Minqi824/ADBench/tree/main/datasets/Classical) are not OT friendly and do not return valid distance when used with GW-LR without any processing.

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_hzxw
title: Thank you for your responses
comment: Thanks for your responses.

After reading your responses, I still cannot understand the importance of applying ICA. In the response, you noted 'GW cannot process raw data directly', but I think it is possible to apply GW as long as we can define a distance between $x^{(i)}$ and $x^{(j)}$ (or between $y^{(k)}$ and $y^{(l)}$). Also, I cannot understand the meaning of 'a working input'.

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_dXjK
title: Response to the Authors
comment: Thank you for your efforts in revising the paper. However, I think the improvement is only marginal. Therefore, I will keep my score. For example,

* W1: Vanilla OT can handle pointclouds of different sizes, and you are still applying the same transformation in Equations (7) and (8). It is still unclear to me why GW OT is needed at all.
* W2: The problem is still unclear to me. What is the difference between $\mathcal{A}$ and $\mathbf{A}$? Why the elements in $\mathcal{A}$ have stars but the ones in $\mathbf{A}$ do not? What is the evaluation measure exactly (e.g., input and output spaces)?
* W4: I am still not sure what is new in GAMAOD compared to GAMA except for its application in OD.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to reviewer
comment: Can you please elaborate on the "standard components assembled in standard ways" comment. As far as we are aware there is only one paper which deals with Unsupervised model selection (MetaOD: Current SOTA meta-learner in OD), Which uses quite a different setup from our approach. Our setup beats MetaOD baseline.

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_E9xX
title: Response to the authors
comment: I have read the authors' responses, and appreciate their clarifications as to why they chose not to compare their proposed LOTUS framework against MetaBu. Nevertheless, my main concern still stands - the level of novelty is rather low, in that the framework consists of standard components assembled in standard ways.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Follow-up
comment: Hi,

It has been a few weeks since we have posted our rebuttal. As the end of discussion phase is coming to an end, we hope you had a chance to read our rebuttal, please let us know if you have any further comments or concerns to which we can respond.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Follow-up
comment: Hi,

It has been a few weeks since we have posted our rebuttal. As the end of discussion phase is coming to an end, we hope you had a chance to read our rebuttal, please let us know if you have any further comments or concerns to which we can respond.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Follow-up
comment: Hi,

It has been a few weeks since we have posted our rebuttal. As the end of discussion phase is coming to an end, we hope you had a chance to read our rebuttal, please let us know if you have any further comments or concerns to which we can respond.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title:  Follow-up
comment: Hi,

It has been a few weeks since we have posted our rebuttal. As the end of discussion phase is coming to an end, we hope you had a chance to read our rebuttal, please let us know if you have any further comments or concerns to which we can respond.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to  Reviewer hzxw
comment: We thank the reviewer for the detailed review. We think you have helped us making this paper better. 

* W1: “The idea of applying Gromov Wasserstein distance to meta-learning is not novel, as the authors discussed in Section 5.3.2. The main difference against the existing MetaBu is that LOTUS uses GW-LR with ICA in an unsupervised setting.”
We have explained it better in our paper now, the previous meta learning approaches have used it for warm starting pipelines in a very expensive manner but we introduce it in a zero-shot scenario for model selection on unsupervised tasks.


* W2: “The authors propose to use ICA with GW-LR, but the reason for that is not technically discussed nor experimentally examined.”
The use of ICA is to make the dataset friendly to GW as GW cannot process raw data directly. Our idea for a transformation function here is to make sure we have a working input for the given distance function. The transformation function can be none as well if the data is already supplied in probability masses or point clouds. And this transformation function can have multiple elements if the datasets are dirty, categorical or have any other flaws. 


* W3: “I could not understand the novelty of GAMAOD that is discussed in Section 3.2.1. The expression like 'to select the optimal unsupervised algorithm with access to labels' is confusing.”
We have fixed this statement and added GAMAOD to appendix to avoid confusion. GAMAOD is the engineering tools we developed to populate our meta-data, this tool was developed as a byproduct of this research but its nota. main contribution. 
* W4: The authors list baseline OD methods in Appendix 2 including LOF, but it is not compared with the proposed method in Table 1.
Thanks for noticing our mistake, we included LOF in the baselines

* W5: We  fix the all listed notational issues 
We have fixed issues mentioned here. In eq 3 we use the subscript epsilon to describe the entropic GW to avoid confusion with simple GW


“Also, if possible, I would like to see the results of MetaBu in the unsupervised setting.”
MetaBu Is an approach to warm-start other autoML tools whereas LOTUS is a zero-shot model selection approach. MetaBu’s approach cannot be easily transformed into an unsupervised setting as it is built on top of hand-selected supervised meta-features. these meta features are very label dependent and do not scale well for the unsupervised problem, hence we use OT distances here on the dataset. As well as there is no search algorithm here to warm start.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to Reviewer vhGa
comment: We thank the reviewer for their comments and review, we have changed significant part of the papers to accommodate this review. 

* W1: The paper is hardly readable. Formalisms are introduced with very confusing explanations. Furthermore, formalisms are not used correctly, e.g., in Eq. 4 GW-LR^(r) is used as a function receiving 2 arguments in the form of 2-tuples each. Later on, in Eq. 10, I cannot recognize any such tuples although the function GW-LR^(r) is used. Also it is questionable what an iterator is supposed to be.
We have rearranged the paper significantly. We have also removed multiple equations which were confusing to reviewers. The iterator is better described in Agorithm1 as a loop so we have introduced it there and removed the equation

* W2: A lot of information is tried to be given but not really delivered to the reader. We have rearranged and rewritten a good part of the paper to fix this, but if you can point us to specific areas where this is required, we can also fix that.

* W3: Wrong claims such as "outlier detection methods cannot be cross-validated " are made and several claims are made throughout the paper lacking support either by a reference to the literature or by providing evidence in the paper itself.
What we meant to say is that we didn’t use cross-validation since it is not required in our setting. We run outlier detection methods on all unlabeled data, and use the ground truth labels only to evaluate them. We do not need to create training and test sets. We could of course use cross-validation or multiple random samples to get an averaged estimate. This would create a number of additional concerns, though. For instance, there may be fewer outliers than folds in some tasks or very few outliers per fold. Aspects such as the size of the dataset would also play a role in choosing a resampling method. For all these reasons, we chose to use all data at this point. In future work, we do aim to evaluate whether this might provides a clear advantage.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to Reviewer dXjK
comment: According to your suggestion we have rewritten and reorganised the paper heavily. 
* W1: We do apply ICA but still use all the components to keep information loss at minimum. ICA is applied as GW cannot handle raw data. The size of two pointclouds is still different therefore they are not suitable for vanilla OT. The purpose of ICA here is to merely make the datasets OT-friendly, transformation can be different for different datasets. We have clarified this in the paper.
* W2: We fix the notations so that its better understandable.
* W3: We fix the described problem, we focus on LOTUS in the paper now with GAMAOD as an additional contribution in the appendix
* W4: We introduce GAMA briefly in methodology section
* W5: We fix the equations
  * W5.1: We clarified A and B before the equation in paper, they refer to distributions. 
  * W5.2: We put the notations before the equation for better understanding.
  *  W 5.3: What we meant to say is that we didn’t use cross-validation since it is not required in our setting. We run outlier detection methods on all unlabeled data, and use the ground truth labels only to evaluate them. We do not need to create training and test sets. We could of course use cross-validation or multiple random samples to get an averaged estimate. This would create a number of additional concerns, though. For instance, there may be fewer outliers than folds in some tasks or very few outliers per fold. Aspects such as the size of the dataset would also play a role in choosing a resampling method. For all these reasons, we chose to use all data at this point. In future work, we do aim to evaluate whether this might provides a clear advantage.
  * W 5.4: We have rearranged the paper so that the CASH formulation and statement is introduced

From: ICLR.cc/2023/Conference/Paper5596/Authors
title:  Response to Reviewer E9xX
comment: We thank the reviewer for the thorough comments and review, we have listed replies to all the weaknesses below
* W1: “There is little novelty in the contributions of the paper. All techniques used are now considered to be standard, with the possible exception of the minor tweak to CASH noted above.”
This paper proposes OT for unsupervised problems where it is more suitable than supervised problems as generation of metafeature is a problem in unsupervised setting. For example PyMFE mostly generates features for supervised tasks only. As far as we are aware only metabu has applied OT to automl on tabular data tasks. MetaBu does not work for unsupervised tasks and only work as a warm starting technique for the model selection algorithm, not as a zero-shot framework by itself. 

* W2: “Although the authors discuss the differences between their metalearning framework and MetaBu (presented at ICLR 2022), they do not include this recent but important competitor in their experimental evaluation.”
We apologize for being unclear here. MetaBu only worked with supervised classification on tabular datasets, therefore we cannot use metabu here. To be more clear we have include table1 where you can see the differences between MetaBU, MetaOD and LOTUS. To be more clear we have included comparison in Table 1 with different AutoML frameworks. 

W3: “Although some additional detail on the experimental results appears in the supplementary materials, it is far from complete. For each dataset, the final AUC scores achieved by both LOTUS (the authors' metalearner) and MetaOD are stated; however, the full table of AUC scores (for each combination of algorithm and dataset) is given only for LOTUS. The algorithm hyperparameters choices and other execution details are not provided.”
We describe in the final appendix that MetaOD crashes on some datasets therefore the details are not included in the AUC table. According to your comments, we have included the best-selected pipeline for all the datasets in the supplementary materials in a csv file as the pipeline text was quite cumbersome for the paper. The comparing pipelines are evaluated at default settings in PyOD. For reproducibility, we also provide the code.

From: ICLR.cc/2023/Conference/Paper5596/Authors
title: Response to common concerns
comment: We thank all the reviewers for their useful and insightful comments. We have thoroughly restructured the paper, significantly improved the writing, cleared up some points of confusion, and added additional results as requested by you. Below, we reply to each individual review. As an overview, the major changes include the following:
1. We restructured the paper to make it much easier to read.
2. We significantly improved the writing in the methodology section and improved the clarity and correctness of the formal description.
3. We moved the implementation aspects (the section on GAMAOD) to the appendix so that readers can focus on the LOTUS method as the main contribution
4. As requested, we added Local Outlier Factors (LOF) as a baseline. 
5. We explain why we cannot directly compare MetaBu and LOTUS. While we think that MetaBu is a very interesting method that is related in many ways, we could not apply it in unsupervised settings. MetaBu is inherently supervised: it requires that new tasks are supervised in order to compute the meta-features. LOTUS uses optimal transport in a purely unsupervised way. It only requires labels from proxy training tasks (at meta-train time), and can make predictions for new unsupervised tasks (at meta-test time).
6. Changes are in blue
7. We add best selected model in file odmodels.csv in the supplementary materials

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_E9xX
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper is concerned with automating unsupervised outlier detection through metalearning from past performance of outlier techniques on reference datasets. The authors assume a repository of both datasets and outlier algorithms, with information as to the degree of suitability of each algorithm for each dataset. Given a test dataset, optimal transport learning is used to find the reference dataset with the most similar underlying distribution, and then outlier detection methods are selected that are known to work best for that distribution. 


strength_and_weaknesses: Strengths:

1) The components of the proposed metalearning framework seem to be a competent assembly of standard techniques:
* OT learning with the negative entropy sinkhorn algorithm for scalability, with low-rank approximation the Gramov-Wasserstein cost to allow comparison between datasets from different domains;
* Automated model selection using a variant of CASH (Combined Algorithm and Hyperparameter Selection) in which cross validation is replaced by training of the entire set without outlier labels and evaluating with labels;

2) The experimental evaluation seems quite comprehensive: the largest open source anomaly detection dataset repository, ADBench, together with 7 well-established outlier detection algorithms from the PyOD library, evaluated against a recent metalearner for outlier detection, MetaOD. The authors claim better performance against MetaOD according to the Bayesian Wilcoxon signed-rank test (ROPE test).  

Weaknesses:

1) There is little novelty in the contributions of the paper. All techniques used are now considered to be standard, with the possible exception of the minor tweak to CASH noted above. 

2) Although the authors discuss the differences between their metalearning framework and MetaBu (presented at ICLR 2022), they do not include this recent but important competitor in their experimental evaluation.

3) Although some additional detail on the experimental results appears in the supplementary materials, it is far from complete. For each dataset, the final AUC scores achieved by both LOTUS (the authors' metalearner) and MetaOD are stated; however, the full table of AUC scores (for each combination of algorithm and dataset) is given only for LOTUS. The algorithm hyperparameters choices and other execution details are not provided.

clarity,_quality,_novelty_and_reproducibility: The paper is generally well-organized and readable. However, there are numerous typesetting issues, spelling mistakes, and grammatical errors that need attention. 

Although the proposed framework seems competently engineered (see S1 above), there is little novelty in the approach (see W1).

Not enough information is provided in the main paper or supplementary material for the experimental results to be reproduceable (see W3 above).


summary_of_the_review: Although the proposed automated metalearning framework for outlier detection appears quite competitive with a competing framework (MetaOD), not enough experimental evidence is given to judge its full impact in practice. This work seems to fall below the level of novelty expected by ICLR, in that the framework consists of standard components assembled in standard ways.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_dXjK
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper studies the problem of unsupervised outlier detection (OD). The authors propose a meta-learner (LOTUS) for OD based on optimal transport. The key idea is to use the OT-based distances to select from the meta datasets the closest one to the new dataset. They also equip LOTUS with an extension of the AutoML tool GAMA.
strength_and_weaknesses: **Strength:**
1. The problem studied is highly relevant.
1. The idea of meta-learning for OD with OT is interesting.

**Weakness:**
The writing of the paper can be largely improved. I found it very difficult to understand and evaluate the contribution of this paper. Concretely, I have the following comments:
1. Why do we need the Gromomv Wasserstein distance? The motivation of the GW distance is mainly for the setting when two point clouds live in different metric spaces. However, in the paragraph above (1), the two point clouds are assumed to live in the same space $R^d$. And in (8), where the GW distance is used in LOTUS, the same transformation $\phi$ is used for the two datasets, so they should live in the same space, right?
1. The problem statement in Sec. 3.1 is unclear. What is $\mathcal{A}$ concretely? What does the notation $A^*_{\lambda^*}$ mean?
1. The proposed method is LOTUS + GAMAOD, but you introduced them in reverse order.
1. Below (6), GAMAOD is introduced as an extension of GAMA. However, it is unclear to me what is new here. Is it true that GAMAOD is a direct application of GAMA on outlier detectors? It would be much more clear if you introduce concretely GAMA and highlight the difference between GAMAOD and GAMA.
1. Some of the equations are really hard to understand. For instance:
    1. What are $A$ and $B$ in (1)?
    1. In (5), the notations $\mathbf{A}$, $\Lambda_{\mathbf{A}}$, and $A_\lambda^j$ are not defined.
    1. In (6), it was mentioned that CV cannot be done without labels, but isn't $y_{\text{train}}$ the labels?
    1. (7) and (8) written together are very confusing. Is (8) one possible realization of (7)? Moreover, where are $A$ and $B$ in (8)?
    1. I cannot understand (10) at all. What are $\Gamma_{\Gamma \in D_{meta}}$, $D_{D \in D_{meta}}$, $A^*_{\lambda^* new}$, and $A^*_{\lambda^* \Gamma}$?

**Minor comments:**
1. In (3), $\varepsilon$ should not be a subscript.
1. The sentence below (3), "between our distributions $A$ and $B$" --> $a$ and $b$.
1. Using $f$ to represent the index as in (5) is not common.
1. You did not introduce the CASH problem on page 5.
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the writing of the paper largely prevents me from evaluating the quality and novelty of the work.
summary_of_the_review: This paper studies a relevant problem with an interesting idea. However, the writing of the paper largely prevents me from understanding and evaluating the contribution of the paper. I thus recommend rejection.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_vhGa
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper "Meta-learning for unsupervised outlier detection with optimal transport" proposes a meta learning approach for algorithm selection in the domain of outlier detection. To this end, the authors propose to rely on optimal transport as a key technology for generalization across different datasets for outlier detection.
strength_and_weaknesses: Strengths:
- Important problem addressed
- Maybe an interesting direction to go

Weaknesses:
- The paper is hardly readable. Formalisms are introduced with very confusing explanations. Furthermore, formalisms are not used correctly, e.g., in Eq. 4 GW-LR^(r) is used as a function receiving 2 arguments in the form of 2-tuples each. Later on, in Eq. 10, I cannot recognize any such tuples although the function GW-LR^(r) is used. Also it is questionable what an iterator is supposed to be.
- A lot of information is tried to be given but not really delivered to the reader
- Wrong claims such as "outlier detection methods cannot be cross-validated " are made and several claims are made throughout the paper lacking support either by a reference to the literature or by providing evidence in the paper itself.
clarity,_quality,_novelty_and_reproducibility: Clarity is missing somewhat throughout the paper. The command of language is not that good, i.e., there are still grammatical mistakes and incomplete sentences throughout the paper. One extreme example is the last paragraph of Section 3.1. The paper could also be better structured and better guide the reader through the paper. Capitalization in section headers is completely random.

The overall quality is rather bad and not on the level of ICLR. It is extremely difficult to follow the formal part as explanations are rather confusing than helpful, and formal symbols are used differently than defined.

Novelty might be a merit of the paper, although, as the authors already acknowledge, optimal transport has already been used for meta-learning. Also here it is not clear what the required augmentation is to enable meta-learning for outlier detection.

Due to the confusing theoretical part and missing details, this paper could not be reproduced. Results for GAMAOD (the other proposed method) are not really mentioned.
summary_of_the_review: The paper falls short in quality, reproducibility, clarity and has limited novelty. Therefore, I recommend to reject the paper.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper5596/Reviewer_hzxw
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a meta-learning approach for outlier detection (OD). GAMAOD extends the existing GAMA framework to generate a collection of pairs of datasets and optimized models. LOTUS is then applied to a new dataset to calculate the similarity between the new dataset and each dataset in the collection. Finally, the optimized model corresponding to the nearest dataset is applied to the new dataset. The similarity between datasets is measured using Independent Component Analysis (ICA) and the Low-Rank Gromov Wasserstein (GW-LR) distance.
Experiments are conducted using tabular datasets showing the advantage of the proposed method against MetaOD and other OD algorithms in PyOD library.

strength_and_weaknesses: Strength
* LOTUS is simple yet achieving comparable or better results than MetaOD
    * MetaOD requires some OD models to generate model-dependent meta-features, whereas LOTUS uses no model to generate meta-features.


Weakness
* The idea of applying Gromov Wasserstein distance to meta-learning is not novel, as the authors discussed in Section 5.3.2. The main difference against the existing MetaBu is that LOTUS uses GW-LR with ICA in an unsupervised setting.
* The authors propose to use ICA with GW-LR, but the reason for that is not technically discussed nor experimentally examined.
* I could not understand the novelty of GAMAOD that is discussed in Section 3.2.1. The expression like 'to select the optimal unsupervised algorithm with access to labels' is confusing.
* The authors list baseline OD methods in Appendix 2 including LOF, but it is not compared with the proposed method in Table 1.
* There are some notational issues in the manuscript (listed below).


clarity,_quality,_novelty_and_reproducibility: The manuscript has some notational issues, for example:
* In Equation (3), why $\epsilon$ is subscripted?
* Fonts are not unified, for example $\epsilon$ in $GW_{\epsilon}$ in p.3.
* In Section 3.2, $D_i$ should be $\{(X_i^{train}, y_i^{train}), (X_i^{test}, y_i^{test})\}$?
* In Equation (5), $\forall$ must be removed. Also, $X_{train}^f$ should be $X_f^{train}$.
* The usage of $\bm{A}, \mathcal{A}, A$ are not unified.
* In p.5, what is the CASH problem.
* In p.5, '. which results in' -> ', which results in'
* In Equation (10), what is the meaning of $\Gamma_{\Gamma \in D_{meta}}$?
* In p.6, Where -> where
* In Algorithm 1, $\phi(D_{new}, D_i)$ -> $\phi(D_{new}) \phi(D_i)$?
* In p.6, 'OTT-JAX library (Cuturi et al., 2022) library'
* In p.7, shon -> shown
* In p.7, 74.2\% -> 74.0\%?
* In p.8, 'is generalises' -> generalises
* In Appendix 2, LDOA -> LODA

summary_of_the_review: The technical novelty of the proposed method is not so significant, but the experimental results suggest the usefulness of the approach. I think the following points should at least be answered in the revision:
* The importance of applying ICA in GW-LR should be discussed, theoretically or experimentally.
* The proposed method should be compared with LOF.

Also, if possible, I would like to see the results of MetaBu in the unsupervised setting.


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold