M ETA - LEARNING FOR UNSUPERVISED OUTLIER DETECTION WITH OPTIMAL TRANSPORT
Anonymous authors
Paper under double-blind review
A BSTRACT
Automated machine learning has been widely researched and adopted in the field of supervised classification and regression, but progress in unsupervised settings has been limited.
We propose a novel approach to automate outlier detection based on meta-learning from previous datasets with outliers.
Our premise is that the selection of the optimal outlier detection technique depends on the inherent properties of the data distribution.
We leverage optimal transport in particular, to find the dataset with the most similar underlying distribution, and then apply the outlier detection techniques that proved to work best for that data distribution.
We evaluate the robustness of our approach and find that it outperforms the state of the art methods in unsupervised outlier detection.
This approach can also be easily generalized to automate other unsupervised settings.
I NTRODUCTION
Outlier detection(OD) is the process of identifying data points that are significantly different from the rest of the data.
These data points can be caused by errors in the data collection process, incorrect values, or unusual events.
Outlier detection can be used to improve the quality of the data or to find unusual events that could be interesting to different business and scientific domains .
The term ”outlier detection” can be interchangeably used with ”anomaly detection”.
For consistency, we will use the term ”outlier detection” in this paper.
Outlier detection has multiple applications such as medicine (Chow & keung Tse, 1990; Ma et al., 2021b), chemistry (Egan & Morgan, 1998) and molecular biology (Cho & Eo, 2016).
Outlier detection has been a particularly hard problem.
A number of Outlier detection algorithms have been introduced in the last two decades (Aggarwal, 2013).
Unsupervised outlier detection is a very challenging task with no universally good model which works optimally on every task (Campos et al., 2015).
AutoML (Hutter et al., 2019) has shown reliable performance and benefits in model selection and hyperparameter optimization (Hutter et al., 2019; Feurer et al., 2015; Thornton et al., 2013).
The research in Automated machine learning has been highly focused on supervised machine learning where we can focus on the performance on the hold-out dataset to define an optimization metric for the search algorithm which finds the optimal algorithms by iterating over the search space.
This setting is very reliable (Feurer et al., 2015) but the research on unsupervised setting is rather limited.
In recent years frameworks like MetaOD (Zhao et al., 2021) have appeared which attempt to solve automated outlier detection via meta-learning (Vanschoren, 2018).
In this work we propose an automated framework for unsupervised machine learning tasks LOTUS (Learning to learn with Optimal Transport for Unsupervised Scenarios), which leverages metalearning (Vanschoren, 2018) and optimal transport distances (Peyr´e & Cuturi, 2019; Scetbon &
Cuturi, 2022).
In this work we use LOTUS to perform model selection on a given unsupervised outlier detection task.
In summary, we make the following 4 contributions:
• A Meta-learner for outlier detection : We propose LOTUS :
Learning to learn with Optimal Transport for Unsupervised Scenarios, an optimal transport based meta-learner which recommends an optimal outlier detection algorithm based on a historical collection of datasets and models in a zero-shot learning scenario.
Our solution can be used in cold start settings for model selection on unsupervised outlier detection.
[Table]
• Experiments and results: We empirically evaluate LOTUS in combination with existing state of the art methods.
We demonstrate the robustness of our approach against existing state of the art meta-learners and learners.
• Open source: We open-source the code for LOTUS for researchers to use and reproduce our experiments.
Our tools can be extended with new datasets and algorithms.
B ACKGROUND
This section describes related work regarding Automated Machine learning for unsupervised outlier detection, optimal transport and meta-learning.
2. AUTOML FOR OUTLIER DETECTION
AutoML (Hutter et al., 2019) for unsupervised outlier detection is an extremely hard problem due lack of an optimization metric to perform algorithm selection.
One can argue that the use of internal metrics like Excess-Mass (Goix, 2016), Mass-Volume (Goix, 2016) and IREOS (Marques et al., 2015) can make algorithm selection possible.
Ma et al.
(2021a) shows in their experiments that these internal metrics are computationally very expensive and do not scale well for large datasets.
This makes it unfeasible to use these metrics in AutoML tools for most real world scenarios.
There has been recent research on AutoML for outlier detection.
PyODDS and MetaOD (Li et al., 2020; Zhao et al., 2021) are among the few tools which have been shown to automate outlier detection.
To the best of our knowledge MetaOD (Zhao et al., 2021) is the current state of the art metalearner for model selection on outlier detection tasks for tabular data.
MetaOD uses meta-learning as a recommendation engine using landmark meta-features and model based meta-features with collaborative filtering (Stern et al., 2010) to perform model selection for a given task.
2. META LEARNING
Meta-learning or learning to learn in AutoML (Vanschoren) is the study of learning from historical performances of machine learning models on a variety of tasks and using this knowledge to find better models for new tasks.
Meta-learning can help to speed up the model selection process and find better architectures.
Meta-learning is often proposed as a solution to cold start problem , by initializing the hyperparameters or search space for the AutoML algorithm.
This is often called warm-starting for AutoML.
Meta-learning in existing AutoML tools: Different AutoML tools use different meta-learning schemes to solve this cold start problem.
AutoSklearn-2.0 (Feurer et al., 2020) learns pipeline portfolios, MetaOD
(Zhao et al., 2021) trains a collaborative filtering
based algorithm (Stern et al., 2010) with landmark-based and model-based metafeatures (Castiello et al., 2005), FLAML (Wang et al., 2021) uses in-built meta-learned defaults for warm starting.
MetaBu
(Rakotoarison et al., 2022) uses Fused Gromov Wasserstein with proximal gradient method on landmark meta-features for warm-starting AutoSklearn (More discussion about LOTUS vs MetaBu is provided in the section 2.4.2).
2. O PTIMAL TRANSPORT AND DATASET DISTANCES
Optimal transport(OT) theory deals with the problem of finding an optimal transport map between two probability measures, often on different metric spaces.
It is closely related to Monge’s problem (Villani, 2008), in which one searches for the optimal transport map between two given measures.
An Optimal transport problem consists of minimizing the cost of transporting mass from one distribution to another.
For cost function(ground metric) between pair of points, we calculate the cost matrix C with dimensionality n × m , the OT problem minimizes the loss function L c ( P ) := ⟨ C, P ⟩ w.r.t a coupling matrix P .
Most common approach with practitioners is to use a regularized approach which is computationally more efficient L ϵc ( P ) :
= ⟨ C, P ⟩ + ϵr ( P ) where r is negative entropy sinkhorn algorithm (Cuturi, 2013) which is computationally more efficient.
A discrete OT problem can be defined with two finite pointclouds, { x ( i ) } ni =1 , { y ( j ) } mj
=1 , x ( i ) , y ( j ) ∈ R d , which can be described as two empirical distributions: µ := (cid:80) ni
=1 a i δ x ( i ) , ν := (cid:80) mj
=1 b j δ y ( j ) .
Here a and b are the probability vectors of size n and m .
In this work we are interested in the the Gromov
Wasserstein(GW) distance between these two discrete probability distributions.
Gromomv Wasserstein allows us to match points taken within different metric spaces.
This problem can be written as a function of ( a, A ) , ( b, B ) between our distributions A and B (Villani, 2008; Scetbon et al., 2022):
[Equation]
where Π a,b := { P ∈ R n × m +
[Equation]
the energy Q A,B is a quadratic function of P which can be described as
[Equation]
In this work we are interested in the Entropic Gromov Wasserstein cost (Peyr ´ e et al., 2016):
[Equation]
where GW ϵ is the Entropic Gromov Wasserstein cost between our distributions A and B , and εH ( P ) is the shannon entropy.
The problem with Gromov Wasserstein is that it is NP-hard and the entropic approximation of GW still has cubic complexity.
To speed up the computations and use it in a realistic AutoML settings we use the Low-Rank Gromov Wasserstein (GW-LR) approximation (Scetbon et al., 2021; Scetbon & Cuturi, 2022; Scetbon et al., 2022), which reduces the computational cost from cubic to linear time.
Scetbon et al.
(2022) consider the GW problem with low-rank couplings, linked by a common marginal g .
Therefore, the set of possible transport plans is restricted to those adopting the factorization of the form P r = Qdiag (1 /g ) R T .
In this form Q and R are thin matrices with dimensionality of n × r , r × m respectively and g is a r − dimensional probability vector.
The
GW-LR distance is be described as:
[Equation]
Our primary inspiration for LOTUS comes from two different works.
Alvarez-Melis & Fusi (2020) proposes optimal transport dataset distance(OTDD) which uses optimal transport to learn a mapping over the joint feature and label spaces.
AlvarezMelis & Fusi
2020) proposed that optimal transport distances can be used as a similarity metric between different datasets from different domains and subdomains.
Work of Nies et al.
(2021) argues that optimal transport measures can be used as a correlation measure between two random variables via transport dependency
There have been other studies exploring the space of dataset and task similarity with distance measures.
Gao & Chaudhari (2021) proposes “coupled transfer distance” which utilises optimal transport distances as a transfer learning distance metric.
Achille et al.
(2021) explores connections between Deep Learning, Complexity Theory, and Information Theory through their proposed asymmetric distance on tasks. 2. RELATED WORKS
In this section we will discuss the difference between closest approaches to LOTUS which are
MetaOD and MetaBu. We have also added Table 1 to show how LOTUS differs from other metalearning approaches. 2.4. LOTUS VS METAOD LOTUS and MetaOD solve the same problem of model selection problem for unsupervised outlier detection.
The major difference in LOTUS and MetaOD is meta-feature generation.
LOTUS aims to capture the similarity of the given source and target representations via optimal transport.
MetaOD captures similarity with a combination of landmark-features and model-based features and uses a rank-based criteria called discounted cumulative gain for model selection.
MetaOD also uses stochastic algorithms such as Isolation Forest and LODA for model-based meta-feature generation which means that the absolute dataset similarity and ranking can differ based on the number of runs.
Our approach generalises better than MetaOD as well for different unsupervised tasks as it aims to find similar dataset independent of task, whereas MetaOD’s similarity is highly coupled with the task of outlier detection.
2.4. LOTUS VS METABU
MetaBu (Rakotoarison et al., 2022) was proposed as a solution to cold start problem in supervised learning scenario.
Rakotoarison et al.
(2022) uses Fused-Gromov-Wasserstein distance with multi dimensional scaling (Cox & Cox, 2008) by first extracting meta-features from the target representation and source representation and proximal gradient method (Xu et al., 2020).
LOTUS is a solution for unsupervised setting whereas MetaBu relies on landmark features from PyMFE (Alcobac¸a et al., 2020) which are more reliable for datasets with labels.
Similar to MetaOD, MetaBu setting is limited to only one task (supervised classification) as it relies on landmark-features which require labels.
MetaBu is used for warm starting not selecting the best pipline in a zero shot setting.
M ETHODOLOGY
We introduce LOTUS, Learning to learn with Optimal Transport for Unsupervised Scenario.
LOTUS meta-learns how well different unsupervised algorithms work on prior labeled datasets.
These can be datasets where the correct labels are known, or proxy tasks.
For instance, for outlier detection we can use extremely imbalanced classification tasks where examples of the smallest class are considered outliers.
More formally, we require:
• A collection of n prior labeled datasets D meta
= { D 1 , ..., D n } with test and train splits such that D i =
( X traini , y traini ) .
• A collection of n optimized algorithms
A ∗ i with associated tuned hyperparameters λ ∗ i (Use of ∗ indicates tuned version of model/hyperparameters)for every dataset in D meta ; A = { A ∗ λ ∗ 1 , ..., A ∗ λ ∗ n }
3. META - TRAINING
In this section, we formally describe the problem of model selection for unsupervised outlier detection.
Problem Statement:
Given a new dataset without any labels, our meta-learner needs to selects an optimal algorithm with associated hyperparameters from a collection of previously evaluated pipeline.
In this setting, we cannot optimize the given model for the dataset as there are no given labels.
This problem becomes from a Combined model selection and hyperparameter optimization problem to a zero-shot model reccomendation problem .
Given a new input dataset (i.e., detection task)
D new =
( X new ) without any labels, Select a model
A ∗ λ ∗ ∈
A to employ on X new .
Where A ∗ λ ∗ is a optimal model with tuned hyperparameters λ ∗ for a similar dataset to X new .
Problem Formulation: A Combined model selection and hyperparameter optimization problem (Thornton et al., 2013) for a supervised learning task is as follows:
Inputs: D new , D meta , A 1: while D i ∈ D meta do 2:
O i ← ψ ( ϕ ( D new , D i ))
▷ Distance calculation 3: s ← argmin i {O 1 , ..., O n } ▷ Retrieval of most similar dataset 4: A ∗ λ ∗ new ← A ∗ λ ∗ s
▷ Model Selection
In equation 5, A ∗ λ ∗ is an optimal combination of learning algorithm from search space A with associated hyperparameter space Λ
A over k cross validation folds of dataset D where D =
X, y } with training and validation splits.
L is our evaluation measure.
[Equation]
The CASH problem from equation 5 relies on the validation split to optimise for the optimal configuration.
However, in unsupervised outlier detection scenario the learning algorithm does not have access to labels but the AutoML framework does.
We do not perform k-fold CV is not useful in this setting.
Our modified CASH formulation to select the optimal unsupervised algorithm with access to labels is as follows:
[Equation]
GAMAOD:
For meta-training in this work we develop GAMAOD as a solution to populate our meta-data.
GAMAOD is an extension to popular AutoML tool GAMA (Gijsbers & Vanschoren, 2021).
GAMA is a general AutoML framework which allows researchers to integrate different search spaces and search algorithms for model selection easily. More details about GAMAOD architecture are provided in Appendix A.
3. META - TESTING
Our premise is that, if a prior dataset exists that is very similar to the new dataset, then its optimal algorithms will likely work well on the new dataset. We consider two datasets similar if they have the same underlying data distribution, which we measure using Optimal Transport.
We first require a transformation function ϕ , the purpose of transformation function is to simply make input data compatible to the distance function, for example if it is raw image data then a typical transformation function is normalization of pixels, if it is dirty data then a transformation function can be a mix of encoders and scalers.
Next, we calculate the dataset similarity O based on some distance metric ψ in equation 7.
[Equation]
Because our distributions lie on different metric spaces, and we require computationally efficient similarity estimates, we adopt the Low Rank Gromov-Wasserstein distance from equation 4 on these transformed distributions, as summarized in equation 8, where r is the selected rank.
[Equation]
The most similar prior dataset D s ∈ D meta is then the dataset with the smallest distance to the new dataset D new .
LOTUS then assigns the optimal configuration from A : A ∗ λ ∗ new =
A ∗ λ ∗ s where A ∗ λ ∗ s is predicted as the optimal configuration for D new .
Meta-testing is summarized in Algorithm 1.
LOTUS framework with meta-training and meta-testing is shown in Figure
EXPERIMENTS ON ADBENCH
For our experiments, we use ADBench (Han et al., 2022) and retrieve all tabular datasets.
This collection consists of 46 datasets.
As we do not have access to multiple benchmarks we use a leave-one-out strategy for the evaluation of our system, i.e., we take out one dataset at a time from ADBench and use only the other datasets in the meta-data.
This ensures independent meta-training on the following datasets.
We compare our approach against 7 outlier detection algorithms available in PyOD (Zhao et al., 2019) and the current state of the art meta-learner for outlier detection MetaOD (Zhao et al., 2021).
From PyOD we compare our approach with the following algorithms: IForest (Liu et al., 2008), ABOD (Kriegel et al., 2008), OCSVM (Sch ¨ olkopf et al., 1999), LODA (Pevn ´ y, 2015), KNN (Angiulli & Pizzuti, 2002; Ramaswamy et al., 2000), HBOS (Goldstein & Dengel, 2012).
For experimental consistency, we use the same search space in our experiments as MetaOD (A.3) to ensure a fair comparison.
We use an asynchronous evolutionary algorithm to iterate over the search space and return the optimal pipeline.
[-ICA and then converted into JAX (Bradbury et al., 2018) pointclouds geometry objects 
-]
R ESULTS AND DISCUSSION 5. EXPERIMENTAL RESULTS
We use the Bayesian Wilcoxon signed-rank test (or ROPE test, Benavoli et al.
(2017; 2014)) to analyze the results of our experiments.
ROPE defines an interval wherein the differences in model performance are considered equivalent to the null value.
Using this test allows us to compare model performances in a more practical sense.
We set the ROPE value to 1% for our experiments.
We use the baycomp library (Benavoli et al., 2017) to run and visualize the analyses.
For pairwise comparison of LOTUS and MetaOD, we use the Bayesian Wilcoxon signed-rank test (or ROPE test Benavoli et al. We use AUC as our performance measure and set the
ROPE value to 1%. Results are shown in Figure 2. We find that, based on experiments over the analysis [Table]
datasets, there is a 74.0 % probability that LOTUS will outperform MetaOD
Since p ( LOTUS p ( MetaOD ) LOTUS proves to be more robust
We show the per-dataset performances in
The results of the ROPE test comparing LOTUS with individual outlier detection techniques are shown in Table 2.
LOTUS proves to be significantly better than other techniques, with default parameters.
In this case P ( LOTUS ) >> P ( Estimator ) .
We also include the critical difference plot of LOTUS vs PyOD estimators in Figure 3, again showing that it performs significantly better.
The detailed experimental results are reported in appendix A.1 table 4 and Figure 4. [Figure]
5. USING OPTIMAL TRANSPORT DISTANCES AS A SIMILARITY MEASURE
In our experiments, we show that LOTUS is more robust and better than current state of the art meta-learner MetaOD for unsupervised outlier detection tasks and other outlier detection algorithms in default configuration.
In our method we experimentally show that using optimal transport distances like GW-LR is a feasible approach for dataset similarity and meta-learning.
We would like to emphasize that this similarity measure should only be used as a relative similarity measure, for e.g. in our case where we use this similarity measure to find the most similar dataset from a collection of datasets in D meta .
To estimate to what degree datasets are similar Nies et al.
(2021) proposes optimal transport based correlation measures that can be leveraged.
Our approach assumes that Wasserstein distances can capture intrinsic properties of datasets and can capture the similarity between them, Alvarez-Melis & Fusi (2020) also proposes their approach with optimal transport distances to provide some sort of distance between dataset.
CONCLUSION AND FUTURE WORK
Model selection for unsupervised outlier detection is a challenging task.
We do not have efficient internal metrics for evaluating an algorithm without ground truth.
In this work, we proposed a new meta-learner: LOTUS , which uses optimal transport distances to capture the similarity between datasets and uses that similarity measure to recommend pipelines from a meta-data.
Through our experiments, we demonstrate that LOTUS outperforms MetaOD and other built-in estimators in
PyOD.
The LOTUS approach also enables researchers to use a simplified meta-learning framework as compared to other landmark and model-based meta-features methods where meta-features are highly specialized according to the domain.
6. LIMITATIONS
LOTUS :
LOTUS depends on the quality of meta-data, i.e. range of datasets and algorithms in our case.
In the worst case scenario, if there are no similar datasets in the D meta , LOTUS can recommend a dataset which is not sufficiently similar to new dataset.
On the other hand, it is expected to improve as more benchmarks and datasets with different properties become available.
OT distance: The computation cost of GW-LR on really large datasets can still be very high.
In these cases we recommend using stratified sampling or random sampling depending on the nature of dataset and problem.
Parameter tuning:
Tuning rank of GW-LR can be tricky.
Low rank can result in faster computation but high loss and high rank can result in less efficient algorithm.
Scetbon et al.
(2022) states an experiment where they study the affect of rank of GW-LR.
This rank can also be tuned by minimizing the loss between GW and GW-LR.
We believe that our approach can be easily extended as a meta-learner to perform model selection in other unsupervised machine learning tasks as well.
These include clustering, distance metric learning, density estimation and covariance estimation.
This approach can also be used as a metalearner to warm-start neural architecture search(NAS) problems.
REPRODUCIBILTY STATEMENT
We opensource both LOTUS and GAMAOD with hyperparameters used for this experiment.
We also provide scripts which can be used to perform these experiment for just one dataset without making the meta-data first(not reccomended).
We aim to provide modularity to researchers therefore we users them to save and retrieve meta-data in whatever format they want.
More information about reproducing our experiments can be found in the README.md of the supplementary code repository.
To reproduce LOTUS for other tasks and dataset, users are simply required to change the datasets and algorithms in meta-data.
The approach works out of the box for other scenarios.
While reproducing the experiments, the results can differ due to stochasticity of few algorithms.
A A PPENDIX A. PERFORMANCES
Table 3 contains the performances of LOTUS and MetaOD on 42 datasets, we had to removedatasets from this experiment because MetaOD returned invalid models for these datasets(i.e.
models with invalid values) .
Scores are in bold where AUC of LOTUS > MetaOD or differ by less than a %.
The dataset names are as they were in ADBench (Han et al., 2022).
Table 4 reports the auc scores over datasets from ADBench.
The bold number shows scores where
LOTUS is better than all other estimators in PyOD.
[Table]
8 celeba 0.
9 census 0.
39 vertebral 0.
41 Waveform 0.
38 thyroid 0.
40 vowels 0.
3 backdoor 0.
32 shuttle 0.
31 satimage0.
26 optdigits 0.
1 ALOI 0.
35 SpamBase 0.
36 speech 0.
34 smtp 0.
22 magic.gamma 0.
23 mammography 0.
24 mnist 0.
20 letter 0.
30 satellite 0.
19 landsat 0.
37 Stamps 0.
18 Ionosphere 0.
21 Lymphography 0.
25 musk 0.
17 InternetAds 0.
16 http 1.
15 Hepatitis 0.
14 glass 0.
13 fraud 0.
11 donors 0.
12 fault 0.
2 annthyroid 0.
27 PageBlocks 0.
28 pendigits 0.
29 Pima 0.
10 cover 0.
A. BASLINES
The 8 baslines estimators and frameworks are listed below with brief description from PyOD’s
(Zhao et al., 2019) documentation for reference here:
MetaOD :
MetaOD is the first automated tool for outlier detection.
MetaOD use collaborative filtering, landmark and model based meta-features to recommend the model for given task.
IForest : IsolationForest ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature.
LOF :
The anomaly score of each sample is called Local Outlier Factor.
It measures the local deviation of density of a given sample with respect to its neighbors.
It is local in that the anomaly score depends on how isolated the object is with respect to the surrounding neighborhood.
More precisely, locality is given by k-nearest neighbors, whose distance is used to estimate the local density.
By comparing the local density of a sample to the local densities of its neighbors, one can identify samples that have a substantially lower density than their neighbors.
These are considered outliers.
ABOD :
For an observation, the variance of its weighted cosine scores to all neighbors could be viewed as the outlying score.
HBOS :
Histogram- based outlier detection assumes the feature independence and calculates the degree of outlier by building histograms.
KNN : kNN class for outlier detection.
For an observation, its distance to its kth nearest neighbor could be viewed as the outlying score.
COF : Connectivity-Based Outlier Factor uses the ratio of average chaining distance of data point and the average of average chaining distance of k nearest neighbor of the data point, as the outlier score for observations.
LODA : Lightweight on-line detector of anomalies detects anomalies in a dataset by computing the likelihood of data points using an ensemble of one-dimensional histograms.
OCSVM :
One class support vector machines unsupervised outlier Detection.
Estimate the support of a high-dimensional distribution.
A. LOTUS+GAMAOD SEARCH SPACE AND M ETA OD REPRODUCIBILITY
We implement the same searchspace as MetaOD’s github repository for a fair comparison 2 , MetaOD also uses all the existing datasets from ADbench.
We believe that we have fairly evaluated MetaOD against out baseline.
We believe that our Benchmark setting was more challenging than the one evaluated in Zhao et al.
(2021) where it take child and parent datasets.
A. A RCHITECTURE
An overview of GAMAOD system can be found in Figure 5.
GAMAOD is build on top of GAMA.
We replace search space from scikit-learn estimators to PyOD estimators.
For evaluation of the pipeline we avoid cross-validation and tune the models on the AUC score.
These models are then used in meta-data
A. E XPERIMENTAL I MPLEMENTATION
Implementation details: We use Independent Component Analysis(ICA) from scikit-learn as our transformation function ϕ .
We use OTT-JAX library (Cuturi et al., 2022) to implement Low Rank
Gromov Wassersstein distance.
For this experiment, we set the rank parameter of Low Rank Gromov
Wasserstein to 6.
The model selection phase of LOTUS in our experiments is as follows:
{+First the datasets are transformed via ICA and then converted into JAX pointclouds geometry objects 
+}
and then we turn these distributions into a quadratic regularized optimal transport problem.
We input this quadratic problem to our Gromov Wasserstein Low Rank solver which returns us the distance(cost) between two datasets.
When a new dataset is given to LOTUS, the pipeline corresponding to the dataset with the lowest distance(except the new dataset itself) is chosen from the optimal pipeline database.