U NSUPERVISED R EPRESENTATION L EARNING BY P REDICTING R ANDOM D ISTANCES
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks have gained tremendous success in a broad range of machine learning tasks due to its remarkable capability to learn semantic-rich features from high-dimensional data.
However, they often require large-scale labelled data to successfully learn such features, which signiﬁcantly hinders their adaption into unsupervised learning tasks, such as anomaly detection and clustering, and limits their applications into critical domains where obtaining massive labelled data is prohibitively expensive.
To enable downstream unsupervised learning on those domains, in this work we propose to learn features without using any labelled data by training neural networks to predict data distances in a randomly projected space.
Random mapping is a theoretical proven approach to obtain approximately preserved distances.
To well predict these random distances, the representation learner is optimised to learn genuine class structures that are implicitly embedded in the randomly projected space.
Experimental results on 19 real-world datasets show our learned representations substantially outperform state-of-the-art competing methods in both anomaly detection and clustering tasks.
I NTRODUCTION
Unsupervised representation learning aims at automatically extracting expressive feature representations from data without any manually labelled data.
Due to the remarkable capability to learn semantic-rich features, deep neural networks have been becoming one widely-used technique to empower a broad range of machine learning tasks.
One main issue with these deep learning techniques is that a massive amount of labelled data is typically required to successfully learn these expressive features.
As a result, their transformation power is largely reduced for tasks that are unsupervised in nature, such as anomaly detection and clustering.
This is also true to critical domains, such as healthcare and ﬁntech, where collecting massive labelled data is prohibitively expensive and/or is impossible to scale.
To bridge this gap, in this work we explore fully unsupervised representation learning techniques to enable downstream unsupervised learning methods on those critical domains.
In recent years, many unsupervised representation learning methods (Mikolov et al., 2013a; Le &
Mikolov, 2014; Misra et al., 2016; Lee et al., 2017; Gidaris et al., 2018) have been introduced, of which most are self-supervised approaches that formulate the problem as an annotation free pretext task.
These methods explore easily accessible information, such as temporal or spatial neighbourhood, to design a surrogate supervisory signal to empower the feature learning.
These methods have achieved signiﬁcantly improved feature representations of text/image/video data, but they are often inapplicable to tabular data since it does not contain the required temporal or spatial supervisory information.
We therefore focus on unsupervised representation learning of high-dimensional tabular data.
Although many traditional approaches, such as random projection (Li et al., 2006), principal component analysis (PCA) (Rahmani & Atia, 2017), manifold learning (Donoho & Grimes, 2003;
Hinton & Roweis, 2003) and autoencoder (Vincent et al., 2010), are readily available for handling those data, many of them (Donoho & Grimes, 2003; Hinton & Roweis, 2003; Rahmani & Atia, 2017) are often too computationally costly to scale up to large or high-dimensional data.
Approaches like random projection and autoencoder are very efﬁcient but they often fail to capture complex class structures due to its underlying data assumption or weak supervisory signal.
In this paper, we introduce a Random Distance Prediction (RDP) model which trains neural networks to predict data distances in a randomly projected space.
When the distance information captures in- trinsic class structure in the data, the representation learner is optimised to learn the class structure to minimise the prediction error.
Since distances are concentrated and become meaningless in high dimensional spaces (Beyer et al., 1999), we seek to obtain distances preserved in a projected space to be the supervisory signal.
Random mapping is a highly efﬁcient yet theoretical proven approach to obtain such approximately preserved distances.
Therefore, we leverage the distances in the randomly projected space to learn the desired features.
Intuitively, random mapping preserves rich local proximity information but may also keep misleading proximity when its underlying data distribution assumption is inexact; by minimising the random distance prediction error, RDP essentially leverages the preserved data proximity and the power of neural networks to learn globally consistent proximity and rectify the inconsistent proximity information, resulting in a substantially better representation space than the original space.
We show this simple random distance prediction enables us to achieve expressive representations with no manually labelled data.
In addition, some task-dependent auxiliary losses can be optionally added as a complementary supervisory source to the random distance prediction, so as to learn the feature representations that are more tailored for a speciﬁc downstream task.
In summary, this paper makes the following three main contributions.
• We propose a random distance prediction formulation, which is very simple yet offers a highly effective supervisory signal for learning expressive feature representations that optimise the distance preserving in random projection.
The learned features are sufﬁciently generic and work well in enabling different downstream learning tasks. • Our formulation is ﬂexible to incorporate task-dependent auxiliary losses that are complementary to random distance prediction to further enhance the learned features, i
e., features that are speciﬁcally optimised for a downstream task while at the same time preserving the generic proximity as much as possible. • As a result, we show that our instantiated model termed RDP enables substantially better performance than state-of-the-art competing methods in two key unsupervised tasks, anomaly detection and clustering, on 19 real-world high-dimensional tabular datasets.
R ANDOM D ISTANCE P REDICTION M ODEL 2. T HE P ROPOSED F ORMULATION AND T HE I NSTANTIATED M ODEL
We propose to learn representations by training neural networks to predict distances in a randomly projected space without manually labelled data.
The key intuition is that, given some distance information that faithfully encapsulates the underlying class structure in the data, the representation learner is forced to learn the class structure in order to yield distances that are as close as the given distances.
Our proposed framework is illustrated in Figure 1.
Speciﬁcally, given data points x i , x j ∈ R D , we ﬁrst feed them into a weight-shared Siamese-style neural network φ ( x ; Θ) .
φ : R D (cid:55)→ R M is a representation learner with the parameters Θ to map the data onto a M -dimensional new space.
Then we formulate the subsequent step as a distance prediction task and deﬁne a loss function as:
[Equation]
where η is an existing projection method and l is a function of the difference between its two inputs. Here one key ingredient is how to obtain trustworthy distances via η .
Also, to efﬁciently optimise the model, the distance derivation needs to be computationally efﬁcient.
In this work, we use the inner products in a randomly projected space as the source of distance/similarity since it is very efﬁcient and there is strong theoretical support of its capacity in preserving the genuine distance information.
Thus, our instantiated model RDP speciﬁes L rdp ( x i , x j ) as follows 1 :
[Equation]
where φ is implemented by multilayer perceptron for dealing with tabular data and η : R D (cid:55)→ R K is an off-the-shelf random data mapping function (see Sections 3.1 and 3.2 for detail). Despite its simplicity, this loss offers a powerful supervisory signal to learn semantic-rich feature representations that substantially optimise the underlying distance preserving in η (see Section 3.3 for detail). L rdp 𝜙 𝒙 𝒊 ;𝛩 Neural Network 𝜙 𝒙 𝒋 ;𝛩 Neural Network Shared weights 𝑥 𝑖1 𝑥 𝑖 𝑥 𝑖𝐷 …
𝒙 𝒊 𝑥 𝑗1 𝑥 𝑗 𝑥 𝑗𝐷 … 𝒙 𝒋
L aux
Random Distance 𝑙 𝜙 𝒙 𝒊 ;𝛩 ,𝜙 𝒙 𝒋 ;𝛩 , 𝜂 𝒙 𝒊 ,𝜂 𝒙 𝒋 2. F LEXIBILITY TO I NCORPORATE T ASK - DEPENDENT C OMPLEMENTARY A UXILIARY L OSS Minimising L rdp learns to preserve pairwise distances that are critical to different learning tasks.
Moreover, our formulation is ﬂexible to incorporate a task-dependent auxiliary loss L aux , such as reconstruction loss (Hinton & Salakhutdinov, 2006) for clustering or novelty loss (Burda et al., 2019) for anomaly detection, to complement the proximity information and enhance the feature learning.
For clustering, an auxiliary reconstruction loss is deﬁned as:
[Equation]
) where φ is an encoder and φ (cid:48) : R M (cid:55)→ R D is a decoder.
This loss may be optionally added into
RDP to better capture global feature representations.
Similarly, in anomaly detection a novelty loss may be optionally added, which is deﬁned as:
[Equation]
By using a ﬁxed η , minimising L adaux helps learn the frequency of underlying patterns in the data (Burda et al., 2019), which is an important complementary supervisory source for the sake of anomaly detection.
As a result, anomalies or novel points are expected to have substantially larger ( φ ( x ; Θ (cid:63) ) − η ( x )) 2 than normal points, so this value can be directly leveraged to detect anomalies.
Note since L adaux involves a mean squared error between two vectors, the dimension of the projected space resulted by φ and η is required to be equal in this case.
Therefore, when this loss is added into RDP, the M in φ and K in η need to be the same.
We do not have this constraint in other cases.
T HEORETICAL A NALYSIS OF RDP
This section shows the proximity information can be well approximated using inner products in two types of random projection spaces.
This is a key theoretical foundation to RDP. Also, to accurately predict these distances, RDP is forced to learn the genuine class structure in the data.
3. W HEN L INEAR P ROJECTION I S U SED
Random projection is a simple yet very effective linear feature mapping technique which has proven the capability of distance preservation.
Let X ⊂ R N × D be a set of N data points, random projection uses a random matrix A ⊂ R K × D to project the data onto a lower K -dimensional space by X (cid:48) =
A X (cid:124) .
The Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) guarantees the data points can be mapped to a randomly selected space of suitably lower dimension with the distances between the points are approximately preserved.
More speciﬁcally, let (cid:15) ∈ (0 , 12 ) and K = 20log n (cid:15).
There exists a linear mapping f : R D (cid:55)→ R K such that for all x i , x j ∈ X :
[Equation]
Furthermore, assume the entries of the matrix A are sampled independently from a Gaussian distribution N (0 , 1) .
Then, the norm of x ∈ R D can be preserved as:
[Equation]
Under such random projections, the norm preservation helps well preserve the inner products:
[Equation]
where ˆ x is a normalised x such that
|| ˆ x || ≤ 1 .
The proofs of Eqns.
(5), (6) and (7) can be found in (Vempala, 1998).
Eqn.
(7) states that the inner products in the randomly projected space can largely preserve the inner products in the original space, particularly when the projected dimension K is large.
3. W HEN N ON - LINEAR P ROJECTION I S U SED
Here we show that some non-linear random mapping methods are approximate to kernel functions which are a well-established approach to obtain reliable distance/similarity information.
The key to this approach is the kernel function k : X ×X (cid:55)→ R , which is deﬁned as k ( x i , x j ) =
(cid:104) ψ ( x i ) , ψ ( x j ) (cid:105) , where ψ is a feature mapping function but needs not to be explicitly deﬁned and (cid:104)· , ·(cid:105) denotes a suitable inner product.
A non-linear kernel function such as polynomial or radial basis function (RBF) kernel is typically used to project linear-inseparable data onto a linear-separable space.
The relation between non-linear random mapping and kernel methods is justiﬁed in (Rahimi &
Recht, 2008), which shows that an explicit randomised mapping function g : R D (cid:55)→ R K can be deﬁned to project the data points onto a low-dimensional Euclidean inner product space such that the inner products in the projected space approximate the kernel evaluation:
[Equation]
Let A be the mapping matrix.
Then to achieve the above approximation, A is required to be drawn from Fourier transform and shift-invariant functions such as cosine function are ﬁnally applied to
Ax to yield a real-valued output.
By transforming the two data points x i and x j in this manner, their inner product g ( x i ) · g ( x j ) is an unbiased estimator of k ( x i , x j ) .
3. L EARNING C LASS S TRUCTURE B Y R ANDOM D ISTANCE P REDICTION
Our model using only the random distances as the supervisory signal can be formulated as:
[Equation]
where y ij = η ( x i ) · η ( x j ) .
Let Y η ∈ R N ×
N
be the distance/similarity matrix of the N data points resulted by η .
Then to minimise the prediction error in Eqn.
(9), φ is optimised to learn the underlying class structure embedded in Y .
As shown in the properties in Eqns. (7) and (8), Y η can effectively preserve local proximity information when η is set to be either the random projectionbased f function or the kernel method-based g function. However, those proven η is often built upon some underlying data distribution assumption, e.g., Gaussian distribution in random projection or Gaussian RBF kernel, so the η -projected features can preserve misleading proximity when the distribution assumption is inexact. In this case, Y η is equivalent to the imperfect ground truth with partial noise.
Then optimisation with Eqn.
(9) is to leverage the power of neural networks to learn consistent local proximity information and rectify inconsistent proximity, resulting in a signiﬁcantly optimised distance preserving space. The resulting space conveys substantially richer semantics than the η projected space when Y η contains sufﬁcient genuine supervision information.
E XPERIMENTS
This section evaluates the learned representations through two typical unsupervised tasks: anomaly detection and clustering.
Some preliminary results of classiﬁcation can be found in Appendix H.
4. P ERFORMANCE E VALUATION IN A NOMALY D ETECTION 4.1. E XPERIMENTAL S ETTINGS
Our RDP model is compared with ﬁve state-of-the-art methods, including iForest (Liu et al., 2008), autoencoder (AE) (Hinton & Salakhutdinov, 2006), REPEN (Pang et al., 2018), DAGMM (Zong et al., 2018) and RND (Burda et al., 2019).
iForest and AE are two of the most popular baselines.
The other three methods learn representations speciﬁcally for anomaly detection.
As shown in Table 1, the comparison is performed on 14 publicly available datasets of various domains, including network intrusion, credit card fraud detection, disease detection and bank campaigning.
Many of the datasets contain real anomalies, including DDoS , Donors , Backdoor , Creditcard , Lung , Probe and U2R .
Following (Liu et al., 2008; Pang et al., 2018; Zong et al., 2018), the rare class(es) is treated as anomalies in the other datasets to create semantically real anomalies.
The
Area Under Receiver Operating Characteristic Curve (AUC-ROC) and the Area Under PrecisionRecall Curve (AUC-PR) are used as our performance metrics.
Larger AUC-ROC/AUC-PR indicates better performance.
The reported performance is averaged over 10 independent runs.
[Table]
Our RDP model uses the optional novelty loss for anomaly detection task by default.
Similar to
RND, given a data point x , its anomaly score in RDP is deﬁned as the mean squared error between the two projections resulted by φ ( x ; Θ (cid:63) ) and η ( x ) .
Also, a boosting process is used to ﬁlter out 5% likely anomalies per iteration to iteratively improve the modelling of RDP.
This is because the modelling is otherwise largely biased when anomalies are presented.
In the ablation study in Section 4.1.3, we will show the contribution of all these components.
4.1. C OMPARISON TO THE STATE - OF - THE - ART COMPETING METHODS
The AUC-ROC and AUC-PR results are respectively shown in Tables 1 and 2.
RDP outperforms all the ﬁve competing methods in both of AUC-ROC and AUC-PR in at least 12 out of 14 datasets.
This improvement is statistically signiﬁcant at the 95% conﬁdence level according to the two-tailed sign test (Demˇsar, 2006).
Remarkably, RDP obtains more than 10% AUC-ROC/AUC-PR improvement over the best competing method on six datasets, including Donors , Ad , Bank , Celeba , Lung and
U2R .
RDP can be thought as a high-level synthesis of REPEN and RND, because REPEN leverages a pairwise distance-based ranking loss to learn representations for anomaly detection while RND is built using L ad aux . In nearly all the datasets, RDP well leverages both L rdp and L ad aux to achieve signiﬁcant improvement over both REPEN and RND.
In very limited cases, such as on datasets
Backdoor and Census where RND performs very well while REPEN performs less effectively, RDP is slightly downgraded due to the use of L rdp .
In the opposite case, such as Probe , on which REPEN performs much better than RND, the use of L ad aux may drag down the performance of RDP a bit.
[Table]
4.1. A BLATION S TUDY
This section examines the contribution of L rdp , L ad aux and the boosting process to the performance of RDP.
The experimental results in AUC-ROC are given in Table
[Table]
where RDP \ X means the RDP variant that removes the ‘X’ module from RDP.
In the last two columns, Org SS indicates that we directly use the distance information calculated in the original space as the supervisory signal, while
SRP SS indicates that we use SRP to obtain the distances as the supervisory signal.
It is clear that the full RDP model is the best performer.
Using the L rdp loss only,
i
[Table]
e., RDP \ L adaux , can achieve performance substantially better than, or comparably well to, the ﬁve competing methods in Table 1.
This is mainly because the L rdp loss alone can effectively force our representation learner to learn the underlying class structure on most datasets so as to minimise its prediction error.
The use of
L adaux and boosting process well complement the L rdp loss on the other datasets.
In terms of supervisory source, RDP and SRP SS perform substantially better than Org SS on most datasets.
This is because the distances in both the non-linear random projection in RDP and the linear projection in SRP SS well preserve the distance information, enabling RDP to effectively learn much more faithful class structure than that working on the original space.
[Table]
4. P ERFORMANCE E VALUATION IN C LUSTERING 4.2. E XPERIMENTAL S ETTINGS
For clustering, RDP is compared with four state-of-the-art unsupervised representation learning methods in four different areas, including HLLE (Donoho & Grimes, 2003) in manifold learning,
Sparse Random Projection (SRP) (Li et al., 2006) in random projection, autoencoder (AE) (Hinton &
Salakhutdinov, 2006) in data reconstruction-based neural network methods and Coherence Pursuit (COP) (Rahmani & Atia, 2017) in robust PCA.
These representation learning methods are ﬁrst used to yield the new representations, and K-means (Hartigan & Wong, 1979) is then applied to the representations to perform clustering.
Two widely-used clustering performance metrics, Normalised
Mutual Info (NMI) score and F-score, are used.
Larger NMI or F-score indicates better performance.
The clustering performance in the original feature space, denoted as Org, is used as a baseline.
As shown in Table 4, ﬁve high-dimensional real-world datasets are used.
Some of the datasets are image/text data.
Since here we focus on the performance on tabular data, they are converted into tabular data using simple methods, i.e., by treating each pixel as a feature unit for image data or using bag-of-words representation for text data 2 .
The reported NMI score and F-score are averaged over 30 times to address the randomisation issue in K-means clustering.
In this section RDP adds the reconstruction loss L clu aux by default, but RDP also works very well without the use of L clu aux .
4.2. C OMPARISON TO THE - STATE - OF - THE - ART COMPETING METHODS
Table 4 shows the NMI and F-score performance of K-means clustering.
Our method RDP enables K-means to achieve the best performance on three datasets and ranks second in the other two datasets.
RDP-enabled clustering performs substantially and consistently better than that based on AE in terms of both NMI and F-score.
This demonstrates that the random distance loss enables RDP to effectively capture some class structure in the data which cannot be captured by using the reconstruction loss.
RDP also consistently outperforms the random projection method, SRP, and the robust PCA method, COP.
It is interesting that K-means clustering performs best in the original space on Sector .
This may be due to that this data contains many relevant features, resulting in no obvious curse of dimensionality issue.
Olivetti may contain complex manifolds which require extensive neighbourhood information to ﬁnd them, so only HLLE can achieve this goal in such cases.
Nevertheless, RDP performs much more stably than HLLE across the ﬁve datasets. [Table]
[Table]
4.2. A BLATION S TUDY
Similar to anomaly detection [Table] this section examines the contribution of the two loss functions L rdp and L cluaux to the performance of RDP, as well as the impact of different supervisory sources on the performance.
The F-score results of this experiment are shown in Table [Table] in which the notations have exactly the same meaning as in Table 3.
The full RDP model that uses both L rdp and L cluaux performs more favourably than its two variants, RDP \ L rdp and RDP \ L cluaux , but it is clear that using
L rdp only performs very comparably to the full RDP.
However, using L cluaux only may result in large performance drops in some datasets, such as R8 , 20news and Olivetti .
This indicates L rdp is a more important loss function to the overall performance of the full RDP model.
In terms of supervisory source, distances obtained by the non-linear random projection in RDP are much more effective than the two other sources on some datasets such as Olivetti and RCV1 .
Three different supervisory sources are very comparable on the other three datasets.
R ELATED W ORK
Self-supervised Learning.
Self-supervised learning has been recently emerging as one of the most popular and effective approaches for representation learning.
Many of the self-supervised methods learn high-level representations by predicting some sort of ‘context’ information, such as spatial or temporal neighbourhood information.
For example, the popular distributed representation learning techniques in NLP, such as CBOW/skip-gram (Mikolov et al., 2013a) and phrase/sentence embeddings in (Mikolov et al., 2013b; Le & Mikolov, 2014; Hill et al., 2016), learn the representations by predicting the text pieces (e.g., words/phrases/sentences) using its surrounding pieces as the context.
In image processing, the pretext task can be the prediction of a patch of missing pixels (Pathak et al., 2016; Zhang et al., 2017) or the relative position of two patches (Doersch et al., 2015).
Also, a number of studies (Goroshin et al., 2015; Misra et al., 2016; Lee et al., 2017; Oord et al., 2018) explore temporal contexts to learn representations from video data, e.g., by learning the temporal order of sequential frames.
Some other methods (Agrawal et al., 2015; Zhou et al., 2017; Gidaris et al., 2018) are built upon a discriminative framework which aims at discriminating the images before and after some transformation, e.g., ego motion in video data (Agrawal et al., 2015;
Zhou et al., 2017) and rotation of images (Gidaris et al., 2018).
There have also been popular to use generative adversarial networks (GANs) to learn features (Radford et al., 2015; Chen et al., 2016).
The above methods have demonstrated powerful capability to learn semantic representations.
However, most of them use the supervisory signals available in image/video data only, which limits their application into other types of data, such as traditional tabular data.
Although our method may also work on image/video data, we focus on handling high-dimensional tabular data to bridge this gap.
Other Approaches.
There have been several well-established unsupervised representation learning approaches for handling tabular data, such as random projection (Arriaga & Vempala, 1999; Bingham & Mannila, 2001; Li et al., 2006), PCA (Wold et al., 1987;
Sch ¨ olkopf et al., 1997; Rahmani & Atia, 2017), manifold learning (Roweis & Saul, 2000; Donoho & Grimes, 2003; Hinton & Roweis, 2003; McInnes et al., 2018) and autoencoder (Hinton & Salakhutdinov, 2006; Vincent et al., 2010).
One notorious issue of PCA or manifold learning approaches is their prohibitive computational cost in dealing with large-scale high-dimensional data due to the costly neighbourhood search and/or eigen decomposition.
Random projection is a computationally efﬁcient approach, supported by proven distance preservation theories such as the Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984).
We show that the preserved distances by random projection can be harvested to effectively supervise the representation learning.
Autoencoder networks are another widely-used efﬁcient feature learning approach which learns low-dimensional representations by minimising reconstruction errors.
One main issue with autoencoders is that they focus on preserving global information only, which may result in loss of local structure information.
Some representation learning methods are speciﬁcally designed for anomaly detection (Pang et al., 2018; Zong et al., 2018; Burda et al., 2019).
By contrast, we aim at generic representations learning while being ﬂexible to incorporate optionally task-dependent losses to learn task-speciﬁc semantic-rich representations.
C ONCLUSION
We introduce a novel Random Distance Prediction (RDP) model which learns features in a fully unsupervised fashion by predicting data distances in a randomly projected space.
The key insight is that random mapping is a theoretical proven approach to obtain approximately preserved distances, and to well predict these random distances, the representation learner is optimised to learn consistent preserved proximity information while at the same time rectifying inconsistent proximity, resulting in representations with optimised distance preserving.
Our idea is justiﬁed by thorough experiments in two unsupervised tasks, anomaly detection and clustering, which show RDP-enabled anomaly detectors and clustering substantially outperform their counterparts on 19 real-world datasets.
We plan to extend RDP to other types of data to broaden its application scenarios.
A I MPLEMENTATION D ETAILS
RDP-enabled Anomaly Detection.
The RDP consists of one fully connected layer with 50 hidden units, followed by a leaky-ReLU layer.
It is trained using Stochastic Gradient Descent (SGD) as its optimiser for 200 epochs, with 192 samples per batch.
The learning rate is ﬁxed to
We repeated the boosting process 30 times to obtain statistically stable results.
In order to have fair comparisons, we also adapt the competing methods AE, REPEN, DAGMM and RND into ensemble methods and perform the experiments using an ensemble size of 30.
RDP-enabled Clustering.
RDP uses a similar network architecture and optimisation settings as the one used in anomaly detection, i e., the network consists of one fully connected layer, followed by a leaky-ReLU layer, which is optimised by SGD with 192 samples per batch and 0.1 learning rate.
Compared to anomaly detection, more semantic information is required for clustering algorithms to work well, so the network consists of 1,024 hidden units and is trained for 1,000 epochs.
Clustering is a signiﬁcant yet common analysis method, which aims at grouping samples close to each other into the same clusters and separating far away data points into different clusters.
Compared to anomaly detection that often requires pattern frequency information, clustering has a higher requirement of the representation expressiveness.
Therefore, if the representative ability of a model is strong enough, it should also be able to learn representations that enable clustering to work well on the projected space.
Note that the representation dimension M in the φ function and the projection dimension K in the η function are set to be the same to alleviate parameter tuning.
This means that M = K = 50 is used in anomaly detection and M =
K
= 1024 is used in clustering.
We have also tried deeper network structures, but they worked less effectively than the shallow networks in both anomaly detection and clustering.
This may be because the supervisory signal is not strong enough to effectively learn deeper representations.
We show in Appendix E that RDP performs stably w.r.t.
a range of representation dimensions in both anomaly detection and clustering tasks.
The runtime of RDP at the testing stage is provided in Appendix F with that of the competing methods as baselines.
For both anomaly detection and clustering tasks, RDP achieves very comparable time complexity to the most efﬁcient competing methods (see Tables 10 and 11 in Appendix F for detail).
B D ATASETS
The statistics and the accessible links of the datasets used in the anomaly detection and clustering tasks are respectively presented in Tables 6 and 7.
DDoS is a dataset containing DDoS attacks and normal network ﬂows.
Donors is from KDD Cup 2014, which is used for detecting a very small number of outstanding donors projects.
Backdoor contains backdoor network attacks derived from the UNSW-NB15 dataset.
Creditcard is a credit card fraud detection dataset.
Lung contains data records of lung cancer patients and normal patients.
Probe and U2R are derived from KDD
Cup 99, in which probing and user-to-root attacks are respectively used as anomalies against the normal network ﬂows.
The above datasets contain real anomalies.
Following (Liu et al., 2008;
Pang et al., 2018; Zong et al., 2018), the other anomaly detection datasets are transformed from classiﬁcation datasets by using the rare class(es) as the anomaly class, which generates semantically real anomalies. [Table] terms. [Table]
[Table]
https://
[Table]
.ics.uci.edu/ml/datasets/ [Table] [Table] % http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html
R8 , 20news , Sector and RCV1 are widely used text classiﬁcation benchmark datasets.
Olivetti is a widely-used face recognition dataset.
[Table]
C AUC-PR P ERFORMANCE OF A BLATION S TUDY IN A NOMALY DETECTION
The experimental results of AUC-PR performance of RDP and its variants in the anomaly detection task are shown in Table 8.
Similar to the results shown in Table 3, using the L rdp loss only, our proposed RDP model can achieve substantially better performance over its counterparts.
By removing the L rdp loss, the performance of RDP drops signiﬁcantly in 11 out of 14 datasets.
This demonstrates that the L rdp loss is heavily harvested by our RDP model to learn high-quality representations from random distances.
Removing L adaux from RDP also results in substantial loss of
AUC-PR in many datasets.
This indicates both the random distance prediction loss L rdp and the task-dependent loss L adaux are critical to RDP.
The boosting process is also important, but is not as critical as the two losses.
Consistent with the observations derived from Table 3, distances calculated in non-linear and linear random mapping spaces are more effective supervisory sources than that in the original space.
D NMI P ERFORMANCE OF A BLATION S TUDY IN C LUSTERING
Table 9 shows the NMI performance of RDP and its variants in the clustering task.
It is clear that our RDP model with the L rdp loss is able to achieve NMI performance that is comparably well to the full RDP model, which is consistent to the observations in Table 5.
Without using the L rdp loss, the performance of the RDP model has some large drops on nearly all the datasets.
This reinforces the crucial importance of L rdp to RDP, which also justiﬁes that using L rdp alone RDP can learn expressive representations.
Similar to the results in Table 5, RDP is generally more reliable supervisory sources than Org SS and SRP SS in this set of results. E S ENSITIVITY W . R . T . THE D IMENSIONALITY OF R EPRESENTATION S PACE
This section presents the performance of RDP using different representation dimensions in its feature learning layer.
The sensitivity test is performed for both anomaly detection and clustering tasks.
[Table] [Table] E. S ENSITIVITY T EST IN A NOMALY D ETECTION
Figures 2 and 3 respectively show the AUC-ROC and AUC-PR performance of RDP using different representation dimensions on all the 14 anomaly detection datasets used in this work.
It is clear from both performance measures that RDP generally performs stably w.r.t.
the use of different representation dimensions on diverse datasets.
This demonstrates the general stability of our RDP method on different application domains.
On the other hand, the ﬂat trends also indicate that, as an unsupervised learning source, the random distance cannot provide sufﬁcient supervision information to learn richer and more complex representations in a higher-dimensional space.
This also explains the performance on quite a few datasets where the performance of RDP decreases when increasing the representation dimension.
In general, the representation dimension 50 is recommended for RDP to achieve effective anomaly detection on datasets from different domains.
E. S ENSITIVITY T EST IN C LUSTERING
Figure 4 presents the NMI and F-score performance of RDP-enabled K-means clustering using different representation dimensions on all the ﬁve datasets in the clustering task.
Similar to the sensitivity test results in the anomaly detection task, on all the ﬁve datasets, K-means clustering performs stably in the representation space resulted by RDP with different representation dimensions.
The clustering performance may drop a bit when the representation dimension is relatively low, e.g., 512.
Increasing the representation to 1,280 may help RDP gain better representation power in some datasets but is not a consistently better choice.
Thus, the representation dimension 1,024 is generally recommended for clustering.
Recall that the required representation dimension in clustering is normally signiﬁcantly higher than that in anomaly detection, because clustering generally requires signiﬁcantly more information to perform well than anomaly detection.
F C OMPUTATIONAL E FFICIENCY
The runtime of RDP is compared with its competing methods in both anomaly detection and clustering tasks.
Since training time can vary signiﬁcantly using different training strategies in deep learning-based methods, it is difﬁcult to have a fair comparison of the training time.
Moreover, the
[Figure]
[Figure]
models can often be trained ofﬂine.
Thus, we focus on comparing the runtime at the testing stage.
All the runtime experiments below were done on a computing server node equipped with 32 Intel
Xeon E5-2680 CPUs (2.70GHz) and 128GB Random Access Memory.
F. T ESTING R UNTIME IN A NOMALY D ETECTION
The testing runtime in seconds of RDP and its ﬁve competing anomaly detection methods onanomaly detection datasets are provided in Table 10.
Since most of the methods integrate representation learning and anomaly detection into a single framework, the runtime includes the execution time of feature learning and anomaly detection for all six methods.
In general, on most large datasets,
RDP runs comparably fast to the most efﬁcient methods iForest and RND, and is faster the two recently proposed deep methods REPEN and DAGMM.
Particularly, RDP runs faster than REPEN and DAGMM by a factor of around ﬁve on high-dimensional and large-scale datasets like Donors and Census.
RDP is slower than the competing methods in processing small datasets.
This is mainly
[Figure]
o r
[Figure]
P e
r
f
o r
[Figure]
because RDP has a base runtime of its boosting process.
Therefore, the runtime of RDP seems to be almost constant across the datasets.
This is a very desired property for handling high-dimensional and large-scale datasets.
[Table]
F. T ESTING R UNTIME IN C LUSTERING
Table 11 shows the testing runtime of RDP and its four competing methods in enabling clustering on ﬁve datasets.
Since exactly the same K-means clustering is used on the features in all the ﬁve cases, we exclude the runtime of the K-means clustering for more straightforward comparison.
The results show that RDP runs comparably fast to the very efﬁcient methods SRP and AE since they do not involve complex computation at the testing stage; RDP runs about ﬁve orders of magnitude faster than HLLE since HLLE takes a huge amount of time in its nearest neighbours searching.
Note that ‘Org’ indicates the clustering performed on the original space, so it involves no feature learning and does not take any time.
[Table]
G C OMPARISON TO S TATE - OF - THE - ART R EPRESENTATION L EARNING M ETHODS FOR R AW T EXT AND I MAGE D ATA
Since RDP relies on distance information as its supervisory signal, one interesting question is that, can RDP still work when the presented data is raw data in a non-Euclidean space, such as raw text and image data?
One simple and straightforward way to enable RDP to handle those raw data is, as what we did on the text and image data used in the evaluation of clustering, to ﬁrst convert the raw texts/images into feature vectors using commonly-used methods, e.g., TF-IDF (Aizawa, 2003) for text data and treating each pixel as a feature unit for image data, and then perform RDP on these vector spaces.
A further question is that, do we need RDP in handling those data since there are now a large number of advanced representation learning methods that are speciﬁcally designed for raw text/image datasets?
Or, how is the performance of RDP compared to those advanced representation learning methods for raw text/image datasets?
This section provides some preliminary results in the clustering task for answering these questions.
G. O N R AW T EXT D ATA
On the raw text datasets R8 and 20news, we ﬁrst compare RDP with the advanced document representation method Doc2Vec 3 as in (Le & Mikolov, 2014).
Recall that, for RDP, we ﬁrst use the bag-of-words model and document frequency information (e.g., TF-IDF) to simply convert documents into high-dimensional feature vectors and then perform RDP using the feature vectors.
Doc2Vec leverages the idea of distributed representations to directly learn representations of documents.
We further derive a variant of RDP, namely Doc2Vec+RDP, which performs RDP on the Doc2Vec projected representation space rather than the bag-of-words vector space.
All RDP,
Doc2Vec and Doc2Vec+RDP project data onto a 1,024-dimensional space for the subsequent learning tasks.
Note that, for the method Doc2Vec+RDP, to better examine the capability of RDP in exploiting the Doc2Vec projected space, we ﬁrst use Doc2Vec project raw text data onto a higherdimensional space (5,120 dimensions for R8 and 10,240 dimensions for 20news), and RDP further learns a 1,024-dimensional space from this higher-dimensional space.
The comparison results are shown in Table 12.
Two interesting observations can be seen.
First,
RDP can signiﬁcantly outperform Doc2Vec on R8 or performs comparably well on 20news .
This may be due to the fact that the local proximity information learned in RDP is critical to clustering; although the word prediction approach in Doc2Vec helps learn semantic-rich representations for words/sentences/paragraphs, the pairwise document distances may be less effective than RDP since Doc2Vec is not like RDP that is designed to optimise this proximity information.
Second,
Doc2Vec+RDP can achieve substantially better performance than Doc2Vec, especially on the dataset 20news where Doc2Vec+RDP achieves a NMI score of 0.198 while that of Doc2Vec is only 0.084.
This may be because, as discussed in Section 3.3, RDP is equivalent to learn an optimised feature space out of its input space (Doc2Vec projected feature space in this case) using imperfect supervision information.
When there is sufﬁcient accurate supervision information, RDP can learn a substantially better feature space than its input space.
This is also consistent with the results in Table 4, in which clustering based on the RDP projected space also performs substantially better than that working in the original space ‘Org’.
[Table]
G. O N R AW I MAGE D ATA
On the raw image dataset Olivetti , we compare RDP with the advanced representation learning method for raw images, RotNet (Gidaris et al., 2018).
RDP uses each image pixel as a feature unit and performs on a 64 × 64 vector space.
RotNet directly learns representations of images by predicting whether a given image is rotated or not.
Similar to the experiments on raw text data, we also evaluate the performance of RDP working on the RotNet projected space, i.e., RotNet+RDP.
All RDP, RotNet and RotNet+RDP ﬁrst learn a 1,024 representation space, and then K-means is applied to the learned space to perform clustering.
In the case of RotNet+RDP, the raw image data is ﬁrst projected onto a 2,048-dimensional space, and then RDP is applied to this higher-dimensional space to learn a 1,024-dimensional representation space.
We use the implementation of RotNet released by its authors 4 .
Note that the original RotNet is applied to large image datasets and has a deep network architecture, involving four convolutional blocks with three convolutional layers for each block.
We found directly using the original architecture is too deep for Olivetti and performs ineffectively as the data contains only 400 image samples.
Therefore, we simplify the architecture of RotNet and derive four variants of RotNet, including
RotNet 4 × 2 , RotNet 4 × 1 , RotNet 3 × 1 and RotNet 2 × 1 .
Here RotNet a × b represents RotNet with a con- volutional blocks and b convolutional layers for each block.
Note that RotNet 2 × 1 is the simplest variant we can derive that works effectively.
We evaluate the original RotNet, its four variants and the combination of these ﬁve RotNets and RDP.
[Table]
The evaluation results are presented in Table 13.
Impressively, RDP can signiﬁcantly outperform
RotNet and all its four variants on Olivetti .
It is interesting
that Org
(i
[Table] e., performing K-means clustering on the original 64 × 64 vector space) also obtains a similar superiority over the RotNet family. This may be because Olivetti is too small to provide sufﬁcient training samples for RotNet and its variants to learn its underlying semantic abstractions. This conjecture can also explain the increasing performance of RotNet variants with decreasing complexity of the RotNet architecture.
Similar to the results on the raw text data
[Table]
applying RDP on the RotNet projected spaces can also learn substantially more expressive representations than the representations yielded by RotNet and its variants, especially when the RotNet methods work well, such as the two cases: RotNet 3 × 1 vs.
RotNet 3 × 1 +RDP and RotNet 2 × 1 vs. RotNet 2 × 1 +RDP. H P ERFORMANCE E VALUATION IN C LASSIFICATION We also performed some preliminary evaluation of the learned representations in classiﬁcation tasks using a feed-forward three-layer neural network model as the classiﬁer. We used the same datasets as in the clustering task.
Speciﬁcally, the representation learning model ﬁrst outputs the new representations of the input data, and then the classiﬁer performs classiﬁcation on the learned representations.
RDP is compared with the same competing methods HLLE, SRP, AE and COP as in clustering.
F-score is used as the performance evaluation metric here.
The results are shown in Table 14.
Similar to the performance in clustering and anomaly detection, our model using only the random distance prediction loss L rdp , i.e., RDP \ L cluaux , performs very favourably and stably on all the ﬁve datasets.
The incorporation of \ L cluaux into the model, i [Table] e., RDP, helps gain some extra performance improvement on datasets like 20news , but it may also slightly downgrade the performance on other datasets.
An extra hyperparameter may be added to control the importance of these two losses.
[Table]