S 3 ADN ET : S EQUENTIAL A NOMALY D ETECTION WITH P ESSIMISTIC C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Anomalies are commonly found in sequential data generated by real-world applications, such as cyberattacks in network trafﬁc, human activity changes in wearable sensors.
Thanks to the development of computing technology, many impressive results have been obtained from deep learning-based anomaly detection approaches in recent years.
This paper proposes a simple neural network framework for detecting anomalies on sequential data, called S elf- S upervised S equential
A nomaly D etection N etwork (S 3 ADNet).
S 3 ADNet ﬁrst extracts the representations from each data point by performing feature augmentation for contrastive learning; then captures the contextual information from the sequential data points for estimating anomaly probabilities by optimizing the context-adaptive objective.
Here, we design a novel loss function based on a pessimistic policy, considering that only anomalies can affect the contextual relationships in sequences.
Our proposed method outperformed other state-of-the-art approaches on the benchmark datasets by F1-score with a more straightforward architecture.
I NTRODUCTION
Sequential anomaly detection (SAD), a subtask of anomaly detection (AD), focuses on detecting outliers or change points for sequential data.
Sequential data are produced at a sub-millisecond rate by multifarious real-world applications, usually being of high dimensionality and high noise.
Besides the complex nature of data, the sequential data points can have shifting distributions and relationships in different contexts, resulting in the concept drift problem (ˇZliobait˙e, 2010; Gama et al., 2014).
Because anomalies can imply certain occurrences that endanger public safety, life, and property, such as intrusions in network connections (Jyothsna et al., 2011; Samrin & Vasumathi, 2017; Moustafa et al., 2019), activity changes and health irregularities in human sensing (Nweke et al., 2018; 2019;
Serhani et al., 2020), abnormal events in video surveillance systems (Popoola &
Wang, 2012; Mabrouk & Zagrouba, 2018), and risky signals in space technology (Hundman et al., 2018; Tariq et al., 2019; Yairi et al., 2017; Shin et al., 2020), SAD is a challenging but crucial task (Pang et al., 2021; Ruff et al., 2021).
Learning meaningful representations from the data and capturing the contextual relationships is the key to solving SAD tasks.
Generally, AD methods are in the unsupervised scheme because anomalies are considered to be much less than normal instances and hard to model.
The approaches that train the models exclusively on the normal samples to identify the anomalies are one-class classiﬁcation methods, such as one-class support vector machine (OC-SVM)
(Sch¨olkopf et al., 1999).
Since deep neural networks are expert at extracting representative features from high-dimensional and high-noise data, more and more methods leverage deep learning to handle AD tasks.
Especially, autoencoders (AEs) (Hinton & Zemel, 1994; Kingma & Welling, 2013) and generative adversarial networks (GANs) (Goodfellow et al., 2020) are frequently used as they are suitable for unsupervised learning, such as deep autoencoding Gaussian mixture model (DAGMM)
(Zong et al., 2018), and AnoGAN (Schlegl et al., 2017).
However, these methods could be weak in capturing the contextual information from sequences, so that some other approaches introduce recurrent neural networks (RNNs) (Rumelhart et al., 1986) to learn the sequential relationships.
For example, the multi-scale convolutional recurrent encoder-decoder (MSCRED) (Zhang et al., 2019), OmniAnomaly (Su et al., 2019), and the temporal hierarchical one-class network (THON) (Shen et al., 2020) apply RNNs for time-series data modeling.
Nevertheless, most of these methods still require picking out the normal data from the mess to train the one-class classiﬁers or generative models in practice.
This paper proposes a self-supervised sequential anomaly detection network (S 3 ADNet) to learn the representations of sequential data points and extract the contextual relationships for anomaly detection.
Inspired by the simple framework for contrastive learning of visual representations (SimCLR) (Chen et al., 2020), we make the network generate two similar embeddings for input data points with feature augmentation, and the feature extraction layers capture the essential identities of the data points by minimizing the contrastive loss.
By optimizing the contextual contrastive loss based on a pessimistic assumption system, the multi-conceptual context (MCC) layer estimates anomaly probabilities with the sequential relationships modeled by the learned representations for each data point.
We can use arbitrary parameter-shared neural networks for the representative feature extraction.
Additionally, we employ the MCC layer rather than RNN structures for a more intuitive context learning.
The proposed model does not require any label information or even selecting the normal data for training one-class classiﬁers, yet makes the data points compare with each other to recognize anomalies.
In the experiment, we compared our model’s estimation results to the state-of-the-art approaches using a widely used network trafﬁc dataset and a human activity sensing dataset.
Our proposed S 3 ADNet obtained competitive F1-score results to those methods with a simpler network architecture.
R ELATED W ORK
In this section, we review existing research endeavors related to our work.
We ﬁrst overview the approaches for AD and SAD tasks.
Then, we outline recent contrastive learning methods for representation learning.
2. A NOMALY D ETECTION
In a narrow sense, AD indicates outlier detection (OD), where the target is the data points with extraordinary properties from most of the rest.
Other than outliers, people are more concerned with change point detection (CPD) on sequential data since the abrupt characteristic shifts can intimate valuable and crucial information for the temporal trends.
OD methods can be grouped into three categories based on the main principle of modeling: 1) boundary-based, 2) density-based, and 3) reconstruction-based approaches.
Boundary-based methods use certain distance indexes in the model optimization or the anomaly score calculation.
OCSVM is a typical boundary-based method that learns a kernel-based boundary surrounding normal data points on a hyperplane (Sch¨olkopf et al., 1999).
Isolation Forest (IF), another widely used approach, constructs trees by randomly splitting branches throughout randomly selected features and considers that the average feature path length of an anomaly to the root is larger than that of a normal instance (Liu et al., 2008).
The main idea of classic density-based methods is performing probability density estimation to maximize the likelihood for normal samples, such as using Gaussian mixture model (GMM) for parametric estimation or kernel density estimation (KDE) for nonparametric estimation (Laxhammar et al., 2009; Kim & Scott, 2012).
The online algorithm employed a discounting strategy to update the density for sequences (Yamanishi et al., 2004).
With the development of deep learning, deep structured energy-based models (DSEBMs) were proposed to model the distribution of normal data points by using deep architectures (Zhai et al., 2016).
Another deep method,
DAGMM, applies autoencoder to learning data’s representations and predict the likelihood based on
GMM (Zong et al., 2018).
Reconstruction-based methods aim to well restore data from compressed feature space for normal data points but not anomalies.
Principal component analysis (PCA) and its variants are traditional ways for data reconstruction (Hawkins, 1974; Sch¨olkopf et al., 1997; Tharrault et al., 2008), while variational AE is of deep style without handcrafted features for the sake (An & Cho, 2015).
Methods using GANs can be considered to be hybrid approaches, for they model the distribution by regenerating data points in the zero-sum-game scheme, such as AnoGAN (Schlegl et al., 2017), adversarially learned anomaly detection (ADLD)
(Zenati et al., 2018), Mahalanobis distance-based adversarial network (MDAN)
(Hou et al., 2020).
For CPD, there is a similar taxonomy to OD.
Fast low-cost online semantic segmentation (FLOSS) calculates the contextual boundaries based on the shapes of sequential patterns to eject change points (Gharghabi et al., 2019).
The autoregressive model updates the parametric probability density function across sequential data points (Yamanishi & Takeuchi, 2002).
Relative unconstrained leastsquares importance ﬁtting (RuLSIF)
identiﬁes whether two contiguous subsequences are from different distributions using the nonparametric divergence estimation (Liu et al., 2013).
KL-CPD gives the distribution discrepancy by using a learnable deep kernel function and reconstructing samples from two sequential segments with RNNs (Chang et al., 2019).
Entropy and shape aware time-series segmentation (ESPRESSO) is a hybrid approach that combines the temporal density and distance to detect changes and obtained better performance than employing one principle (Deldari et al., 2020).
Another self-supervised contrastive learning method TS-CP 2 takes the subsequences which are not contiguous with the current subsequence as the negative samples for the training and then predicts whether the two subsequences are from the same distribution (Deldari et al., 2021).
TS-CPemploys temporal convolutional networks (TCNs) (Bai et al., 2018) to learn contextual representations, while ours introduces the MCC layer to capture the contextual information and allows using arbitrary feature extraction networks for each data point.
2. C ONTRASTIVE L EARNING
Recently, contrastive learning (CL) has grabbed much attention due to the beneﬁt of avoiding the high cost of labeling large-scale datasets for deep representation learning in an unsupervised/selfsupervised manner (Jaiswal et al., 2021).
CL has been used for various data formats, including images (Chen et al., 2020;
He et al., 2020), time series (van den Oord et al., 2018; Franceschi et al., 2019), videos (Sermanet et al., 2018; Qian et al., 2021), texts (van den Oord et al., 2018; Gao et al., 2021), and graphs (You et al., 2020; Zhu et al., 2020).
The basic strategy of CL is to train a model that makes the feature distances between similar instances closer but the ones between diverse instances as far as possible.
For this sake, CL methods design pretext tasks to optimize the distances and use contrastive loss or triplet loss as the learning objective (Chen et al., 2020).
Recently, data augmentation has been commonly introduced into the pretext tasks to help the feature extraction.
While image data are relatively easy to augment by using conventional computer vision techniques (Shorten & Khoshgoftaar, 2019), there are several word-level (Wei & Zou, 2019) and sentence-level (Kobayashi, 2018) approaches for textual data augmentation.
For other data formats, some domain-agnostic methods were proposed, such as adding Gaussian noise (GN) to embeddings (DeVries & Taylor, 2019) or dropout noise (DN) in representation layers (Gao et al., 2021; Liang et al., 2021).
GN and DN are not only easy to implement but also have low computational costs for CL tasks.
In SimCLR, given a cluster of augmentation Λ and an input x , two augmentation operator λ a ∼ Λ and λ b ∼ Λ generate two similar instances x a =
λ a ( x ) and x b = λ b ( x ) .
Then, the feature extraction network f encodes x a and x b into two representations h
a = f ( x a ) and h b = f ( x b ) .
Next, the head network g projects h a and h b into two embeddings z a = g ( h a ) and z b = g ( h b ) , whose similarity is maximized by minimizing a contrastive loss, the NT-Xent loss:
[Equation]
where T is temperature, and N denotes the minibatch size.
Since the value range of Equation 2 is [ − 1 , 1] , it is usual to set 0 <
T < 1 for an efﬁcient learning.
Here, we can regard the similarity as state energy E .
According to the Boltzmann distribution, the probability of a state ρ satisﬁes ρ ∝ exp ( − E/kT ) , where the right hand side is called the Boltzmann factor, and k is a constant (Kittel, 2004).
If we use similarity to represent a sequential relationship, we can model the probability of the relationships.
Also, we can adapt the temperature to control the sensitivity of the relationship’s strength to the probability.
This ﬁnding motivated the objective function design in our method, which we will describe in Section 3.2.2.
M ETHODOLOGY
Before presenting the details of our proposed method, we depict the overview of S 3 ADNet’s architecture, as shown in Figure 1.
In the following sections, we ﬁrst introduce the assumptions behind pessimistic contrastive learning (PCL).
Then, we illustrate the pretext tasks and objective functions in our method.
[Figure]
3. A SSUMPTIONS
Given a sequence of elements [ z 1 , z 2 , · · · , z i , · · · , z j , · · · , z L − 1 , z L ] (1 ≤ i < j ≤ L ) , we model the anomalies based on the following pessimistic policy.
Assumption 1 (existence of anomalies)
Anomalies can occur at any element in the sequence.
We assume that there are anomalies in the training data.
Picking out normal samples from a largescale high-dimensional dataset is expensive, not to mention that sometimes we lack prior knowledge of the data.
Therefore, the target of our method is the data mixed with anomalies.
Assumption 2 (bias of the number of anomalies)
The number of anomalies is much less than the normal elements.
The higher bias can help the model capture the statistical features to recognize the anomalies.
If the numbers of abnormal and normal instances are close, our method may fail.
However, if the anomalies are much more than the normal samples, we can treat the virtual normal ones as “anomalies”, such as the experiment on the KDDCup99 dataset.
Assumption 3 (independence between anomalies)
An anomaly that occurs at an element is independent of the other anomaly occurrences in the sequence.
Though anomalies could be relevant in real-world anomaly detection tasks, this assumption can simplify the modeling.
Let p i and p j be the probabilities of anomalies occurring at elements z i and z j , respectively, then the additive probability p ij = p i + p j − p
i
p
j , ¯ p ij = 1 − p ij .
However, determining whether an anomaly has occurred still depends on contextual information.
Assumption 4 (relationships among sequential elements with anomalies)
For two elements in the sequence, if and only if at least one anomaly occurs, they have an exceptional relationship.
We assume that only anomalies can affect the quality of a relationship between two elements in the sequence.
Here, relationship models the degree of correlation between two elements over the sequence.
Then, we need to quantify the relationship for the modeling in the following deﬁnition.
Deﬁnition 1 (quantiﬁcation of two elements having a relationship in a sequence)
Given two elements z i and z j in a sequence, r ij ∈ R quantiﬁes the relationship between z i and z j .
If the two elements have an exceptional relationship, then r ij < 0 .
Otherwise, r ij ≥ 0 .
| r ij | measures the strength of the relationship.
According to Assumption 3, 4, and Deﬁnition 1, we obtain the following corollary.
Corollary 1 (relation between sequential anomaly and relationship) p ij = 0 ⇔ r ij ≥ 0 ; p ij = 1 ⇔ r ij < 0 .
3. P ESSIMISTIC C ONTRASTIVE L EARNING
We need well representations for sequential data points to apply the above assumptions to calculate the quantiﬁed relationships.
As a result, we construct two pretext tasks with diverse objective functions for feature extraction and anomaly detection.
3.2. F EATURE E XTRACTION BY S EQUENTIALLY C ONTRASTING
The contrastive feature extraction task mostly follows SimCLR.
The encoder network in Figureconsists of a feature network for learning representations and a head network for projecting embeddings.
Nonetheless, we perform two types of domain-agnostic feature augmentation methods for non-image and non-textual data, and modify the NT-Xent loss function for the training.
The ﬁrst augmentation method is adding GN ε ∼ N (cid:0) 0 , σ 2 I (cid:1) into the embeddings (DeVries & Taylor, 2019).
The noise is sampled from N ( 0 , I ) and then element-wise multiplied by σ 2 , which is ﬁt by using a full-connected layer attached to the output of the feature network.
The second approach is employing DN by operating dropout with a small probability parameter in every layer of the feature network (Liang et al., 2021), where the bidirectional Kullback-Leibler divergence (biKLD) is used to regularize the augmentation.
We revise Equation 1 with the regularization to the data augmentation into the following formulation:
[Equation]
where α is a hyperparameter to control the regularization, and D KL denotes the Kullback-Leibler divergence (KLD).
The regularization term in Equation 3 turns out to be the biKLD as described in (Liang et al., 2021) in the loss function for minibatches:
[Equation]
For convenience, we call this function Re-Sent 2 (regularized sequential normalized temperaturescaled cross entropy loss).
In the beginning of the training, we set a certain number of epochs to solely use Equation 4 as the warm-up for better representations for the next task.
3.2. A NOMALY E STIMATION BY C ONTEXTUALLY C ONTRASTING
For the sake of detecting anomalies in sequences, we leverage the contextual information given by the representative features of each sequential data point.
Here, a concept represents a bias due to speciﬁc sequential contexts, such as locality and periodicity.
Since sequence data may have multiple concepts, we propose the MCC layer to capture the contextual information of different concepts from the embeddings of the representations learned from sequential data, as shown in Figure 1. The estimation of an anomaly probability p i is formulated by
[Equation]
where sgn indicates the sign function, | · | means the element-wise absolute value function, and c is an index of concept.
Given a sequential embedding Z ∈ R L × m , the MCC layer obtains multiconceptual correlations
M ∈ R L × C × L by performing tensor contractions on Z and its transposed copy with a learnable weight
W MCC ∈ R m × C × m and the tanh activation.
Next, the contextual pooling (CXP) aggregates the correlations for each data point in different concepts into G ∈ R L × C .
After that, the multi-conceptual pooling (MCP) selects the concept value, to which the data point is most possibly attributed, from G for each data point.
Here, we adopt the robust average pooling to CXP, which calculates the mean by ignoring the largest and smallest values (Equation 7).
To MCP, we use the absolute maximum pooling to select the one with the largest absolute value (Equation 6).
The pooling result g ∈ R L is scaled by 1 / √ m and activated with the sigmoid function.
Finally, the activation result (cid:98) p ∈ [0 , 1] L is used to predict the anomaly probabilities.
The tanh activation and the scaling can make the output values of MCC not too large and the initial values approximate to zero, i.e., the anomaly probabilities are estimated to be around 0 .
5 at the beginning of training.
We have modeled the anomaly probability so far.
Next, we formulate the relationships in sequences by using the assumptions mentioned in Section 3.1.
Due to r ij ∈ R according to Deﬁnition 1, we calculate a relationship by
[Equation]
where τ : Z + , Z + → R + is an adaptation function.
Since the sigmoid function σ : R → [0 , 1] keeps the monotonicity, we can model the probability of having an exceptional relationship as
[Equation]
Based on Corollary 1, the objective is to minimize the difference between the distribution of anomalies’ occurrence P and the distribution of data points’ relationships Q .
We could minimize the biKLD to achieve this objective.
However, observing D KL ( P (cid:107) Q )
=
H ( P, Q ) − H
( P ) where
H ( · , · ) gives the cross entropy and H ( · ) calculates Shannon entropy, the second term leads the distributions into entropy maximization in the optimization.
By Assumption 2, the anomaly probability is considered to be much low.
Therefore, we use the following functions instead of biKLD:
[Equation]
where β is a penalty to restrain the entropy maximization.
We also clamp the probabilities with a small value to avoid log 0 .
For the n -th sequence in a minibatch, we formulate the loss function as
[Equation]
where s denotes the lookahead size to handle the number of forwarding relationships to calculate.
For easy reference, we term this loss function CARE (context-adaptive relative entropy loss).
CARE for a minibatch of augmented embedding pairs is given by
[Equation]
[Equation] Moreover, we hope that the contextual information can reﬂect the representations of data points, and hence we combine Equation 4 and 14 into a global loss function for multi-task learning (Ruder, 2017).
The global objective is formulated by
[Equation]
where λ RS and λ CR are the weights for the two loss functions, respectively.
When starting to optimize Equation 15, we also reduce the learning rate of the encoder network for ﬁnetuning.
E XPERIMENTS
This section reports the comparative evaluation of our proposed S 3 ADNet.
We applied a simple fully connected neural network (FCNN) and a simple convolutional neural network (CNN) as the encoder models for two datasets, respectively.
We show and explain the evaluation results, and give an ablation study in the following.
S 3 ADNet was implemented by using Python 3.8 and PyTorch 1.8.
4. D ATASETS
We evaluated S 3 ADNet on the widely used network trafﬁc dataset KDDCup99 1 , and a human activity sensing dataset HASC 2 .
Table 1 summarizes the properties of the datasets.
[Table]
Details of the two datasets and the experimental setup for them are described in Appendix A.
4. R ESULTS
We applied three S 3 ADNet variants based on the data augmentation approach used for the comparative evaluation: 1) with only GN, 2) with only DN, and 3) with both GN and DN.
As S 3 ADNet can predict the anomaly probability directly, the prediction less than 0 .
5 was supposed to be negative (i.e., p i = 0 ), otherwise positive (i.e., p i = 1 ).
Additionally, we use F1-score as the primary metric for the comparison as it is better for imbalanced datasets.
2 shows the result on KDDCup99, while Table 3 reports the one on HASC.
On the KDDCup99 dataset, our method was outperformed the other state-of-the-art methods.
Since the number of learnable parameters in S 3 ADNet (when c = 8 ) was only 1 / 15 of the ones in MDAN, and 1 / 6 of those in TS-CP 2 , our proposed method is simpler and more effective.
The CPD task on the HASC dataset was challenging due to the small size and the few targets, but our proposed S 3 ADNet still achieved the highest score in the 100 -margin setting and the second-highest score in the 60 -margin setting.
A smaller batch size was better for this task.
However, the training was unstable, and the F1-scores oscillated in both two tasks.
We will continue the discussion on the problems in the next section.
4. A BLATION S TUDY
We report the best scores for each hyperparameter settings in the following.
The default values of the hyperparameter are shown in Table 4.
The models were trained ﬁve times in each setting.
[Table]
[Table]
[Table]
[Table]
I MPACTS OF THE N UMBER OF C ONCEPTS
The results shown in Table 5 are obtained by selecting c ∈ { 1 , 4 , 8 , 12 } to train S 3 ADNet with DN on
KDDCup99 and HASC.
We found that more than one concept could help improve the performance.
Nevertheless, because the weight M is dense, a large c increases the number of parameters and may affect the convergence of training.
[-• KDDCup
-][Table] 4.3. I MPACTS OF A DAPTATION F UNCTIONS Since the adaptation function for the KDDCup99 dataset was constant [Table]
we used the results on HASC to analyze the effect of adaptation functions.
In this case
[Table]
we selected T ∈ { 0 . 05 , 0 . 1 } and k ∈ { 0 . 1 , 0 .
5 , 1 } with different adaptation functions.
We found that a smaller temperature was better for variant adaptation functions.
D ISCUSSION
Notwithstanding the outperformance of S 3 ADNet in the evaluation, there are still some limitations in our method.
Firstly, similar to the other deep learning methods, our method requires many hyperparameters to set up the model.
Although several hyperparameter optimization algorithms are available to us (Yu & Zhu, 2020), they need validation sets and speciﬁc metrics for the trials.
While our method is unsupervised/self-supervised, it could not be swift enough to adapt to an entirely new dataset having no validation set.
A potentially viable solution is using the ensemble learning technique to combine the results from several models with low prediction losses.
Nevertheless, there was the second problem.
We observed that the training of S 3 ADNet could be unstable like the GAN-based methods.
This problem results in the difﬁculty of selecting well enough models by the prediction losses.
One possible reason is that the value instability from KLD.
We have attempted to use the Jensen–Shannon divergence instead of KLD, but the performance was even worse.
It is the future work to seek a better measure.
Another cause could be in the modeling of relationships.
Before the context learning stage, the encoder has been trained to make the identical instances have the representations as similar as possible but not concerning the contextual relationships.
After that, however, the identical relationship could be broken by optimizing Equation 14.
It seems reasonable to attempt some tricks for training stabilization, such as stop-gradient (Chen & He, 2021).
One more reason could be that the degrees of freedom of the MCC weight were too high due to the density.
It may be helpful to employ sparsiﬁcation or factorization to the weight.
Thirdly, there are still potential options to improve the performance.
We proposed a method based on the density assumption so far.
Since the hybrid method ESPRESSO acquired the top score in the 200 -margin setting of the CPD task on HASC, it could be wise to take advantage of other principles in our model.
Furthermore, though we have applied a simple FCNN and a simple CNN as the encoder networks in the experiment, we have not investigated the selection of the hyperparameters of encoders in depth.
Also, we found that they were still hard to extract representative features from complex multivariate time-series data, such as SMAP and MSL (Hundman et al., 2018), SMD (Su et al., 2019).
It needs more studies on selecting encoder networks for those data, such as leveraging
RNN to learn representations of short sequences.
C ONCLUSION
This paper proposed a ﬂexible and straightforward framework S 3 ADNet, for detecting anomalies on sequential data by using contrastive learning under the pessimistic assumptions.
The future work is to stabilize the training and attempt to leverage other aspects to improve the performance.
A D ETAILS OF E XPERIMENT S ETUP A. D ETAILS OF D ATASETS
{+• KDDCup
+}
(Stolfo et al., 2000).
In the experiment, we used the KDDCup99 10% dataset.
As mentioned in Section 3.1, the number of records labeled as “normal” in the dataset is highly less than that of intrusions, so that we regarded the instances labeled as attacks as positive samples for the SAD task.
For each variable, the discrete ones were one-hot encoded, while the contiguous ones were standardized by removing the median and scaling to unit variance according to the ﬁrst and third quartile values.
We split the dataset into two random halves, and employed the ﬁrst half of data for training and the rest for testing in each epoch.
• HASC (Kawaguchi et al., 2011).
We utilized the same HASC subset used by KL-CPD and TS-CP 2 , having only the 3-axis accelerometer logs.
The target of the task is to detect the human activity pattern changes.
As following the setting in (Deldari et al., 2021), we set three chunk sizes w ∈ { 60 , 100 , 200 } to vary the subset to three chunked versions and used the whole subset for both the training and testing in the evaluation.
The values of each axis were standardized by removing the mean and scaling.
For each dataset the samples in a minibatch were generated in a sliding window fashion where the window size was the sequence length L , and the stride was 1 . [Equation] [Equation]
M ODEL S ETUP
According to the properties of the two datasets, we applied diverse but simple encoder networks for them.
The details of the architectures and the hyperparameter settings are described as follows.
• KDDCup99.
Due to the sparsity of each data point x ∈ R 121 , we employed a 2-layer FCNN for the feature network and a 1-layer FCNN for the head network.
In the feature network, the output size of each layer was 32 .
The embedding size of the head network was m = 8 .
For the hyperparameter optimization, the sequence length L ∈ { 4 , 8 , 16 } , the batch size N = 256 , and the warm-up was up to { 5 , 10 , 20 } epochs.
Since the data points were shufﬂed, we set the adaptation function to τ ( i, j ) = k ( constant ).
• HASC.
Depending on the chunk size w , each data point was a dense matrix X ∈ R 3 × w .
This time, we applied a 2-layer 1D CNN for the feature network but a 1-layer FCNN for the head network.
For each convolution layer, the output channel size was 32 , the kernel size was selected from { 3 , 5 } , and each input’s head and tail were padded with a zero.
A maximum pooling and an average pooling were adapted to each output channel from the second CNN layer to extract the concatenated representative features with the size 64 .
Then, the representations were projected to embeddings with the size m = 16 .
For the hyperparameter selection, the sequence length L ∈ { 4 , 8 } , the batch size N ∈ { 4 , 8 } , and the number of warm-up epochs were chosen from { 20 , 30 } . The adaptation function was selected from constant , τ ( i, j ) = k ln( j − i + 1) ( log ), τ ( i, j ) = k √ j − i ( root ), and τ ( i, j )
= 1 . 1 j − i k ( exp ), making the relationships with further data points weaker. The following settings were common for both datasets.
The activation was LeakyReLU
(Maas et al., 2013) with a negative slope of 0 .
2 in both of the feature networks.
For the data augmentation using
DN, the dropout probability was set to 0 .
The objective functions were optimized by applying
RMSProp (Graves, 2013) with the learning rate 0 .
1 , the momentum 0 .
9 , the smoothing parameter
[Figure]
0 . 99 , and the weight decay 1 .
× 10 − 4 .
Since KLD could approach inﬁnity, the gradients were clipped in the range [ − 10 , 10] .
After the warm-up epochs, the learning rate for the encoder network was reduced to 1 .
[-× 10 − 3 or 1 . 
-]
× 10 − 4 for the ﬁnetuning.
For the multi-task learning, we set λ RS =and λ CR ∈ { 3 , 5 } .
Every training trial was up to 100 epochs.
The lookahead size was given by s = (cid:98) rL (cid:99) , where r ∈ { 0 . 25 , 0 . 5 , 0 . 75 } .
Others were, the number of concepts C ∈ { 1 , 4 , 8 , 12 } , the regularization weight α ∈ { 0 . 1 , 0 . 5 , 1 } , the entropy penalty β ∈ { 0 . 1 , 0 .
5 , 1 } , and the temperature T ∈ { 0 . 05 , 0 . 1 } .
Furthermore, the coefﬁcients k in adaptation functions were selected from { 0 . 1 , 0 . 25 , 0 . 5 , 1 } ,
B E XAMPLES OF T RAINING LOSS PATTERNS
Figure 2 shows two examples of the training loss patterns on KDDCup99.
The changes of contrast losses and augment losses were intense in warm-up epochs.
If the hyperparameter setting is selected appropriately, there could be signiﬁcant and sustainable drops in relate losses, as shown in Figure 2a.
The instability in Figure 2b may be attributed to KLD.