Original: Exploiting Variable Correlation with Masked Modeling for Anomaly Detection in Time Series
Revised: Exploiting Variable Correlation with Masked Modeling for Anomaly Detection in Time Series

Original: Abstract
Revised: Abstract

Original: Online anomaly detection in multi-variate time series is a challenging problemparticularly when there is no supervision information.
Revised: Online anomaly detection in multi-variate time series is a challenging problem particularly when there is no supervision information.

Original: Autoregressive predictivemodels are often used for this task, but such detection methods often overlookcorrelations between variables observed in the most recent step and thus misssome anomalies that violate normal variable relations.
Revised: Autoregressive predictive models are often used for this task, but such detection methods often overlook correlations between variables observed in the most recent step and thus miss some anomalies that violate normal variable relations.

Original: In this work, we propose amasked modeling approach that captures variable relations and temporal relationsin a single predictive model.
Revised: In this work, we propose a masked modeling approach that captures variable relations and temporal relations in a single predictive model.

Original: Our method can be combined with a wide range ofpredictive models.
Revised: Our method can be combined with a wide range of predictive models.

Original: Our experiment shows that our new masked modeling methodimproves detection performance over pure autoregressive models when the timeseries itself is not very predictable.
Revised: Our experiment shows that our new masked modeling method improves detection performance over pure autoregressive models when the time series itself is not very predictable.

Original:  Introduction
Revised:  Introduction

Original: Unsupervised anomaly detection is the problem of detecting data points that do not conform to anormal data distribution (Chandola, Banerjee, and Kumar, 2009; Chalapathy and Chawla, 2019).
Revised: Unsupervised anomaly detection is the problem of detecting data points that do not conform to a normal data distribution (Chandola, Banerjee and Kumar, 2009; Chalapathy and Chawla, 2019).

Original: Inmany applications, there are abundant anomaly-free training data but no information about possibleanomalies.
Revised: In many applications, there are abundant anomaly-free training data but no information about possible anomalies.

Original: In the context of sequential data, novelties usually manifest as events that appear "outof context" or that do not follow the typical time-dependent data generating process (Cook, Mısırlı,and Fan, 2019; Blázquez-García et al., 2021; Choi et al., 2021).
Revised: In the context of sequential data, novelties usually manifest as events that appear "out of context" or that do not follow the typical time-dependent data generating process (Cook, Mısırlı, and Fan, 2019; Blázquez-García et al., 2021; Choi et al., 2021).

Original: The ability to detect novelty in timeseries data in an online manner is very useful in real-time monitoring systems.
Revised: The ability to detect novelty in time series data in an online manner is very useful in real-time monitoring systems.

Original: In our motivatingapplication, an agent in a game needs to detect abnormal game transitions and adjust game strategyin real time.
Revised: In our motivating application, an agent in a game needs to detect abnormal game transitions and adjust game strategy in real time.

Original: In this work, we focus on online anomaly detection in time series and assume no accessto anomalies.
Revised: In this work, we focus on online anomaly detection in time series and assume no access to anomalies.

Original: Autoregressive predictive models are often used for anomaly detection.
Revised: Autoregressive predictive models are often used for anomaly detection.

Original: Specifically, the predictivemodel takes previous observations x <t = ( x 1 , . . .
Revised: Specifically, the predictive model takes previous observations x <t = ( x 1 , . . .

Original: , x t − 1 ) and makes a prediction ˆ x t of the currentstep.
Revised: , x t − 1 ) and makes a prediction ˆ x t of the current step.

Original: Then the received observation x t at the current time step is compared against the prediction: ifx t deviates much from the prediction, then x t is claimed to be anomalous.
Revised: Then the received observation x t at the current time step is compared against the prediction: if x t deviates much from the prediction, then x t is claimed to be anomalous.

Original: The detection performancelargely depends on the accuracy of the predictive model.
Revised: The detection performance largely depends on the accuracy of the predictive model.

Original: Research in this direction has proposedvarious methods to improve the predictive model of such detection system (Malhotra et al., 2015;
Revised: Research in this direction has proposed various methods to improve the predictive model of such detection system (Malhotra et al., 2015;

Original: Hundman et al., 2018; Munir et al., 2018; Deng and Hooi, 2021; Chen et al., 2021).
Revised: Hundman et al., 2018; Munir et al., 2018; Deng and Hooi, 2021; Chen et al., 2021).

Original: However, thesemethods often overlook correlations between variables observed in the most recent step and thusmiss some anomalies that violate normal variable relations.
Revised: However, these methods often overlook correlations between variables observed in the most recent step and thus miss some anomalies that violate normal variable relations.

Original: In our motivating applications, such areabundant as game rules enforce strong correlations between variables and stochasticity in the gamemakes the time-series difficult to predict.
Revised: In our application of detecting anomalous events in games, game rules enforce strong correlations between variables and stochasticity in the game makes the time-series difficult to predict, rendering typical autoregressive methods less powerful at detecting anomalies. To further motivate this fact, we can consider an example from the popular game of Monopoly. Consider a game of monopoly between two players, one of which might be dishonest. The honest player will attempt to check on every turn that their opponent did not break any rules, such as moving to positions of the board outside the one dictated by their dice roll.

Revised: If the honest player employs an autoregressive model, that takes the past few board states and attempts to predict the next one, they will have a very hard time catching their opponent cheating.

Revised: That is because without observing the opponents dice roll in the current turn, it is not possible to predict with confidence what the next board state will be.

Original:  In this work, we devise a masking strategy that allows any predictive model to capture not onlytemporal relations but also correlations between variables in the same time step.
Revised: However, if current step information is available to the model, and it can observe the dice-roll, detecting cheating becomes much easier. As extreme as this example may seem, this effect is widespread in the game. Other variables such as player’s cash, properties owned and rent fees also exhibit similarly strong correlations within single time-steps and are themselves difficult to predict due to the stochasticity of the game. In this work, we devise a masking strategy that allows any predictive model to capture not only temporal relations but also correlations between variables in the same time step.

Original: Inspired by maskedlanguage modeling (Devlin et al., 2018), we withhold data entries at the current time step x t , then we train the model to recover masked entries based on other entries as well as previous observations.
Revised: Inspired by masked language modeling (Devlin et al., 2018), we withhold data entries at the current time step x t , then we train the model to recover masked entries based on other entries as well as previous observations.

Original: Inthis new approach, the model is able to capture relations between data entries within x t .
Revised: In this new approach, the model is able to capture relations between data entries within x t .

Original: We perform experiments on our game datasets and show that our method improves the detectionperformance of a predictive model.
Revised: We perform experiments on our game datasets and show that our method improves the detection performance of a predictive model.

Original: Along with detection, we also measure localization performance,that is the ability to identify anomalous variables in the input vector, rather than just the input vectoras a whole.
Revised: Along with detection, we also measure localization performance, that is the ability to identify anomalous variables in the input vector, rather than just the input vector as a whole.

Original:  Related Work
Revised:  Related Work

Original: Autoregressive predictive models are commonly used for anomaly detection in the online setting
Revised: Autoregressive predictive models are commonly used for anomaly detection in the online setting

Original: Malhotra et al.
Revised: Malhotra et al.

Original: These models are commonly implemented with either a fixed window sizeover temporal context or a recurrent neural network (RNN) (Filonov, Lavrentyev, and Vorontsov,2016).
Revised: These models are commonly implemented with either a fixed window size over temporal context or a recurrent neural network (RNN) (Filonov, Lavrentyev, and Vorontsov, 2016).

Original: Deng and Hooi (2021) improve the predictive model in the anomaly detection task: theproposed model learns a graph over features to better capture relationships between the sensors intime series.
Revised: Deng and Hooi (2021) improve the predictive model in the anomaly detection task: the proposed model learns a graph over features to better capture relationships between the sensors in time series.

Original: However, they solely rely on past observations to predict observations at the current step.
Revised: However, they solely rely on past observations to predict observations at the current step.

Original:  Masked Autoregressive Training3.1 Problem Setup
Revised:  Masked Autoregressive Training 3.1 Problem Setup

Original: Suppose we have a normal system that generates time series X
Revised: Suppose we have a normal system that generates time series X

Original: =
Revised: =

Original: ( x τ : τ = 1 , . .
Revised: ( x τ : τ = 1 , . .

Original:  , T ) accordingto some unknown distribution p ∗ ( x τ | x <τ ) .
Revised: . , T ) according to some unknown distribution p ∗ ( x τ | x <τ ) .

Original: Here the observation x τ ∈ R d at each time step τ is avector of d variables.
Revised: Here the observation x τ ∈ R d at each time step τ is a vector of d variables.

Original: In the problem of online anomaly detection in time series, we only observe thetime series x ≤ t =
Revised: In the problem of online anomaly detection in time series, we only observe the time series x ≤ t =

Original: ( x τ : τ = 1 , . .
Revised: ( x τ : τ = 1 , . .

Original: . , t ) up to t , and the task is to decide whether x t follows the sameunderlying distribution p ∗ ( x t | x <t ) .
Revised: . , t ) up to t , and the task is to decide whether x t follows the same underlying distribution p ∗ ( x t | x <t ) .

Original: Note that there are no training examples of anomalies, so this isan unsupervised problem.
Revised: Note that there are no training examples of anomalies, so this is an unsupervised problem.

Original: In many applications, we are interested whether a single entry x t,i is from the modelp ∗ ( x t,i | x t,η \ i , x <t ) .
Revised: In many applications, we are interested whether a single entry x t,i is from the model p ∗ ( x t,i | x t,η \ i , x <t ) .

Original: Here the subscript \ i denotes all entries in a vector except i and η denotesthe set of normal entries in x t .
Revised: Here the subscript \ i denotes all entries in a vector except i and η denotes the set of normal entries in x t .

Original: While η is also time dependent, we omit the subscript to simplify thenotation.
Revised: While η is also time dependent, we omit the subscript to simplify the notation.

Original: In online anomaly detection, a typical method is to train a predictive model p ( x τ | x <τ ) .
Revised: In online anomaly detection, a typical method is to train a predictive model p ( x τ | x <τ ) .

Original: Then, anobservation x t is compared to the prediction ˆ x t from p ( x t | x <t ) .
Revised: Then, an observation x t is compared to the prediction ˆ x t from p ( x t | x <t ) .

Original: If x t deviates significantly fromthe prediction, then it is decided that x t contains anomalous entries.
Revised: If x t deviates significantly from the prediction, then it is decided that x t contains anomalous entries.

Original: Similarly, a single entry x t,i iscompared to the prediction ˆ x t,i to decide whether x t,i is anomalous.
Revised: Similarly, a single entry x t,i is compared to the prediction ˆ x t,i to decide whether x t,i is anomalous.

Original: Typically, this method implicitlyassumes entries x t,i are conditionally independent given x <t .
Revised: Typically, this method implicitly assumes entries x t,i are conditionally independent given x <t .

Original: In our motivating application ofmodeling game data where x t represents a game state at time t , two variables x t,i and x t,j mightlook normal according to their marginal distributions p ( x t,i | x <t ) and p ( x t,j | x <t ) but are anomalousaccording to their joint distribution p ( x t,i , x t,j | x <t ) .
Revised: In our motivating application of modeling game data where x t represents a game state at time t , two variables x t,i and x t,j might look normal according to their marginal distributions p ( x t,i | x <t ) and p ( x t,j | x <t ) but are anomalous according to their joint distribution p ( x t,i , x t,j | x <t ) .

Original: This type of anomaly is only obvious when wemodel the relation between x t,i and x t,j .
Revised: This type of anomaly is only obvious when we model the relation between x t,i and x t,j .

Original: 3.2 Masked modeling for detecting and locating anomalies
Revised: 3.2 Masked modeling for detecting and locating anomalies

Original: The goal of our method is to approximate the true data distribution p ∗ ( x t,i | x t,η \ i , x <t ) .
Revised: The goal of our method is to approximate the true data distribution p ∗ ( x t,i | x t,η \ i , x <t ) .

Original: The two mainchallenges we face are determining the normal set of variables η to condition on and approximatingthe conditional with a neural model.
Revised: The two main challenges we face are determining the normal set of variables η to condition on and approximating the conditional with a neural model.

Original: The first problem is addressed by an error-based filter thatapproximates the set η using an underlying autoregressive model g ψ ( · ) .
Revised: The first problem is addressed by an error-based filter that approximates the set η using an underlying autoregressive model g ψ ( · ) .

Original: This model can be eitherpretrained and fine-tuned with our module, or trained from scratch in an end to end manner.
Revised: This model can be either pretrained and fine-tuned with our module, or trained from scratch in an end to end manner.

Original: Toaddress the second problem we notice that the form of p ∗ has structure analogous to masked modelingand employ a masking procedure that allows us to efficiently fit the distribution.
Revised: To address the second problem we notice that the form of p ∗ has structure analogous to masked modeling and employ a masking procedure that allows us to efficiently fit the distribution.

Original: Error-based filter As highlighted earlier, x t,i depends on the temporal context x <t and the valuesof normal observations x t,η \ i .
Revised: Error-based filter As highlighted earlier, x t,i depends on the temporal context x <t and the values of normal observations x t,η \ i .

Original: Identifying the set η is not trivial, as it is tantamount to solving the fullanomaly localization problem.
Revised: Identifying the set η is not trivial, as it is tantamount to solving the full anomaly localization problem.

Original: However, we can achieve an initial estimate by predicting the currentstep ˜ x t using only the temporal context x <t and then comparing it against current observations x t :those entries with large predictive errors will not be in our approximation of η .
Revised: However, we can achieve an initial estimate by predicting the current step ˜ x t using only the temporal context x <t and then comparing it against current observations x t : those entries with large predictive errors will not be in our approximation of η .

Original: Let the predictive model be
Revised: Let the predictive model be

Original: [Equation]
Revised: [Equation]

Original: The predictive model can be implemented with RNNs and alike.
Revised: The predictive model can be implemented with RNNs and alike.

Original: Let e t be the error vector containingthe predictive error at each entry, then the binary representation η of the set η is computed as follows.
Revised: Let e t be the error vector containing the predictive error at each entry, then the binary representation η of the set η is computed as follows.

Original: [Equation]
Revised: [Equation]

Original: Again we omit the subscript t from η for simplicity.
Revised: Again we omit the subscript t from η for simplicity.

Original: Here r ∈ R is a hyperparameter.
Revised: Here r ∈ R is a hyperparameter.

Original: Importantly,this is not how we detect anomalies, as the second step of the module will leverage current-stepinformation to improve the prediction of x t,i .
Revised: Importantly, this is not how we detect anomalies, as the second step of the module will leverage current-step information to improve the prediction of x t,i .

Original: Instead, the error filter allows the module to base itsfinal prediction only on normal observed entries in η by removing suspicious entries.
Revised: Instead, the error filter allows the module to base its final prediction only on normal observed entries in η by removing suspicious entries.

Original: Using the errorfilter we can produce x ∗ t which corresponds to x t,η and treats variables not in η as missing values.
Revised: Using the error filter we can produce x ∗ t which corresponds to x t,η and treats variables not in η as missing values.

Original: There are various ways to impute those values but for simplicity we impute with the correspondingvalues of x t − 1 .
Revised: There are various ways to impute those values but for simplicity we impute with the corresponding values of x t − 1 .

Original: Masked modeling module We then approximate the conditional distribution p ( x t,i | x t,η \ i , x <t )using the masking mechanism.
Revised: Masked modeling module We then approximate the conditional distribution p ( x t,i | x t,η \ i , x <t ) using the masking mechanism.

Original: The main challenge here is the computational cost: while anautoregressive model p ( x t | x <t ) only needs to predict once, the masked model needs to predict eachvariable in x t separately.
Revised: The main challenge here is the computational cost: while an autoregressive model p ( x t | x <t ) only needs to predict once, the masked model needs to predict each variable in x t separately.

Original: To address this issue, we consider two strategies: using a shared model topredict all variables and compressing information x <t once for all predictions.
Revised: To address this issue, we consider two strategies: using a shared model to predict all variables and compressing information x <t once for all predictions.

Original: Let f ϕ ( · ) denote the shared predictive module that has d outputs and predicts all variables in x t .
Revised: Let f ϕ ( · ) denote the shared predictive module that has d outputs and predicts all variables in x t .

Original: Hereϕ denotes all trainable parameters of the module.
Revised: Here ϕ denotes all trainable parameters of the module.

Original: To prepare the input, we first impute values not in ηfor the module.
Revised: To prepare the input, we first impute values not in η for the module.

Original: There are different ways to impute these values (e.g. using zeros).
Revised: There are different ways to impute these values (e.g. using zeros).

Original: Here we choose touse values from the previous step.
Revised: Here we choose to use values from the previous step.

Original: [Equation]
Revised: [Equation]

Original: Then the masked predictive module f ϕ ( · ) works as follows.
Revised: Then the masked predictive module f ϕ ( · ) works as follows.

Original: Let m i =
Revised: Let m i =

Original: onehot( i ) denote the one-hotencoding of i .
Revised: onehot( i ) denote the one-hot encoding of i .

Original: [Equation]
Revised: [Equation]

Original: Here ⊙ represents element-wise multiplications.
Revised: Here ⊙ represents element-wise multiplications.

Original: We also feed η to the module to indicate imputedvalues.
Revised: We also feed η to the module to indicate imputed values.

Original: The shared network f ϕ takes a simple MLP architecture.
Revised: The shared network f ϕ takes a simple MLP architecture.

Original: It runs efficiently even though itpredicts d times in total for all variables in x ∗ t .
Revised: It runs efficiently even though it predicts d times in total for all variables in x ∗ t .

Original: Note that the information about x <t is encoded inthe prediction ˜ x t of the autoregressive model, and is only computed once .
Revised: Note that the information about x <t is encoded in the prediction ˜ x t of the autoregressive model, and is only computed once .

Original: Additional running timeconsiderations are discussed in the appendix section.
Revised: Additional running time considerations are discussed in the appendix section.

Original: Training loss and Anomaly score The training loss of the model is still the same as for anautoregressive model:
Revised: Training loss and Anomaly score The training loss of the model is still the same as for an autoregressive model:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Here the loss function can adapt to different types of variables.
Revised: Here the loss function can adapt to different types of variables.

Original: For example, it can be the squareddifference if x t,i is a numerical variable or the binary cross-entropy loss if x t,i is a binary variable.
Revised: For example, it can be the squared difference if x t,i is a numerical variable or the binary cross-entropy loss if x t,i is a binary variable.

Original: Asimilar calculation is also used for compute anomaly scores at test time for each variable, which areaggregated to form the anomaly score for the time step.
Revised: A similar calculation is also used for compute anomaly scores at test time for each variable, which are aggregated to form the anomaly score for the time step.

Original: For more details please refer to the appendix.
Revised: For more details please refer to the appendix.

Original:  Experiments
Revised:  Experiments

Original: We conduct 2 experiments to evaluate the performance of models with the masking module onanomaly detection and localization.
Revised: We conduct 2 experiments to evaluate the performance of models with the masking module on anomaly detection and localization.

Original: An ablation of the error filter is available in the appendix.
Revised: An ablation of the error filter is available in the appendix.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Datasets We collect data from two games Polycraftv

Revised: Datasets We collect data from two games Polycraftv

Original: (Smaldone et al., 2017) and Monopoly(Haliem et al., 2021).
Revised: (Smaldone et al., 2017) and Monopoly (Haliem et al., 2021).

Original: Polycraftv2 is an open environment where agents collect resources and craftitems.
Revised: Polycraftv2 is an open environment where agents collect resources and craft items.

Original: Monopoly is a simulation derived from the well-known board game Monopoly.
Revised: Monopoly is a simulation derived from the well-known board game Monopoly.

Original: The games areplayed by planning agents based on the DIARC architecture(Schermerhorn et al., 2007).
Revised: The games are played by planning agents based on the DIARC architecture(Schermerhorn et al., 2007).

Original: Anomaliesin games include changes of rules, objects, or entity relations in the game.
Revised: Anomalies in games include changes of rules, objects, or entity relations in the game.

Revised: Labels for the timesteps where anomalies occur, as well as for the specific variables affected by them are recorded automatically when the anomalous game mechanic is in effect.

Original: We perform evaluation onboth novelty detection and localization.
Revised: We perform evaluation on both novelty detection and localization.

Original: For novelty detection, we use the positive label for anomaloustime steps.
Revised: For novelty detection, we use the positive label for anomalous time steps.

Original: For novelty localization we use the positive label for anomalous features.
Revised: For novelty localization we use the positive label for anomalous features.

Original: Table 4 in theappendix summarizes the dataset statistics for each game.
Revised: Table 4 in the appendix summarizes the dataset statistics for each game.

Original: Experiment settings We augment two autoregressive models with our module: multilayerperceptron (MLP) and
Revised: Experiment settings We augment two autoregressive models with our module: multilayerperceptron (MLP) and

Original: GDN (Xu et al., 2021).
Revised: GDN (Xu et al., 2021).

Original: Further information on training settings is available inthe appendix.
Revised: Further information on training settings is available in the appendix.

Original: We evaluate anomaly detection by comparing step-wise anomaly scores from differentmodels against anomaly labels.
Revised: We evaluate anomaly detection by comparing step-wise anomaly scores from different models against anomaly labels.

Original: Following previous work, we set the threshold to the maximumerror over the validation set and compute the F1 score for each model.
Revised: Following previous work, we set the threshold to the maximum error over the validation set and compute the F1 score for each model.

Original: We also show the precisionand recall at this threshold.
Revised: We also show the precision and recall at this threshold.

Original: For anomaly localization, we take the micro-average of F1 scores of allvariables at different times: we pool anomaly scores for all variables and then compare them againsttrue labels to compute the F1 score, precision, and recall.
Revised: For anomaly localization, we take the micro-average of F1 scores of all variables at different times: we pool anomaly scores for all variables and then compare them against true labels to compute the F1 score, precision, and recall.

Original: Results Results on anomaly detection are shown in table 1 and anomaly localization in table 2.
Revised: Results Results on anomaly detection are shown in table 1 and anomaly localization in table 2.

Original: Both games have strong correlations between features as enforced by game rules, and sources ofstochasticity that make the time-series difficult to predict.
Revised: Both games have strong correlations between features as enforced by game rules, and sources of stochasticity that make the time-series difficult to predict.

Original: In Polycraftv2, resource amounts arestrongly correlated as some are used to directly craft others.
Revised: In Polycraftv2, resource amounts are strongly correlated as some are used to directly craft others.

Original: Similarly, in Monopoly the player’scash is strongly correlated with property ownership.
Revised: Similarly, in Monopoly the player’s cash is strongly correlated with property ownership.

Original: The stochasticity in the games comes fromrandom effects such as dice rolls and agent’s decision-making.
Revised: The stochasticity in the games comes from random effects such as dice rolls and agent’s decision-making.

Original: As a result, these conditions allow themasking module to offer statistically significant improvement in F1 score for both models in bothenvironments.
Revised: As a result, these conditions allow the masking module to offer statistically significant improvement in F1 score for both models in both environments.  Conclusion

Revised: We introduce a novel masked training approach for unsupervised online detection of anomalies in time series.

Original: Supplement
Revised: Compared with autoregressive predictive methods, the masked training method exploits the relations between features in current date observations to increase predictive power and therefore more effectively discover and localize anomalies in the input vector. The new masked training method shows significant improvement in anomaly detection in interactive games, where the time series is not very predictable due to the inherent stochasticity of human and computer agent players and the features in the observed game states have strong intra-step correlations. Acknowledgements and Disclosure of Funding This work was funded by the DARPA SAILON program under grant W911NF-20-2-0006. We thank Drs.

Original: 5.1 Anomaly Score calculation
Revised: Eric Kildebeck and Walter Voit, as well as the other members of the UT Dallas PolyCraft team for making available their Polycraftv environment which was central to the development of this work. Additionally, we would like to thank Mayank Kejriwal, Shilpa Thomas, Min-Hsueh Chiu and other members of the University of Southern California team for the Monopoly simulator, which was also indispensable in developing this work.

Revised: Finally, we would like to thank the anonymous reviewers for their helpful comments and suggestions.

Revised:  Supplement 6.1 Anomaly Score calculation

Original: For any predictive model including those with our module, we compare model predictions toobservations to calculate the anomaly score.
Revised: For any predictive model including those with our module, we compare model predictions to observations to calculate the anomaly score.

Original: We first consider locating anomalies to individual variables.
Revised: We first consider locating anomalies to individual variables.

Original: Here we compare x t,i and ˆ x t,i to get theanomaly score α t,i for entry i at time t .
Revised: Here we compare x t,i and ˆ x t,i to get the anomaly score α t,i for entry i at time t .

Original: Let r t,i = err( x t,i , ˆ x t,i ) be the predictive error with sometype of error function.
Revised: Let r t,i = err( x t,i , ˆ x t,i ) be the predictive error with some type of error function.

Original: For example, it can be the absolute error if x t,i is continuous.
Revised: For example, it can be the absolute error if x t,i is continuous.

Original: We also computea single anomaly score a t for the entire time step t by aggregating anomaly scores of single entries.
Revised: We also compute a single anomaly score a t for the entire time step t by aggregating anomaly scores of single entries.

Original: [Equation]
Revised: [Equation]

Original: There are various choices for the aggregation function.
Revised: There are various choices for the aggregation function.

Original: For example, we can take the average ormaximum of the d anomaly scores.
Revised: For example, we can take the average or maximum of the d anomaly scores.

Original: In our experiments, we use the maximum as the step-wiseanomaly score.
Revised: In our experiments, we use the maximum as the step-wise anomaly score.

Original: Figure 1 illustrates the architecture.
Revised: Figure 1 illustrates the architecture.

Original: 5.2 Running Time Considerations
Revised: 6.2 Running Time Considerations

Original: If the number d of variables is very large, variables can be grouped together in masking to furtherreduce the required computation.
Revised: If the number d of variables is very large, variables can be grouped together in masking to further reduce the required computation.

Original: Suppose we create k groups from d variables and denote eachgroup with a binary vector.
Revised: Suppose we create k groups from d variables and denote each group with a binary vector.

Original: [Equation]
Revised: [Equation]

Original: Then the conditional distribution we need to fit is p ∗ ( x t, m k | x t, (1 − m k ) ⊙ η , x <t ) , which is a general-157 ization of the conditional distribution of a single variable.
Revised: Then the conditional distribution we need to fit is p ∗ ( x t, m k | x t, [Equation] 1 − m k ) ⊙ η , x <t ) , which is a general- ization of the conditional distribution of a single variable.

Original: The prediction ˆ x t is computed in the sameway as (5) except that the summation loops over k masks.
Revised: The prediction ˆ x t is computed in the same way as (5) except that the summation loops over k masks.

Original: However, in practice we find that for non-time critical applications the lightweight architecture of themodule means the extra computational cost is not very significant.
Revised: However, in practice we find that for non-time critical applications the lightweight architecture of the module means the extra computational cost is not very significant.

Original: Table 3 shows per training epochrunning times for models with the masked module without variable grouping.Models Polycraftv2 (s)
Revised: Table 3 shows per training epoch running times for models with the masked module without variable grouping.

Original: Monopoly (s)

Original: MLP 4.65 ± 0.

Original: 8.36 ± 0.

Original: [Table]
Revised: [Table]

Original: Table 4 shows the statistics of the game datasets.
Revised: Table 4 shows the statistics of the game datasets.

Original: 5.4 Training information and Hyperparameters
Revised: 6.4 Training information and Hyperparameters

Original: The game datasets include binary, numerical and categorical features.
Revised: The game datasets include binary, numerical and categorical features.

Original: Categories in categoricalfeatures are encoded as vectors.
Revised: Categories in categorical features are encoded as vectors.

Original: Numerical features are normalized by subtracting their training datamean and dividing by the variance.
Revised: Numerical features are normalized by subtracting their training data mean and dividing by the variance.

Original: To scale up the training, all models use fixed window size for thetemporal context and are trained for a maximum of 200 epochs with early stopping.
Revised: To scale up the training, all models use fixed window size for the temporal context and are trained for a maximum of 200 epochs with early stopping.

Original: We run eachmodel 10 times and collect the mean.
Revised: We run each model 10 times and collect the mean.

Original: Common Training parameters: Adam, learning rate 1e-3, beta1,beta2= (0.9,0.99).
Revised: Common Training parameters: Adam, learning rate 1e-3, beta1,beta2= (0.9,0.99).

Original: Early stoppingwith patience 15.
Revised: Early stopping with patience 15.

Original: Dataset : [-Polycraftv
-]
Revised: Dataset : [-Polycraftv
-]Polycraftv

Original: GDN:
Revised: GDN:

Original: Embedding dimension 64, topk 5, Hidden dimension 16, [-dropout=0.
-]
Revised: Embedding dimension 64, topk 5, Hidden dimension 16, [-dropout=0.
-]dropout=0.

Original: Dataset # variables # Train # Test % Anomalous Steps % Anomalous Variables
Revised: Dataset # variables # Train # Test % [Table] [Table]

Original: Polycraftv     1.
Revised: Polycraftv     1.

Original: Monopoly    2.
Revised: Monopoly    2.

Original: Masked GDN: masks 10, Embedding dimension 64, topk 15, Hidden dimension[-
-]
Revised: Masked GDN: masks 10, Embedding dimension 64, topk 15, Hidden dimension[-
-]

Original: MLP: Hidden dimension 32, last layer dimension 16, number of hidden layers 4, dropout [-0.
-]
Revised: MLP: Hidden dimension 32, last layer dimension 16, number of hidden layers 4, dropout [-0.
-]0.

Original: Masked MLP: masks 10, Hidden dimension 16, last layer dimension 16, number of hidden layers [-4,dropout 0.
-]
Revised: Masked MLP: masks 10, Hidden dimension 16, last layer dimension 16, number of hidden layers [-4,dropout 0.
-]4, dropout 0.

Original: Dataset : Monopoly
Revised: Dataset : Monopoly

Original: GDN:
Revised: GDN:

Original: Embedding dimension 64, topk 5, Hidden dimension 16, [-dropout=0.
-]
Revised: Embedding dimension 64, topk 5, Hidden dimension 16, [-dropout=0.
-]dropout=0.

Original: Masked GDN: masks 19, Embedding dimension 64, topk 15, Hidden dimension[-
-]
Revised: Masked GDN: masks 19, Embedding dimension 64, topk 15, Hidden dimension[-
-]

Original: MLP: Hidden dimension 256, last layer dimension 64, number of hidden layers 2, dropout [-0.
-]
Revised: MLP: Hidden dimension 256, last layer dimension 64, number of hidden layers 2, dropout [-0.
-]0.

Original: Masked MLP: masks 19, Hidden dimension 64, last layer dimension 64, number of hidden layers [-2,dropout 0.
-]
Revised: Masked MLP: masks 19, Hidden dimension 64, last layer dimension 64, number of hidden layers [-2,dropout 0.
-]2, dropout 0.

Original: Additional Experiments
Revised: Additional Experiments

Original: [Figure] Ablation [Figure] error 
Revised: 6.4.1 Ablation of error filter

Original: [Figure]

Original: In this experiment we perform ablation of the error filter by varying the value of r .
Revised: In this experiment we perform ablation of the error filter by varying the value of r .

Original: We would like tonote that this experiment was conducted after the completion of all previous experiments and did notinfluence the selection of r .
Revised: We would like to note that this experiment was conducted after the completion of all previous experiments and did not influence the selection of r .

Original: Experiment Settings We perform the ablation study on the anomaly localization tasks for monopolyand Polycraftv2 as these tasks are particularly susceptible to the problem of conditioning on anomalies.
Revised: Experiment Settings We perform the ablation study on the anomaly localization tasks for monopoly and Polycraftv2 as these tasks are particularly susceptible to the problem of conditioning on anomalies.

Original: We vary the hyperparameter r to values of 0 , 0 . 5 , 0 .
Revised: We vary the hyperparameter r to values of 0 , 0 . 5 , 0 .

Original: and also completely remove the filter bysetting r > 1 thus using the full current step.
Revised: and also completely remove the filter by setting r > 1 thus using the full current step.

Original: Results The results shown in Figure 2 demonstrate that the error filter is necessary to extract the fullbenefit of the masked modeling module.
Revised: Results The results shown in Figure 2 demonstrate that the error filter is necessary to extract the full benefit of the masked modeling module.

Original: In both games, when r > 1 the model suffers in performanceas it conditions predictions on anomalous entries.
Revised: In both games, when r > 1 the model suffers in performance as it conditions predictions on anomalous entries.

Original: When r is small, the predictive power of the modelis decreased so detection performance suffers.
Revised: When r is small, the predictive power of the model is decreased so detection performance suffers.

Original: In Polycraftv2 r = 0 .
Revised: In Polycraftv2 r = 0 .

Original: 95 provides the best trade-off.
Revised: 95 provides the best trade-off.

Original: In
Revised: In

Original: Monopoly r = 1 provides the best trade-off.
Revised: Monopoly r = 1 provides the best trade-off.

Original: While our chosen value of r = 0 .
Revised: While our chosen value of r = 0 .

Original: 95 seems reasonableacross tasks, the optimal choice is application dependent and not easily tuned.
Revised: 95 seems reasonable across tasks, the optimal choice is application dependent and not easily tuned.