Original: Distribution Shift in Airline Customer Behavior during COVID- 
Revised: Distribution Shift in Airline Customer Behavior during COVID-

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Traditional AI approaches in customized (personalized) contextual pricing applications assume that the data distribution at the time of online pricing is similarto that observed during training.
Revised: Traditional AI approaches in customized (personalized) contextual pricing applications assume that the data distribution at the time of online pricing is similar to that observed during training.

Original: However, this assumption may be violated inpractice because of the dynamic nature of customer buying patterns, particularlydue to unanticipated system shocks such as COVID-19.
Revised: However, this assumption may be violated in practice because of the dynamic nature of customer buying patterns, particularly due to unanticipated system shocks such as COVID-19.

Original: We study the changes incustomer behavior for a major airline during the COVID-19 pandemic by framingit as a covariate shift  detection problem.
Revised: We study the changes in customer behavior for a major airline during the COVID-19 pandemic by framing it as a covariate shift and concept drift detection problem.

Original: We identify which customers changedtheir travel and purchase behavior and the attributes affecting that change using (i) Fast Generalized Subset Scanning and (ii) Causal Forests.
Revised: We identify which customers changed their travel and purchase behavior and the attributes affecting that change using (i) Fast Generalized Subset Scanning and (ii) Causal Forests.

Original: In our experiments withsimulated and real-world data, we present how these two techniques can be usedtodetect covariate shifts through qualitative analysis.
Revised: In our experiments with simulated and real-world data, we present how these two techniques can be used through qualitative analysis.

Original:  Introduction
Revised:  Introduction

Original: The novel coronavirus pandemic has had a seismic impact on many industries, including travel.
Revised: The novel coronavirus pandemic has had a seismic impact on many industries, including travel.

Original: During these unprecedented times, along with industrial operations, customer behavior also changeddrastically [1].
Revised: During these unprecedented times, along with industrial operations, customer behavior also changed drastically [1].

Original: Due to this, machine learning systems built on sequential decision making wereaffected the most.
Revised: Due to this, machine learning systems built on sequential decision making were affected the most.

Original: Machine learning applications often implicitly or explicitly assume that data setsare drawn from stationary distributions, and the sudden shift in underlying data makes the modelprone to break.
Revised: Machine learning applications often implicitly or explicitly assume that data sets are drawn from stationary distributions, and the sudden shift in underlying data makes the model prone to break.

Original: Pricing based on context is one such application that is prominently driven bycustomers’ behaviour [2, 3].
Revised: Pricing based on context is one such application that is prominently driven by customers’ behaviour [2, 3].

Original: The motivation of this work stems from examining the performance ofmachine learning models deployed to price add-on (ancillary) products at an international airline.
Revised: The motivation of this work stems from examining the performance of machine learning models deployed to price add-on (ancillary) products for an international airline.

Original: Toillustrate, the performance of one of the deployed models  dropped from 75% during training to 50%during testing despite exhaustive ofﬂine analysis with K-fold cross validation as shown in Figure 1.
Revised: To illustrate, the performance of one of the deployed models that predicts the probability of ancillary purchase dropped from 75% during training to 50% during testing despite exhaustive ofﬂine analysis with K-fold cross validation as shown in Figure 1.

Original: Since the onset of COVID-19, one of the features used in the model started experiencing distributionchange and needed further investigation.
Revised: Since the onset of COVID-19, one of the features used in the model started experiencing distribution change and needed further investigation.

Original: Demand forecasting is frequently used for price determination given customers’ context [4, 5].
Revised: Demand forecasting is frequently used for price determination given customers’ context [4, 5].

Original: Pricingapplications that rely on forecasting need a distribution shift detection as the price that a customeris willing to pay can be affected by unanticipated events, such as COVID-19, that alter the systemdynamics and change the data generating process.
Revised: Pricing applications that rely on forecasting need a distribution shift detection as the price that a customer is willing to pay can be affected by unanticipated events, such as COVID-19, that alter the system dynamics and change the data generating process.

Original: In this work, we (a) investigate  covariate shift fora forecasting model used for dynamic pricing, where the  distribution of the covariates between thetraining andtest sets differ [6], (b) explore two potential techniques for detecting covariate shift foronline use-case, and (c) discuss the proposal of a robust framework for contextual pricing applicationsin a real-world setting. [Figure]
Revised: In this work, we (a) investigate the problem of covariate shift and concept drift for a forecasting model used for dynamic pricing, where the marginal distribution of the covariates, P ( X ) , and the conditional

Original:  Distribution shifts have been extensively studied in the academic literature.
Revised: Workshop on Distribution Shifts, 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

Original: From probability theory,

Original: Kullback-Leibler divergence and f -divergence are known to measure the differences in the distributions [7, 8]. Moreover, Wald-Wolfwitz test [9] and exponentially weighted moving average [10, 11,12] based statistical approaches been proposed to detect the distrbution shifts. For small samples,techniques involving density estimation [13], margin-based approaches such as the one-class supportvector machines [14], generative adversarial nets [15] have been explored. Two-sample-testing-basedapproach which uses pre-trained classiﬁers for dimensionality reduction has shown to perform wellat detecting dataset shifts [16].
Revised: distibution, P ( Y | X ) , between the training and test sets differ [6], (b) explore two potential techniques for detecting distribution shift for online use-case, speciﬁcally, we use Fast Generalized Subset Scan [7] to detect covariate shift in terms of P ( X ) and Causal Forests [8] to identify concept drift, that is, change in P ( Y | X ) , and (c) discuss the proposal of a robust framework for contextual pricing applications in a real-world setting.

Original: Literature also addresses distribution shifts in the context of domainadaptation.

Original: Recent works in out-of-distribution (OOD) sample detection have shown the possibilityof explicitly adapting the loss to aid the task [17, 18, 19]. However, these methods can only beapplied to a batch of observations and not in real-time. Jesson et al. [20] made one of the early effortsusing causal inference for shift detection, but there is still scope for building methods to detect shiftinduced by an intervention.
Revised: [Figure]

Original:  Methods
Revised:  Methods

Original: In this section, we deﬁne how Fast Generalized Subset Scan [21] and Causal Forests [22] can be usedfor covariate shiftdetection due tothe shocks, such as COVID-19, experienced by a system.
Revised: In this section, we deﬁne how Fast Generalized Subset Scan [7] and Causal Forests [8] can be used for detecting distribution shift due to shocks, such as COVID-19, experienced by a system.

Original: 3.1 Fast Generalized Subset Scan
Revised: 2.1 Fast Generalized Subset Scan

Original: Fast Generalized Subset Scan (FGSS) is an unsupervised anomalous pattern detection technique,proposed by McFowland, Speakman, and Neill [21].
Revised: Fast Generalized Subset Scan (FGSS) is an unsupervised anomalous pattern detection technique, proposed by McFowland, Speakman, and Neill [7].

Original: We use FGSS for covariate shift detection withthe objective of ﬁnding shifted patterns in the test set. 
Revised: We use FGSS for covariate shift detection (change in P ( X ) ) with the objective of ﬁnding shifted patterns in the test set. Given a set of observations

Revised: R 1 ...R N and features A 1 ...

Original: Under the null hypothesis that there is noanomalous pattern in the test set, we use FGSS to detect self-similar groups of observations in thetest set for which some subset  of features are shifted.
Revised: A M in a test set, under the null hypothesis that there is no anomalous pattern in the test set, we use FGSS to ﬁnd a subset S ∗ = R ∗ × A ∗ of self-similar groups that are anomalous, where R ∗ ⊆ { R 1 ...R N } and A ∗ ⊆ { A 1 ...

Revised: A M } , using a scoring function F ( S ) deﬁning the anomalousness of the subset S .

Original: If the null hypothesis is true, then the test set isgenerated from the same distribution as the training set.
Revised: If the null hypothesis is true, then the test set is generated from the same distribution as the training set.

Original: Otherwise, the training and test distributionsare different with a signiﬁcantly higher score F ( S ) for a subset of observations in the test set, deﬁnedbelow.
Revised: Otherwise, the training and test distributions are different with a signiﬁcantly higher score F ( S ) for those subset of observations in the test set.

Original: [Equation]
Revised: [Equation]

Original: where N ( S ) represents the size of the subset S and N α ( S ) represents the total number of p-values(obtained by passing the observed values in the test  through the inverse CDF of the training set) thatare signiﬁcant at level α in S .
Revised: where N ( S ) represents the size of the subset S and N α ( S ) represents the total number of p-values (obtained by passing the observed values in the test set through the inverse eCDF of the training set) that are signiﬁcant at level α in S .

Original: The function φ ( α, N α ( S ) , N ( S )) is monotonically increasing w.r.t.
Revised: To efﬁciently ﬁnd the subset S ∗ = arg max F ( S ) , we need the function φ ( α, N α ( S ) , N ( S )) to be monotonically increasing w.r.t.

Original: N α , monotonically decreasing w.r.t.
Revised: N α , monotonically decreasing w.r.t.

Original: N and α , and is convex.
Revised: N and α , and be convex [7].

Original: We use the Berk-Jones statistic [23] asφ in our experiments which satisfy these properties.
Revised: We use the Berk-Jones statistic [9] as φ in our experiments which satisﬁes these properties.

Original: 3.2 Causal Forests
Revised: 2.2 Causal Forests

Original: Causal forests in a supervised method from Generalized Random Forests [22] that estimates heterogeneity in treatment effects.
Revised: Causal forests in a supervised method from Generalized Random Forests [8] that estimates heterogeneity in treatment effects.

Original: A treatment effect refers to a causal effect of a treatment or interventionon an outcome variable.
Revised: A treatment effect refers to a causal effect of a treatment or intervention on an outcome variable.

Original: If the treatment effect is heterogeneous, causal forests can be used to estimate the Conditional Average Treatment Effect (CATE).
Revised: Causal forests can be used to estimate the Conditional Average Treatment Effect (CATE).

Original: This is useful in identifying the observations forwhich the treatment is positive and that beneﬁt the most from a treatment; essentially an estimationof optimal policy assignment [24].
Revised: This is useful in identifying the observations for which the treatment is positive and that beneﬁt the most from a treatment; essentially an estimation of optimal policy assignment [10].

Original: CATE cannot be directly observed for a unit because of the“fundamental problem of causal inference” [25], making it impossible to observe unit-level causaleffects and the reason why we can never directly observe the counterfactual condition of a unit ofobservation.
Revised: CATE cannot be directly observed for a unit because of the “fundamental problem of causal inference” [11], making it impossible to observe unit-level causal effects and the reason why we can never directly observe the counterfactual condition of a unit of observation.

Original: For each observation X ∈ R m where m is the number of covariates, there are twopotential outcomes Y 1 and Y 0 corresponding to the binary treatment variable D ∈ { 0 , 1 } , but onlyone of them is observed.
Revised: For each observation X ∈ R m where m is the number of covariates, there are two potential outcomes Y 1 and Y 0 corresponding to the binary treatment variable D ∈ { 0 , 1 } , but only one of them is observed.

Original: The conditional expectation of an outcome for the treatment or control,µ d ( x ) is deﬁned as:
Revised: The conditional expectation of an outcome for the treatment or control, µ d ( x ) is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: and CATE ( τ ( x ) ) is the difference in expectation of the potential outcomes given x,
Revised: and CATE ( τ ( x ) ) is the difference in expectation of the potential outcomes given x,

Original: [Equation]
Revised: [Equation]

Original: We use causal forests to estimate the causal effect of COVID-19 intervention and detect the covariateshift in features.
Revised: We use causal forests to estimate the causal effect of COVID-19 intervention and identify the concept drift P ( Y | X ) in the system.

Original: This requires some data observed after the intervention to be leaked into the trainingdata so as to learn the unit-level interventional change in treatment.
Revised: This requires some data observed post-intervention to be used for training so as to learn the unit-level interventional change in treatment.

Original:  Experiments
Revised:  Experiments

Original: Ancillaries are optional products or services sold by businesses to complement their primary product[26].
Revised: Ancillaries are optional products or services sold by businesses to complement their primary product [12].

Original: In this work, we utilized the following datasets from the airline industry for an ancillary market:
Revised: In this work, we utilized the following datasets from the airline industry for an ancillary market:

Original: (1) simulated interaction of ancillary pricing, and (2) real-world ancillary booking requests data froma large airline containing price variability.
Revised: (1) simulated interaction of ancillary pricing, and (2) real-world ancillary booking requests data from a large airline containing price variability.

Original: 4.1 Simulated dataset
Revised: 3.1 Simulated dataset

Original: To test our approach, we generated a simulated dataset of customer and ﬂight seat (ancillary)interactions using open-sourced ﬂight simulator 1 .
Revised: To test our approach, we generated a simulated dataset of customer and ﬂight seat (ancillary) interactions using open-sourced ﬂight simulator 1 .

Original: In this dataset, we artiﬁcially varied the arrivalrates of the customers in train and test data.
Revised: In this dataset, we artiﬁcially varied the arrival rates of the customers in train and test data.

Original: The details of the data-generation process can be foundin Appendix A.1.
Revised: The details of the data-generation process can be found in Appendix A.1.

Original: Figure 2(a) shows the generated data distribution.
Revised: Figure 2(a) shows the generated data distribution.

Revised: The training set represent the control period ( d = 0 ) and test set represents the treatment period ( d = 1 ).

Original: Figures 2(b) and (c) showthe results from Causal Forests and FGSS, respectively.
Revised: Figures 2(b) and (c) show the results from Causal Forests and FGSS, respectively.

Original: FGSS is able to identify 26.01% points inthe test set as shifted.
Revised: FGSS is able to identify 26.01% points in the test set as shifted.

Original: The shifted points have AdvancedPurchase feature values greater than 5000despite the test distribution having a peak around 4000.
Revised: The shifted points have AdvancedPurchase feature values greater thandespite the test distribution having a peak around 4000.

Original: This is because the train distribution has fewpoints in the values between 0 and 4000, which the FGSS model identiﬁes as normal behavior.
Revised: This is because the train distribution has few points in the values between 0 and 4000, which the FGSS model identiﬁes as normal behavior.

Original: Onthe other hand, Causal Forests is able to identify 23.26% points with positive shift and only 0.56%points with negative shift, indicating a superior performance of Causal Forests model to identify shifton intervention.
Revised: On the other hand, Causal Forests is able to identify 23.26% points with positive shift and only 0.56% points with negative shift, indicating a superior performance of Causal Forests model to identify shift on intervention.

Original: The positive shift occurs over the range of test set values, which aligns with thedistribution of test set simulated signiﬁcantly different from the train set.
Revised: The positive shift occurs over the range of test set values, which aligns with the distribution of test set simulated signiﬁcantly different from the train set.

Original: 4.2 Real-world dataset{+[Figure] Following Equation (3), positive shift identiﬁed by Causal Forests represent the customers that are more likely to purchase an ancillary in the treatment period, ( P 1 ( Y | X = x i ) > P 
+}
Revised: {+[Figure] Following Equation (3), positive shift identiﬁed by Causal Forests represent the customers that are more likely to purchase an ancillary in the treatment period, ( P 1 ( Y | X = x i ) > P 
+}

Revised: ( Y | X = x i ) with signiﬁcance α ) and similarly negative shift represent the customers that are more likely to purchase in the control period, ( P 1 ( Y | X

Revised: = x i ) <

Original: {+P 
+}
Revised: {+P 
+}

Revised: ( Y | X = x i ) with signiﬁcance α ), while no shift represent the customers whose probability of purchase in the train and test sets are similar (

Revised: Y | X =

Original:  The real-world dataset consists of customer booking requests from March 2019 to September 2020.
Revised: x i 3.2 Real-world dataset The real-world dataset consists of customer booking requests from March 2019 to September 2020.

Original: We use this dataset to evaluate the performance of FGSS and Causal Forests in detecting distributionshift due to COVID-19 in real-world setting.
Revised: We use this dataset to evaluate the performance of FGSS and Causal Forests in detecting distribution shift due to COVID-19 in real-world setting.

Original: The airline identiﬁes the time period starting March 2020as the “COVID” era because a large portion of scheduled ﬂights started getting canceled/rescheduledand observed decreased ticket sales.
Revised: The airline identiﬁes the time period starting Marchas the “COVID” era because a large portion of scheduled ﬂights started getting canceled/rescheduled and observed decreased ticket sales.

Original: We use ﬁve features in our experiments - AdvancedPurchase ,
Revised: We use ﬁve features in our experiments - AdvancedPurchase ,

Original: LengthOfStay , GroupSize , TotalDuration and TripType .
Revised: LengthOfStay , GroupSize , TotalDuration and TripType .

Original: The description of the features can be foundin Appendix: Table 1.
Revised: The description of the features can be found in Appendix: Table 1.

Original: Figure 3 and 4 show the results obtained on the hold-out test set.
Revised: Figure 3 and 4 show the results obtained on the hold-out test set.

Original: Both Causal
Revised: Both Causal

Original: Forests and FGSS results indicate that there is signiﬁcant shift in all of the features.
Revised: Forests and FGSS results indicate that there is signiﬁcant shift in all of the features.

Original: LengthOfStayfeature has shift towards value 0 (indeﬁnite stay at destination), indicating most customers are notopting for vacation/business travel but essential movement only.
Revised: LengthOfStay feature has shift towards value 0 (indeﬁnite stay at destination), indicating most customers are not opting for vacation/business travel but essential movement only.

Original: The same conclusion can be drawnfrom the shift observed in TripType feature with customers preferring to travel one-way.
Revised: The same conclusion can be drawn from the shift observed in TripType feature with customers preferring to travel one-way.

Original: GroupSizefeature has a shift towards value 1 , indicating customers are traveling alone and not in large groupsduring the pandemic.
Revised: GroupSize feature has a shift towards value 1 , indicating customers are traveling alone and not in large groups during the pandemic.

Original: AdvancedPurchase has a shift towards smaller values indicating customers are not booking tickets way ahead of the travel date due to the uncertain nature of the pandemic.
Revised: AdvancedPurchase has a shift towards smaller values indicating customers are not booking tickets way ahead of the travel date due to the uncertain nature of the pandemic.

Original: TotalDuration has a shift towards smaller values as well, indicating the operational changes made by the airline to operate on shorter routes during the pandemic [Figure]
Revised: TotalDuration has a shift towards smaller values as well, indicating the operational changes made by the airline to operate on shorter routes during the pandemic.

Original:  Conclusion
Revised:  Conclusion

Original: We explored two techniques, Fast Generalized Subset Scan and Causal Forests, for covariate shiftdetection when a system experiences a shock.
Revised: We explored two techniques, Fast Generalized Subset Scan and Causal Forests, for covariate shift detection when a system experiences a shock.

Original: We applied these techniques to an airline ancillarypurchase use case and saw that some of the features used for ancillary pricing experienced a shiftduring COVID-19.
Revised: We applied these techniques to an airline ancillary purchase use case and saw that some of the features used for ancillary pricing experienced a covariate shift while the probability of purchase experienced a concept drift during COVID-19.

Original: We identiﬁed the observations in the test set that follow a signiﬁcantly differentdistribution compared to the training set.
Revised: We identiﬁed the observations in the test set that follow a signiﬁcantly different distribution compared to the training set.

Original: Causal forests, while able to detect a shift in real-time,need a signiﬁcant amount of data post-intervention for training.
Revised: Causal forests, while able to detect a concept drift in real-time, need a signiﬁcant amount of data post-intervention for training.

Original: On the other hand, FGSS can onlydetect shifts on a batch of data but does not need data post-intervention for training.
Revised: On the other hand, FGSS can only detect covariate shift on a batch of data but does not need data post-intervention for training.

Original: Hence, thereis a possibility to combine the two approaches, both addressing the limitations of one another, fora uniﬁed and robust covariate shift detection.
Revised: Hence, there is a possibility to combine the two approaches for a uniﬁed and robust distribution shift detection.

Original: At the same time, the construction of a good testingpipeline, along with an agreement by multiple models as well as a visual inspection, can jointly helpidentify shift patterns.
Revised: At the same time, the construction of a good testing pipeline, along with an agreement by multiple models as well as a visual inspection, can jointly help identify shift patterns.

Original: These patterns can then be cross-checked with designers’ hypothesis anddomain experts’ knowledge for validation and potential changes to the system before being correctedfor enhanced model performance.
Revised: These patterns can then be cross-checked with designers’ hypothesis and domain experts’ knowledge for validation and potential changes to the system before being corrected for enhanced model performance.

Original: In the future, we aim to extend this work to other applications, testmultiple shift correction approaches and provide recommendations for adapting to a shift induced bysudden shocks to the system (see Appendix A.2 for details). 
Revised: In the future, we aim to extend this work to other applications, test multiple shift correction approaches and provide recommendations for adapting to a shift induced by sudden shocks to the system (see Appendix A.2 for details). Acknowledgments and Disclosure of Funding

Revised: We sincerely and gratefully acknowledge our airline partners for their continuing support.

Revised: The academic partners are also thankful to deepair (www.deepair.io) for funding this research.

Original: A AppendixA.1 Additional Information
Revised: A Appendix A.1 Additional Information

Original: The simulated dataset consists of interactions for selecting ﬂight seat after the ticket is purchased.
Revised: The simulated dataset consists of interactions for selecting ﬂight seat after the ticket is purchased.

Original: Customer arrivals are simulated using non-homogeneous poisson process [27].
Revised: Customer arrivals are simulated using non-homogeneous poisson process [13].

Original: Multinomial logitis used for customer choice model in the simulator [28].
Revised: Multinomial logit is used for customer choice model in the simulator [14].

Original: The offered prices are randomly sampledfrom minimum to maximum allowed price for that ﬂight.
Revised: The offered prices are randomly sampled from minimum to maximum allowed price for that ﬂight.

Original: The intervention is artiﬁcially createdby changing the customer arrival rate.
Revised: The intervention is artiﬁcially created by changing the customer arrival rate.

Original: We have simulated 2 datasets with different parameters ofthe poisson process for 10 long-distance ﬂights as train and test (Fig. 2(a)).
Revised: We have simulated 2 datasets with different parameters of the poisson process for 10 long-distance ﬂights as train and test (Fig. 2(a)).

Original: We use the arrival rate( AdvancedPurchase ) and number of seats sold ( Sold ) as features, with AdvancedPurchase featurehaving an apparent covariate shift.
Revised: We use the arrival rate ( AdvancedPurchase ) and number of seats sold ( Sold ) as features, with AdvancedPurchase feature having an apparent covariate shift.

Original: For Causal Forests, we leak 40% points randomly sampled fromthe test set into training.
Revised: For Causal Forests, we leak 40% points randomly sampled from the test set into training.

Original: Table 1 presents the description of features discussed in Section 4.2.
Revised: Table 1 presents the description of features discussed in Section 3.2.

Original: FGSS requires all features to becategorical.
Revised: FGSS requires all features to be categorical.

Original: We discretize all numeric features by binning them at quartiles for FGSS training.
Revised: We discretize all numeric features by binning them at quartiles for FGSS training.

Original: Wedivide the real-world data into two parts: pre-COVID
Revised: We divide the real-world data into two parts: pre-COVID

Original: (March 2019 to February 2020) and COVID(March 2020 to September 2020).
Revised: (March 2019 to February 2020) and COVID (March 2020 to September 2020).

Original: We use the pre-COVID period for training FGSS and Causal
Revised: We use the pre-COVID period for training FGSS and Causal

Original: Forests model.
Revised: Forests model.

Original: Further, we leak March 2020 to August 2020 COVID data into training for Causal
Revised: Further, we leak March 2020 to August 2020 COVID data into training for Causal

Original: Forests, and leave September 2020 as the hold-out test set to report results.
Revised: Forests, and leave September 2020 as the hold-out test set to report results.

Original: [Table]
Revised: [Table]

Original: A.2 Robust Pricing Model
Revised: A.2 Robust Pricing Model

Original: Shukla et al.
Revised: Shukla et al.

Original: [3] proposed a two-stage pricing model that uses the purchase probability prediction forprice recommendation.
Revised: [3] proposed a two-stage pricing model that uses the purchase probability prediction for price recommendation.

Original: Figure 5 shows the proposed pricing framework with shift detection beforethe supervised model prediction.
Revised: Figure 5 shows the proposed pricing framework with shift detection before the supervised model prediction.

Original: The shift detection layer ﬁrst checks if there is a shift in the systemor not.
Revised: The shift detection layer ﬁrst checks if there is a shift in the system or not.

Original: Causal Forests being a supervised technique can be used to detect shift "on-line".
Revised: Causal Forests being a supervised technique can be used to detect shift "on-line".

Original: However,
Revised: However,

Original: Causal Forests require data observed after intervention for training and identiﬁcation of the start ashift induced by an intervention to the system is usually a difﬁcult task.
Revised: Causal Forests require data observed after intervention for training and identiﬁcation of the start a shift induced by an intervention to the system is usually a difﬁcult task.

Original: We propose to take advantageof the unsupervised nature of FGSS to achieve that.
Revised: We propose to take advantage of the unsupervised nature of FGSS to achieve that.

Original: FGSS can learn the normal behavior fromhistorical data and run in batch mode to detect anomalous patterns during testing.
Revised: FGSS can learn the normal behavior from historical data and run in batch mode to detect anomalous patterns during testing.

Original: If the percentage ofobserved anomalous patterns breach a user-deﬁned threshold for a signiﬁcant period of time, Causal
Revised: If the percentage of observed anomalous patterns breach a user-deﬁned threshold for a signiﬁcant period of time, Causal

Original: Forests can use that data as treatment for training.
Revised: Forests can use that data as treatment for training.

Original: Once the shift has been detected in "real-time", it can be handled in multiple ways: a single modeltrained after reweighting shifted points, multiple models - one for each shifted pattern, or heuristicsbased on domain knowledge to better price the product.
Revised: Once the shift has been detected in "real-time", it can be handled in multiple ways: a single model trained after reweighting shifted points, multiple models - one for each shifted pattern, or heuristics based on domain knowledge to better price the product.

Original: We aim to perform further experiments infuture to test the feasibility of these 3 approaches and propose an efﬁcient strategy to handle covariateshift for decision-making under uncertainty.
Revised: We aim to perform further experiments in future to test the feasibility of these 3 approaches and propose an efﬁcient strategy to handle covariate shift for decision-making under uncertainty.