On-the-ﬂy learning of adaptive strategies with bandit algorithms
Abstract
Automation of machine learning model development is increasingly becoming an established research area.
While automated model selection and automated data pre-processing have been studied in depth, there is, however, a gap concerning automated model adaptation strategies for streaming data with non-stationarities.
This has previously been addressed by heuristic generic adaptation strategies in the batch streaming setting.
While showing promising performance, these strategies contain some limitations.
In this work, we propose using multi-armed bandit algorithms for learning adaptive strategies from incrementally streaming data on-the-ﬂy.
Empirical results using established bandit algorithms show a comparable performance to two common stream learning algorithms.
1. Introduction
Machine learning on non-stationary streaming data is a common situation part of which is adapting the model to changes in the underlying data generating process (DGP).
While automated model selection and automated data pre-processing have been studied in depth, there is, however, a gap concerning automated model adaptation strategies for streaming data with non-stationarities.
In many situations one cannot expect a static model trained on historical data to maintain performance as time proceeds.
Here adaptation is deﬁned as changes in model training set , parameters and structure all designed to track changes in the underlying DGP over time.
To cope with non-stationary data, many proposed algorithms for machine learning on streaming data involve one or more mechanisms for adapting the model, which will be further referred to as adaptive mechanisms (AM) .
Deploying various AMs greatly increases the performance of models, however, in most of the cases, the AMs deployment choice (which we will refer to as adaptation strategy ) is tied to the custom algorithm design choice.
With the multitude of available AMs, the design of the adaptation strategy can be a tedious and time consuming task.
Despite advances in automated machine learning, we note that there is a gap concerning automated development of models’ adaptation strategy.
This has been previously addressed using meta level heuristic adaptation strategies (Bakirov et al., 2021).
While showing promising results, these strategies had some shortcomings; they a) required a batch setting, b) increased the runtime and c) did not provide any theoretical performance guarantees.
Here, we address these issues by learning the adaptation strategies on-the-ﬂy, this is, during the data stream process using the theoretically sound multi-armed bandit algorithms on incremental data.
Multi-armed bandit algorithms are designed to maximise a cumulative reward during a sequential process where decisions are made at each step from a limited set of possibilities.
This is ideal for solving our problem as adaptation strategies seek to minimise the mean loss over many steps using a small set of AMs.
In addition, the theoretical properties of the bandit algorithms provide some optimality guarantees for adaptive strategies.
In this paper, we ﬁrst formulate the AM selection in the bandit scenario.
We then test several prominent bandit algorithms for the Dynamic Weighted Majority (DWM) (Kolter and Maloof, 2007) and Paired Learner (PL) (Bach and Maloof, 2010) strategies for AM selection.
We compare the results to the original DWM algorithm using 26 synthetic and 7 real-world classiﬁcation data sets.
We conclude empirically that the bandit based adaptive strategies leads to comparable performance with the custom strategies.
Our contributions are a) a novel formulation of AM selection as a bandit learning problem, and b) an empirical analysis of relevant bandit learning algorithms on AM selection.
2. Background
Recently AutoML for streaming non-stationary data has became a focus of the community’s attention.
An approach to adaptation to changing environments was proposed in (Martín Salvador et al., 2016) where repeated automated deployment of Auto-WEKA for Multi-Component Predictive Systems (MCPS) to learn from new batches of data was used for life-long learning and the adaptation of complex MCPS when applied to changing streaming data from process industries.
Celik and Vanschoren (2021) represent a development of this idea with the inclusion of drift detection and the experimentation using several open source AutoML frameworks.
An interesting approach closely tied with the Auto_Sklearn is described in (Madrid et al., 2019) where authors propose using the ensemble nature of this framework to deal with streaming data, by adapting the weights of experts and adding new ones.
Biedenkapp et al.
(2020) report promising results using reinforcement learning for dynamic algorithm conﬁguration in online settings.
Wu et al.
(2021) introduce a champion-challenger scheme for online AutoML, replacing an existing model with dynamically created new models with bounds on regret.
Bakirov et al.
(2021) describe the framework of model adaptation with multiple adaptive mechanisms (AMs) and propose heuristics algorithms for automated adaptive strategies based on ﬂexible deployment of these AMs.
3. Formulation
To formalise the adaptation with multiple AMS, we use the framework from (Bakirov et al., 2021), adopted for incremental learning.
We consider the predictive method at time t as a function ˆ y t = f t ( x t , Θ f ) .
where ˆ y t is the prediction, f t is the prediction function, and Θ f is the associated parameter set.
Our estimate, f t , evolves via adaptation with each t -th data instance.
We denote the a-priori predictive function at batch t as f − t , and the a-posteriori predictive function, i.e. the adapted function given the observed output, as f + t .
An adaptive mechanism , g ( · ) , may thus formally be deﬁned as an operator which generates an updated prediction function based on the instance { x t , y t } and other optional inputs.
This can be written as g t ( x t , y t , Θ g , f − t , ˆ y t ) : f − t → f + t or alternatively as f + t = f − t ◦ g t for conciseness.
Note f − t and ˆ y t are optional arguments and Θ g is the set of parameters of g .
The function is propagated onto the next data instance as f − t +1 = f
+ t and predictions themselves are always made using the a-priori function f − t .
We examine a situation when a choice of multiple, diﬀerent AMs, {∅ , g 1 , ..., g H } = G , is available.
Any AM g h t ⊂ G can be deployed after each data instance arrival, where h t denotes the AM deployed for t -th data instance.
As the history of all adaptations up to the data instance, t , have in essence created f − t , we call that sequence g h 1 , ..., g h t an adaptation sequence .
Note that we also include the option of applying no adaptation denoted by ∅ , thus all adaptive algorithms having at least the option of not adapting ﬁt multiple AMs framework.
In this formulation, only one element of G is applied for each data instance.
Deploying multiple adaptation mechanisms on the same batch are accounted for with their own symbol in G .
Using data instance { x t , y t } for adaptation, it is possible to obtain H predictive models, f − t ◦ g 1 , · · · , f − t ◦ g H .
After the true label is revealed, the reward of the selected model in f − t ◦ g h t , r h t can be calculated 1 .
Thus, the task of coming up with the adaptation strategy amounts to the selection of a predictive model from F t (equivalent to the selection of g h t from G ) for all t ≥ 2 .
It is easy to note that if we assume a ﬁxed unknown reward distribution for each AM, the problem of learning an adaptive strategy is a classical multi-armed bandit problem (Lattimore and Szepesvári, 2020) where the arms are the set G of AMs.
Multi-armed bandit algorithms bound regret , that is the diﬀerence between the chosen and the single best action’s reward, sub-linearly depending on the number of rounds T under some conditions.
For adaptive strategy learning, a sub-linear regret bound means that the cumulative reward R t = T (cid:80) t
=1 r t using the bandit algorithm for the choice of
AMs will converge to this of the optimal single AM selection.
There is a caveat which makes the bounds of classical bandit algorithms not applicable 2 to the
AM selection problem.
These algorithms assume that the distribution of AMs’ rewards are time invariant.
This assumption would not hold in stream data scenario as reward distributions will depend on the context such as non-stationarities in data and the choice of previously deployed AMs.
This can be handled by the contextual bandit algorithms (Tewari and Murphy, 2017), which take additional information such as these into account.
4. Experiments
Our experiments are based on the DWM and PL algorithms.
DWM is a dynamic ensemble and PL a combination of stable and reactive learners, both very common settings for the stream learning.
Both of these settings include multiple AMs, not all of which are used in DWM and PL.
For our experiments we will use both settings; a) using all AMs and b) as using only the AMs which were part of the original algorithms ( custom AMs ).
These options simulate prior information available at the design of adaptive strategy; if known, the poor performing AMs would typically not be included in the set G .
Without such prior information, it would make sense to use all available AMs for adaptive strategy learning.
Used datasets are given in supplementary materials.
Below we present the details of stream learning methods and the used bandit algorithms.
4.1 Dynamic Weighted Majority style adaptation
DWM is a dynamic weighted classiﬁers ensemble which enables all three adaptation possibilities common to these methods; the individual retraining or incremental learning of the experts with new data, adjustment of the weights and addition/removal of experts.
These can be combined in all possible ways resulting in 8 AMs.
We will consider 6 of them as all possible AMs 3 .
AM1 (No adaptation).
No changes are applied to the predictive model, corresponding to ∅ .
AM2 (Incremental Learning).
Each predictor is updated with a new data instance.
AM3 (Weights Update and Experts Pruning).
Weights of predictors which misclassify
the current data instance
are
decreased and experts
with
weights lower than ﬁxed threshold are removed.
AM4 .
AM2 (Weights Update and Experts Pruning) followed by AM1 (Incremental Learning).
AM5 .
AM2 (Weights Update and Experts Pruning) followed by the creation of a new expert.
AM6 .
AM2 (Weights Update and Experts Pruning) followed by AM1 (Incremental Learning) followed by the creation of a new expert.
DWM original adaptive Strategy.
At time t , after an arrival of new instance { x t , y t } , predictors make predictions and the ﬁnal label is calculated as shown earlier in this section.
Then, all predictors learn on this instance and update their weights (AM4) and the ﬁnal prediction is calculated.
If an instance is misclassiﬁed, a new predictor with the weight of one is created (AM6).
4.2 Paired Learner style adaptation
PL maintains two learners - a stable learner which is updated with all of incoming data and which is used to make predictions, and a reactive learner, which is trained only on a window of the most recent data.
For this method, three adaptive mechanisms are possible, which are described below.
AM1 (No adaptation).
No changes are applied to the predictive model, corresponding to ∅ .
AM2 (Updating stable learner).
Stable learner is updated with a new data instance.
AM3 (Switching to reactive learner).
Stable learner is discarded and replaced by reactive learner.
PL (custom adaptive strategy).
Original PL adaptive strategy revolves around comparing the accuracy values of stable ( u ts ) and reactive ( u tr ) learners on each batch of data.
Every time when u ts < u tr a change counter is incremented.
If the counter is higher than a deﬁned threshold θ , an existing stable learner is discarded and replaced by the reactive learner, while the counter is set to 0.
As before, a new reactive learner is continued to be trained with the most recent data.
4.3 Bandit algorithms
For learning of adaptive strategies we have experimented with four bandit algorithms, including two variants of LinUCB listed below.
For each of these algorithms we consider a cold-start scenario, where we start with uniform estimated rewards per AM and a warm-start (denoted below with (w)), where we run the algorithm on a dataset beforehand twice and use the resulting estimated rewards to report the outcome of the third run 4 .
The warm start scenario simulates processing extended data streams as well as reduces the randomness eﬀects in algorithms, all of which use random selection for the cases when two or more AMs have the same estimated reward.
The most basic out of three, (cid:15) -greedy algorithm (Lattimore and Szepesvári, 2020) estimates the rewards of the actions at time t as mean accumulated reward for this action: E ( r g )
=
R g,t t .
Then, the action with the highest reward is chosen with a probability of 1 − (cid:15) , (so called exploitation) and a random action is chosen with a probability of (cid:15) (exploration).
The second bandit algorithm is Kullback-Leibler Upper Conﬁdence Bounds (KL-UCB) (Garivier and Cappé, 2011), a superior variation of the standard Upper Conﬁdence Bounds (UCB) algorithm (Lattimore and Szepesvári, 2020).
UCB chooses the action with a highest upper conﬁdence bound of reward at each time step.
While time-invariant rewards assumption that KL-UCB has does not hold in our case, meaning that its reward bounds are not applicable, it still provides good results.
To satisfy the condition of rewards’ dependency on non-stationarity and previous actions (the context), we include a contextual bandit algorithm LinUCB (linear UCB)
(Li et al., 2010).
It assumes a linear relationship between the context and the reward, which is estimated via ridge regression.
For a standard version of LinUCB, at time t we include two context variables, the result of the last classiﬁcation (1 if ˆ y t − 1 =
y t − 1 , 0 otherwise) and the last deployed action h t − 1 .
For experiments with sets of all possible AMs both for DWM and PL adaptations styles, we have also used hybrid LinUCB, which also includes the context on the actions.
LinUCB has the regret bound of ˜ O ( √ KdT ) where K is the number of arms and d is number of context features.
5. Results 5.1 Dynamic Weighted Majority
Synthetic data.
The results of the experiments using DWM AMs are shown via Nemenyi plots on Figure 1a,b. With all available AMs, DWM is a leader, however KL-UCB(w) and LinUCB
Hybrid(w) achieve statistically comparable performance.
LinUCB algorithms show a relatively poor performance.
For DWM custom AMs DWM again has the best mean performance rank.
This time LinUCB algorithms provide comparable outcome.
DWM
KL-UCB(w)
LinUCB Hyb(w)
LinUCB Hyb eps-greedy(w)
LinUCB
LinUCB(w)
KL-UCB eps-greedy
CD
Friedman p = 2.5862e- (a) DWM style all AMs
DWM
LinUCB(w)
LinUCB eps-greedy(w)
KL-UCB(w) eps-greedy
KL-UCB
CD
Friedman p = 0.
(b) DWM style Custom AMs
LinUCB Hyb(w)
KL-UCB(w)
PL
LinUCB Hyb
KL-UCB
LinUCB eps-greedy eps-greedy(w)
LinUCB(w)
CD
Friedman p = 9.7838e- (c) PL style all AMs
KL-UCB(w)
PL
LinUCB(w)
KL-UCB eps-greedy(w) eps-greedy
LinUCB
CD
Friedman p = 2.2773e-
(d) PL style custom AMs
Real data.
Results on real data on all AMs (Table 1a) show DWM having the top accuracy for most of the datasets. However, only in three cases, the performance of PL is more than 1% higher than the runner-up algorithm.
Among bandit based methods KL-UCB(w) appears to be a strong contender, while (cid:15) -greedy is lagging behind.
The situation is diﬀerent when considering custom AMs only (Table 2a), as algorithms don’t have to test the “weak” AMs (e.g. “do nothing") and suﬀer a possible reward penalty.
Here LinUCB algorithms, especially LinUCB(w) are leaders, with KL-UCB also showing good performance.
5.2 Paired Learner
Synthetic data.
The results of the experiments using PL AMs are shown via Nemenyi plots on
Figure 1c,d. With all available AMs many bandit based adaptive strategies achieve comparable average performance to the original PL algorithm, with both LinUCB Hybrid(w) and KL-UCB(w) being slightly better.
LinUCB and (cid:15) -greedy
based strategies were less successful for this case.
Similarly for PL custom AMs KL-UCB(w) shows a slightly better performance.
Real data.
Results on real data on all AMs (Table 1b) show a slightly diﬀerent picture, with PL having the top accuracy for most of the datasets.
However, the performance of PL is more than 1% higher than the runner-up algorithm only in two cases.
Among bandit based methods both LinUCB and KL-UCB appear to be strong contenders, while (cid:15) -greedy is lagging behind.
Similar situation is observed using custom AMs only (Table 2b).
Here KL-UCB is a strong runner up to PL .
6. Conclusions
In this paper we have used several bandit algorithms for learning the adaptive strategies for streaming data given a set of adaptive mechanisms on-the-ﬂy.
These methods provide easy automation of the adaptation process, avoiding the need to come up with the custom adaptation strategy, some of [Table] [Table] them oﬀering theoretical guarantees in addition.
The results show that some bandit algorithms, such as LinUCB and KL-UCB compare favourably and often beat the custom streaming methods, particularly with warm-start.
These algorithms are not computationally intensive, with the most time consuming operation being an inverse calculation of d 2 dimensional matrix at each step for
LinUCB.
Further research includes identifying a well-suited set of context features for contextual bandit algorithms, for example considering a window of recent actions instead of only the last one.
Another direction is considering a full-information setting, where all of AM would be updated at each time step instead of limited-information bandit setting, where a reward estimate of only the chosen AM is updated.
This is likely to result in faster convergence.
To improve the convergence transfer-learning type methods can be considered as well, where rewards learned on one dataset can be used for another dataset
7. Supplementary material
[Table] [Table]
Gaussian  4x52.05% move 0.04% overlap
Gaussian  4x52.05% move 10.39% overlap
Gaussian  8x27.63% move 0.04% overlap Gaussian  8x27.63% move 10.39% overlap Gaussian  20x11.25% move 0.04% overlap Gaussian  20x11.25% move 10.39% overlap
Gaussian  4x52.05% move 0.013% overlap
Gaussian  4x52.05% move 10.24% overlap
Gaussian  8x27.63% move 0.013% overlap
Gaussian  8x27.63% move 10.24% overlap
Gaussian  20x11.25% move 0.013% overlap
Gaussian  20x11.25% move 10.24% overlap
[Figure]
RotatingHyperplane
SwitchingGaussian
PassingGaussian
MovingGaussian
[Table]
The task is prediction of hour of the day (03:00, 10:00, 17: and 21:00) based on supplied and transferred power measured in Italy.
(Zhu, 2010; Chen et al., 2015).
[Table]
ceptive Contraceptive dataset from UCI repository (Newman et al., 1998) with artiﬁcially added drift (Minku et al., 2010).
[Table]
Iris dataset (Anderson, 1936; Fisher, 1936) with artiﬁcially added drift (Minku et al., 2010).
Yeast
Yeast dataset from UCI repository (Newman et al., 1998) with artiﬁcially added drift (Minku et al., 2010).
Gas
Dataset from chemical sensors utilized in simulations for discrimination task of 6 gases at various levels of concentrations (Vergara et al., 2011).
Gestures
Dataset composed by features extracted from 7 videos with people gesticulating, for Gesture Phase Segmentation.
(Madeo et al., 2013).
Electricity
Widely used concept drift benchmark dataset thought to have seasonal and other changes as well as noise.
Task is the prediction of whether electricity price rises or falls while inputs are days of the week, times of the day and electricity demands (Harries, 1999).