B ACK 2F UTURE : L EVERAGING B ACKFILL D YNAMICS FOR I MPROVING R EAL - TIME P REDICTIONS IN F UTURE
A BSTRACT
For real-time forecasting in domains like public health and macroeconomics, data collection is a non-trivial and demanding task.
Often after being initially released, it undergoes several revisions later (maybe due to human or technical constraints) - as a result, it may take weeks until the data reaches a stable value.
This socalled ‘backfill’ phenomenon and its effect on model performance have been barely addressed in the prior literature.
In this paper, we introduce the multi-variate backfill problem using COVID-19 as the motivating example.
We construct a detailed dataset composed of relevant signals over the past year of the pandemic.
We then systematically characterize several patterns in backfill dynamics and leverage our observations for formulating a novel problem and neural framework,
Back2Future that aims to refines a given model’s predictions in real-time.
Our extensive experiments demonstrate that our method refines the performance of diverse set of top models for COVID-19 forecasting and GDP growth forecasting.
Specifically, we show that Back2Future refined top COVID-19 models by 6.65% to 11.24% and yield 18% improvement over non-trivial baselines.
In addition, we show that our model improves model evaluation too; hence policy-makers can better understand the true accuracy of forecasting models in real-time
I NTRODUCTION
The current COVID-19 pandemic has challenged our response capabilities to large disruptive events, affecting the health and economy of millions of people.
A major tool in our response has been forecasting epidemic trajectories which enabled policymakers to plan interventions (Holmdahl & Buckee, 2020).
Broadly two classes of approaches have been devised: traditional mechanistic epidemiological models (Shaman & Karspeck, 2012; Zhang et al., 2017), and the fairly newer statistical approaches (Brooks et al., 2018; Adhikari et al., 2019; Osthus et al., 2019b) including deep learning models (Adhikari et al., 2019; Panagopoulos et al., 2021; Rodríguez et al., 2021a), which have become among the top-performing ones for multiple forecasting tasks (Reich et al., 2019).
These models also use newer digital indicators like search queries (Ginsberg et al., 2009; Yang et al., 2015) and social media (Culotta, 2010).
Epidemic forecasting is still a challenging enterprise (Metcalf & Lessler, 2017; Biggerstaff et al., 2018) because it is affected by weather, mobility, strains, and others.
However, real-time forecasting also brings new challenges.
As noted in multiple CDC real-time forecasting initiatives for diseases like flu (Osthus et al., 2019a) and COVID-19 (Cramer et al., 2021), as well as in macroeconomics (Clements & Galvão, 2019; Aguiar, 2015) the initially released public health data is revised many times after and is known as the ’backfill’ phenomenon.
The various factors that affect backfill are multiple and complex, ranging from surveillance resources to human factors like coordination between health institutes and government organizations within and across regions (Chakraborty et al., 2018;
Reich et al., 2019; Altieri et al., 2021; Stierholz, 2017).
While previous works have addressed anomalies (Liu et al., 2017), missing data (Yin et al., 2020), and data delays (Žliobaite, 2010) in general time-series problems, the backfill problem has not been addressed.
In contrast, the topic of revisions has not received as much attention, with few exceptions.
For example in epidemic forecasting, a few papers have either (a) mentioned about the ‘backfill problem’ and its effects on performance (Chakraborty et al., 2018; Rodríguez et al., 2021b; Altieri et al., 2021; Rangarajan et al., 2019) and evaluation (Reich et al., 2019); or (b) proposed to address the problem via simple models like linear regression (Chakraborty et al., 2014) or ’backcasting’ (Brooks et al., 2018) the observed targets.
The related problem of nowcasting involves prediction of revised stable value of current week’s target from sequence of right-truncated past values [Aditya: cite one paper ].
Prior works have used data assimilation and sensor fusion from a readily available stable set of features to refine unrevised features for accurate nowcasting (Farrow, 2016; Osthus et al., 2019a).
However, most methods focus only on revisions in the target and typically study in the context of influenza forecasting, which is substantially less noisy and more regular than the novel COVID-19 pandemic or assume access to stable values for some features which is not the case for COVID-19.
In economics, Clements & Galvão (2019) surveys several domain-specific (Carriero et al., 2015) or essentially linear techniques for data revision/correction behavior of several macroeconomic indicators (Croushore, 2011).
Motivated from above, we study the challenging problem of multi-variate backfill for both features and targets.
We go further beyond prior work and also show how to leverage our insights towards a general neural framework to improve model predictions and performance evaluation (i.e. rectification of current target from the evaluator’s perspective).
Our specific contributions are the following:
• Multi-variate backfill problem: We introduce the multi-variate backfill problem using real-time epidemiological forecasting as the primary motivating example.
In this challenging setting, which generalizes (the limited) prior work, the forecast targets, as well as exogenous features, are subject to retrospective revision.
Using a carefully collected diverse dataset for COVID-19 forecasting for the past year, we discover several patterns in backfill dynamics, show that there is a significant difference in real-time and revised feature measurements, and highlight the negative effects of using unrevised features for incidence forecasting in different models both for model performance and evaluation.
Building on our empirical observations, we formulate the problem B FRP , which aims to ‘correct’ given model predictions to achieve better performance on eventual fully revised data.
• Spatial and Feature level backfill modeling to refine model predictions: Motivated by the patterns in revision and observations from our empirical study, we propose a deep-learning model
Back2Future (B2F) to model backfill revision patterns and derive latent encodings for features.
B2F combines Graph Convolutional Networks that capture sparse, cross-feature, and cross-regional backfill dynamics similarity and deep sequential models that capture temporal dynamics of each features’ backfill dynamics across time.
The latent representation of all features is used along with the history of the model’s predictions to improve diverse classes of models trained on real-time targets, to predict targets closer to revised ground truth values.
Our technique can be used as a ‘wrapper’ to improve model performance of any forecasting model (mechanistic/statistical).
• Refined top models’ predictions and improved model evaluation: We perform an extensive empirical evaluation to show that incorporating backfill dynamics through B2F consistently improves the performance of diverse classes of top-performing COVID-19 forecasting models (from the CDC COVID-
Forecast Hub, including the top-performing official ensemble) significantly.
B2F also enables forecast evaluators and policy-makers better evaluate the ‘eventual’ true accuracy of participating models (against revised ground truth).
This allows the model evaluators to quickly estimate models that perform better w.r.t revised stable targets instead of potentially misleading current targets.
Our methodology can also be further adapted for nowcasting and other general time-series forecasting problems.
We also show the generalizability of our framework and model B2F to other domains by significantly improving predictions of non-trivial baselines for US National GDP forecasting (Marcellino, 2008).
N ATURE OF BACKFILL DYNAMICS
In this section we study important properties of the revision dynamics of our signals.
We introduce some concepts and definitions to aid in the understanding of our empirical observations and method.
Real-time forecasting.
We are given a set of signals F = Reg × Feat , where Reg is the set of all regions (where we want to forecast) and set Feat contains our features and forecasting target(s) for each region.
At prediction week t , x ( t ) i, 1: t is a time series from 1 to t for feature i , and the set of all signals results in the multi-variate time series X ( t ) 1: t 1 .
Similarly, Y ( t ) 1: t is the forecasting target(s) time series.
Further, let’s call all data available at time t , D ( t ) 1: t = {X ( t ) 1: t , Y ( t ) 1: t } as real-time sequence .
For clarity we refer to ‘signal’ i ∈ F as a sequence of either a feature or a target, and denote it as d ( t ) i, 1: t .
Thus, at prediction week t , the real-time forecasting problem is: Given D ( t ) 1: t , predict next k values of forecasting target(s), i.e. ˆ y t +1: t + k .
Typically for CDC settings and this paper, our time unit is week, k
= 4 (up to 4 weeks ahead)
and our target is COVID-19 mortality incidence ( Deaths ).
Revisions.
Data revisions (‘backfill’) are common.
At prediction week t + 1 , the real-time sequence
D ( t +1) 1: t +1 is available.
In addition to the length of the sequences increasing by one (new data point), values of D ( t +1) 1: t +1 already in D ( t ) 1: t may be revised i.e., D ( t ) 1: t ̸ = D ( t +1) 1: t .
Note that previous work has studied backfill limited to Y ( t ) , while we address it in both X ( t ) and Y ( t ) .
Also, note that the data in the backfill is the same used for real-time forecasting, but just seen from a different perspective.
Backfill sequences: Another useful way we propose to look at backfill is by focusing on revisions of a single value.
Let’s focus on value of signal i at an observation week t ′ .
For this observation week, the value of the signal can be revised at any t > t ′ , which induces a sequence of revisions.
We refer to revision week r ≥ 0 as the relative amount of time that has passed since the observation week t ′ .
Defn.
(Backfill Sequence BS EQ )
For signal i and observation week t ′ , its backfill sequence is
BS EQ ( i, t ′ ) =
⟨ d ( t ′ )
i,t ′ , d ( t ′ +1)
i,t ′ , . . .
, d ( ∞ ) i,t ′ ⟩ , where d ( t ′ )
i,t ′ is the initial value of the signal and d ( ∞ )
i,t ′ is the final/stable value of the signal.
Defn.
(Backfill Error BE RR ) For revision week r of a backfill sequence, the backfill error is
BE RR ( r, i, t ′ )
=
| d ( t ′ + r ) i,t ′ − d ( ∞ )
i,t ′ | / | d ( ∞ ) i,t ′ | .
Defn.
(Stability time ST IME ) of a backfill sequence BS EQ is the revision week r ∗ that is the minimum r for which the backfill error BE RR < ϵ for all r > r ∗ , i.e., the time when BS EQ stabilizes.
Note: We ensured that BS EQ length is at least 7, and found that in our dataset most signals stabilize before r = 20 .
For d ( ∞ ) i,t ′ , we use d ( t f ) i,t ′ , at the final week t f in our revisions dataset.
In case we do not find BE RR < ϵ in any BS EQ , we set ST IME to the length of that BS EQ .
We use ϵ = 0 .
Example:
For BS EQ { 223 , 236 , 236 , 404 , . . . , 404 } , BE RR for third week is | 236 − 404 |= 0 .
41 and ST IME is 4.
2. D ATASET DESCRIPTION
[Table]
We collected and pre-processed important publicly available signals from a variety of trusted sources that are relevant to COVIDforecasting to form the COVID-
Surveillance Dataset ( CoVDS ).
See Table 1 for the list of 20 features ( | Feat | = 21 , including
Deaths ). We collected revised features every week from Aprilto July 2021.
Our analysis covers 30 observation weeks from June 2020 to December 2020 (to ensure all our backfill sequences are of length at least 7) for all | Reg | = 50 US states.
The rest of the unseen data from Jan 2021 to July 2021 is used strictly for evaluation.
Patient line-list: traditional surveillance signals used in epidemiological models (Chakraborty et al., 2014; Brooks et al., 2018) derived from line-list records e.g. hospitalizations from CDC (CDC, 2020), positive cases, ICU admissions from COVID Tracking (COVID-Tracking, 2020).
Testing: measure changes in testing from CDC and COVIDTracking used by Rodríguez et al.
(2021b).
Mobility: quantify change in people’s movement to several point of interests (POIs); derived from mobility reports released by Google (2020); Apple (2020).
Exposure: digital signal measuring closeness between people at POIs,(Chevalier et al., 2021)
Social Survey: used by (Wang et al., 2020; Rodríguez et al., 2021b)
CMU/Facebook
Symptom Survey Data contains self-reported responses about COVID-19 symptoms.
2. O BSERVATIONS
We first study different facets of the significance of backfill in CoVDS .
Using our definitions, we generate a backfill sequence for every combination of signal, observation week, and region (not all signals are available for all regions).
In total, we generate more than 30 , 000 backfill sequences.
Backfill error BE RR is significant.
We computed BE RR for the initial values, i.e., BE RR ( r = 0 , i, t ′ ) , for all signals i and observation weeks t ′ .
Obs.
( BE RR across signals and regions)
Compute the average BE RR for each signal; the median of all these averages is 32% , i.e. at least half of all signals are corrected by 32% of their initial value.
Similarly in at least half of the regions the signal corrections are 280% of their initial value.
We also found large variation of BE RR .
For features (Figure 1a), compare avg.
BE RR = 1743% of five most corrected features with 1 .
6% of the five least corrected features.
Also, in contrast to related work that focuses on traditional surveillance data Yang et al. (2015), perhaps unexpectedly, we found that digital indicators also have a significant BE RR (average of 108% ).
For regions (see Figure 1b), compare 1594% of the five most corrected regions with 38% of the five least corrected regions.
[Figure]
Stability time ST IME is significant.
A similar analysis for ST IME found significant variation across signals (from 1 weeks for to 21 weeks for COVID-19, see Figure 1c for ST IME across feature types) and regions (from 1.55 weeks for GA to 3.83 weeks for TX, see Figure 1d).
This also impacts our target, thus, actual accuracy is not readily available which undermines real-time evaluation and decision making.
Obs. 2.
( ST IME of features and target) Compute the average ST IME for each signal; the average of all these averages for features is around 4 weeks and for our target Deaths is around 3 weeks, i.e. on average, it takes over 3 weeks to reach the stable values of features.
Backfill sequence BS EQ patterns.
There is significant similarity among BS EQ s. We cluster BS EQ s via K-means using Dynamic Time Warping (DTW) as pair-wise distance (as DTW can handle sequences of varying magnitude and length).
We found five canonical categories of behaviors (see
Figure 2), each of size roughly 11.58% of all BS EQ s.
Also, each cluster is not defined only by signal
[Figure]
Obs. 3.
( BS EQ similarity and variety)
Five canonical behaviors were observed in our backfill sequences (Figure 2).
No cluster has over 21% of BS EQ s from the same region, and no cluster has over 14% of BS s signal.
[Figure]
Model performance vs BE RR .
To study the relationship between model performance (via Mean Absolute Error MAE of a prediction) and BE RR , we use
R EV D IFF MAE: the difference between MAE computed against real-time target value and one against the stable target value.
We analyze the top-performing realtime forecasting models as per the comprehensive evaluation of all models in COVID-
Forecast Hub (Cramer et al., 2021).
YYG and UM ASS -MB are mechanistic while CMU-TS and GT-DC are statistical models.
The top performing E NSEMBLE is composed of all contributing models to the hub.
We expect a well-trained real-time model will have higher R EV D IFF MAE with larger BE RR in its target (Reich et al., 2019).
However, we found that higher BE RR does not necessarily mean worse performance.
See Figure 3—YYG has even better performance with more revisions.
This may be due to the more complex backfill activity/dependencies in COVID in comparison to the more regular seasonal flu.
Obs. 4.
(Model performance and backfill) Relation between BE RR and R EV D IFF MAE can be non-monotonous and positively or negatively correlated depending on model and signal.
Real-time target values to measure model performance : Since targets undergo revisions (5%
BE RR on average), we study how this BE RR affects the real-time evaluation of models.
From
Figure 4, we see that the scores are not similar with real-time scores over-estimating model accuracy.
The average difference in scores is positive which implies that evaluators would overestimate models’ forecasting ability.
Obs. 5.
MAE evaluated at real-time overestimates model performance by 9.6 on average, with the maximum for TX at 22.63.
R EFINING THE FUTURE VIA B2F
[Figure]
Our observations naturally motivate improving training and evaluation aspects of real-time forecasting by leveraging revision information.
Thus, we propose the following two problems.
Let predictions of model M for week t + k be y
( M, k ) t .
Since models are trained on real-time targets, y ( M, k ) t is the model’s estimate of target y
( t + k ) t + k .
k -week ahead Backfill Refinement Problem, B FRP k
( M ) : At prediction week t , we are given a revision dataset {D ( t ′ ) 1: t ′ } t ′ ≤ t , which includes our target Y ( t ) 1: t .
For a model M trained on real-time targets, given history of model’s predictions till last week ⟨ y ( M, k ) 1 , . . .
y ( M, k ) t − 1 ⟩ and prediction for current week y ( M, k ) t , our goal is to refine y ( M, k ) t to better estimate the stable target y ( t f ) t + k , i.e. the ’future’ of our target value at t + k .
Leaderboad Refinement problem, L BRP : At each week t , evaluators are given a current estimate of our target y ( t ) t and forecasts of models submitted on week t − k .
Our goal is to refine y ( t ) t to ˆ y t , a better estimate of y ( t f ) t , so that using ˆ y t as a surrogate for y ( t f ) t to evaluate predictions of models provides a better indicator of their actual performance (i.e., we obtain a refined leaderboard of models).
Since L BRP aims to refine the current estimate of the target, it is closely related to the nowcasting problem.
L BRP is also a special case of B FRP : Assume a hypothetical model M eval whose predictions are real-time ground truth, i.e. y ( M eval , 0) t
= y ( t ) t , ∀ t .
Then, refining M eval is equivalent to refining y ( t ) t to better estimate y ( t f ) t which leads to solving L BRP .[-Thus, L BRP is a special case of B FRP 
 ( M eval ) .-]
Overview: We leverage observations from Section 2 to derive Back2Future
(B2F), a deep-learning model that uses revision information from BS EQ to refine predictions.
Obs. 1 and 2 show that realtime values of signals are poor estimates of stable values.
Therefore, we leverage patterns in BS EQ of past signals and exploit cross-signal similarities (Obs. 3) to extract information from BS EQ s. We also consider that the relation of models’ forecasts to BE RR of targets is complex (Obs. 4 and 5) to refine their predictions.
B2F combines these ideas through its four modules: • G RAPH G EN : Generates a signal graph (where each node maps to a signal in Reg × Feat ) whose edges are based on BS EQ similarities.
• B SEQ E NC :
Leverages the signal graph as well as temporal dynamics of BS EQ s to learn a latent representation of BS EQ
s using a Recurrent Graph Neural Network.
• M ODEL P RED E NC :
Encodes the history of the model’s predictions, the real-time value of the target, and past revisions of the target through a recurrent neural network.
• R EFINER :
Combines encodings from B SEQ E NC and
M ODEL P RED E NC to predict the correction to model’s real-time prediction.
In contrast to previous works that studies target BE RR (Reich et al., 2019), we simultaneously model all BS EQ available till current week t using spatial and signal similarities in the temporal dynamics of BS EQ .
Recent works that attempt to model spatial relations for COVID19 forecasting need explicitly structural data (like cross-region mobility)
(Panagopoulos et al., 2020) to generate a graph or use attention over temporal patterns of regions’ death trends.
B2F, in contrast, directly models the structural information of signal graph (containing features from each region) using
BS EQ similarities.
Thus, we first generate useful latent representations for each signal based on
BS EQ revision information of that feature as well as features that have shown similar revision patterns in the past.
Due to the large number of signals that cover all regions, we cannot model the relations between every pair using fully connected modules or attention similar to (Jin et al., 2020).
Therefore, we first construct a sparse graph between signals based on past BS EQ similarities.
Then we inject this similarity information using Graph Convolutional Networks (GCNs) and combine it with deep sequential models to model temporal dynamics of BS EQ of each signal while combining information from BS EQ s of signals in the neighborhood of the graph.
Further, we use these latent representations and leverage the history of a model M ’s predictions to refine its prediction.
Thus, B2F solves B FRP k
( M ) assuming M is a black box, accessing only its past forecasts.
Our training process, that involves pre-training on model-agnostic auxiliary task, greatly improves training time for refining any given model M .
The full pipeline of B2F is also shown in Figure 5.
Next, we describe each of the components of B2F in detail.
For the rest of this section, we will assume that we are forecasting k weeks ahead given data till current week t .
[Figure]
G RAPH G EN generates an undirected signal graph
G t =
( V, E t ) whose edges represent similarity in BS EQ s between signals, where vertices
V = F
=
Reg ×
Feat .
We measure similarity using
DTW distance due to reasons described in Section 2.
G RAPH G EN leverages the similarities across BS EQ patterns irrespective of the exact nature of canonical behaviors which may vary across domains.
We compute the sum of DTW distances of BS EQ s for each pair of nodes summed over t ′ ∈ { 1 , 2 , . . .
, t − 5 } .
We threshold t ′ till t − 5 to make the BS EQ s to be of reasonable length (at least 5) to capture temporal similarity without discounting too many BS EQ s. Top τ node pairs with lowest total DTW distance are assigned an edge.
B SEQ E NC .
While we can model backfill sequences for each signal independently using a recurrent neural network, this doesn’t capture the behavioral similarity of BS EQ across signals.
Using a fully-connected recurrent neural network that considers all possible interactions between signals also may not learn from the similarity information due to the sheer number of signals ( 50 × 21 = 1050 ) while greatly increasing the parameters of the model.
Thus, we utilize the structural prior of graph G t generated by G RAPH G EN and train an autoregressive model B SEQ E NC which consists of graph recurrent neural network to encode a latent representation for each of backfill sequence in
B t = { BS EQ ( i, t ) : i ∈ F} .
At week t , B SEQ E NC is first pre-trained and then it is fine-tuned for a specific model M (more details later in this section).
Our encoding process is in Figure 5.
Let BS EQ t ′ + r
( i, t ′ ) be first r + 1 values of BS EQ ( i, t ′ )
(till week t ′ + r ).
For a past week t ′ and revision week r , we denote h ( t ′ r ) i,t [-(cid:
 )-]′ ∈
R m to be the latent encoding of BS EQ t ′ r
( i, t ′ ) where t ′ r
= t ′ + r and t ′ ≤ t ′ r ≤ t .
We initialize h (0) i,t ′ for any observation week t ′ to be a learnable parameter h (0) i ∈ R m specific to signal i .
For each week t ′ r we combine latent encoding h ( t ′ r − 1) i,t ′ and signal value d ( t ′ r ) i,t ′ using a GRU (Gated Recurrent Unit) (Cho et al., 2014) cell to get the intermediate embedding v ( t ′ )
i,t .
Then, we leverage the signal graph G t and pass the embeddings { v ( t ′ r ) i,t ′ : i ∈ F} through a Graph Convolutional layer (Kipf & Welling, 2016) to get h ( t ′ r ) i,t ′ : v ( t ′ r ) i,t ′ =
GRU BE ( d ( t ′ r ) i,t ′ , h ( t ′ r − 1) i,t ′ ) , [Equation] t [Equation]
i
[Equation]
Thus, h ( t ′ r ) i,t ′ contains information from BS EQ t ′ r ( i, t ′ ) and structural priors from G t .
Using h ( t ′ r ) i,t ′ ,
B SEQ E NC predicts the value d ( t ′ r +1)
i,t ′ by passing through a 2-layer feed-forward network FFN i : ˆ d ( t ′ r +1) i,t ′ =
FFN i ( h ( t ′ r ) i,t ′ ) .
During inference, we only have access to real-time values of signals for the current week.
We autoregressively predict h ( t + l ) i,t for each signal by initially passing { d ( t ) i,t } i ∈F through B SEQ E NC and using the output { ˆ d ( t +1)
i,t } i ∈F as input for B SEQ E NC .
Iterating this l times we get { h ( t + l ) i,t } i ∈F along with { ˆ d ( t + l ) i,t } i ∈F where l is a hyperparameter.
M ODEL P RED E NC .
To learn from history of a model’s predictions and its relation to target revisions,
M ODEL P RED E NC encodes the history of model’s predictions, previous real-time targets, and revised (up to current week)
targets using a Recurrent Neural Network.
Given a model M , for each observation week t ′ ∈ { 1 , 2 , . . .
, t − 1 − k } , we concatenate the model’s predictions y
( M, k ) t ′ , real-time target y ( t ′
[Equation]
k ) t ′ + k as seen on observation week t ′ and revised target y ( t ) t ′
+ k as C tt ′
=
y ( M, k ) t ′ ⊕ y [Equation] [Equation] k
[Equation]
y [Equation] [Equation] + k , where ⊕ is the concatenation operator.
A GRU is used to encode the sequence { C ( t ) 1 , . .
, C ( t ) t − 1 − k } : { [Equation] , z [Equation]
t ) t
[Equation]
R EFINER .
It leverages the information from above three modules of B2F to refine model M ’s prediction for current week y
( M, k ) t .
Specifically, it receives the latent encodings of signals { h ( t + l ) i,t } i ∈F from B SEQ E NC , z ( t ) t − k − 1 from M ODEL P RED E NC , and the model’s prediction y
( M, k ) t for week t .
BS EQ encoding from different signals may have variable impact on refining the signal since a few signals may not very useful for current week’s forecast (e.g., small revisions in mobility signals may not be important in some weeks).
Moreover, because different models use signals from CoVDS differently, we may need to focus on some signals over others to refine its prediction.
Therefore, we first take attention over BS EQ encodings from all signals { h ( t + l ) i,t } i ∈F w.r.t y ( M, k ) t .
We use multiplicative attention mechanism with parameter w ∈ R m based on Vaswani et al.
[Equation]
Finally we combine ¯ h ( t ) and z ( t ) t − k − 1 through a two layer feed-forward layer FNN RF which outputs a 1-dim value followed by tanh activation to get the correction γ t ∈ [ − 1 , 1] i.e., γ t = tanh(FFN RF (¯ h ( t ) ⊕ z ( t ) t − k − 1 )) .
Finally, the refined prediction is y ∗ ( M, K ) t =
( γ + 1) y
( M, K ) t .
Note that we limit the correction by B2F by at most the magnitude of model’s prediction because the average BE RR of targets is 4 .
9% and less than 0 .
6% of them have BE RR over 1.
Therefore, we limit the refinement of prediction to this range.
Training: There are two steps of training involved for B2F: 1) model agnostic autoregressive BS EQ prediction task to pre-train B SEQ E NC ; 2) model-specific training for B FRP .
Autoregressive BS EQ prediction: Pre-training on auxiliary tasks to improve the quality of latent embedding is a well-known technique for deep learning methods (Devlin et al., 2019;
Radford et al., 2018).
We pre-train B SEQ E NC to predict the next values of backfill sequences { x ( t ′ r +1) t ′ ,i } i ∈F .
Note that we only use BS EQ sequences { BS EQ t ( t ′ , i ) } i ∈F ,t ′ <
t available till current week t for training
B SEQ E NC .
The training procedure in itself is similar to Seq2Seq prediction problems (Sutskever et al., 2014) where for initial epochs we use the ground truth inputs at each step (teacher forcing) and then transition to using output predictions of previous time step by the recurrent module as input to next time step.
Once we pre-train B SEQ E NC , we can use it for B FRP as well as L BRP for current week t for any model M .
Fine-tuning usually takes less than half the epochs required for pre-training enabling quick refinement of multiple models in parallel.
Model specific end-to-end training:
Given the pre-trained B SEQ E NC , we train, end-to-end, the parameters of all modules of B2F.
The training set consists of past model predictions ⟨ y ( M, k ) 1 , y ( M, k ) 2 , . . .
y ( M, k ) t − 1 ⟩ and backfill sequences { BS EQ t ( t ′ , i ) } i ∈F ,t ′ ≤ t .
For datapoint y ( M, k ) t ′ of week t ′ <
t − k , we input backfill sequences of signals whose observation week is t ′ into B SEQ E NC to get latent encodings { h ( t ) t ′ ,i } i ∈F .
We also derive z ( t ) t ′ from M ODEL P RED E NC and finally R EFINER ingests z ( t ) t ′ , { h tt ′ ,i } i ∈F and y ( M, k ) t ′ to get γ t ′ .
Overall, we optimize the loss function:
L ( t ) =
(cid:80) t − k − 1 i = (cid:16) γ t ′ y
( M, k ) t ′ − y ( t ) t ′
+ k (cid:17) 2 .
Following real-time forecasting, we train B2F each week from scratch (including pre-training).
Throughout training and forecasting for week t , we use G t as input to B SEQ E NC since it captures average similarities in BS EQ s till current week t .
B ACK 2F UTURE E XPERIMENTAL R ESULTS
In this section, we describe a detailed empirical study to evaluate the effectiveness of our framework
B2F. All experiments were run in an Intel i7 4.8 GHz CPU with Nvidia Tesla A4 GPU.
The model typically takes around 1 hour to train for all regions.
The appendix contains additional details (all hyperparameters and results for June-Dec 2020 and k = 1 , 3 and GDP forecasting).
We also release the code and datasets at www.github.com/AdityaLab/Back2Future .
Setup: We perform real-time forecasting of COVID -19 related mortality ( Deaths ) for 50 US states.
We leveraged observations (Section 2) from BS EQ
We tuned the model hyperparameters using data from June 2020 to Aug. 2020 and tested it on the rest of dataset including completely unseen data from Jan. 2021 to June 2021.
For each week t , we train the model using the CoVDS dataset available till the current week t (including BS EQ s for all signals revised till t ) for training.
As described in Section 3, for each week, we first pre-train B SEQ E NC on
BS EQ data
and then train all components of B2F for each model we aim to refine.
Then, we predict the forecasts Deaths y ∗ ( M, k ) t for each model M .
Similarly we also evaluated B2F for real-time
GDP forecasting task with detailed results in Appendix.
We observed that setting hyperparameter τ = c |F| where c ∈ { 2 , 3 , 4 , 5 } provided best results.
Note that τ influences the sparsity of the graph as well as the efficiency of the model since sparser graphs lead to fast inference across GConv layers.
We also found setting l = 5 provided the best performance.
Evaluation: Refined prediction y ∗ ( M, K ) t are evaluated against the most revised version of the target y ( t f ) t ′ + k , where t f is second week of Feb 2021.
For evaluation, we use standard metrics in this domain Reich et al.
; Adhikari et al.
Let absolute error of prediction e ( M, k ) t = | y
( t f ) t + k − y ∗
M, K ) t | for a week t and model M .
We use (a) Mean Absolute Error MAE( M ) =
1 T ′ (cid:80) T ′ i =1 | e ( M, k ) t | and (b) Mean Absolute Percentage Error MAPE = 1 T ′ (cid:80) T ′ i =1 e ( M, k ) t / | y ( t f ) t + k | .
Candidate models: We focus on refining/rectifying the top models from the COVID-19 Forecast
Hub described in Section 2; these represent different variety of statistical and mechanistic models.
MAPE scores averaged over all regions from Jan 2021 to June
[Table]
Baselines:
Due to the novel problem, there are no standard baselines.
Therefore, our baselines are used to study the effect of using BS EQ s and model bias(MB) on refinement.
(a) FFN: train a simple feed-forward neural network for regression task that takes as inputs model’s prediction and real-time target to predict the stable target
(b)
B2F-MB: exploits only model bias and uses the M ODEL P RED E NC architecture and append a linear layer that takes encodings from M ODEL P RED E NC and model’s prediction (c) B2F-B SEQ : exploits only BS EQ without model bias and only uses B SEQ E NC architecture and append a linear layer that takes encodings from B SEQ E NC and model’s prediction (d) B2F-N O G RAPH : does not use BS EQ similarity from
G RAPH G EN by removing graph convolutional layers and retain only RNNs.
Refining real-time model-predictions: We compare the mean percentage improvement (decrease) in scores of B2F refined predictions of diverse set of top models w.r.t stable targets over 50 US states.
We observe that B2F is the only method, compared to baselines, that improves scores for all candidate models consistently (Table 2). The poor scores of baselines also shows the necessity of incorporating both backfill information (unlike FFN and B2F-MB) and model prediction history (unlike B2F-B SEQ and B2F-N O G RAPH ).
We achieve substantial avg.
improvements of 6.93% and 6.79% in MAE and MAPE respectively with low standard deviation across 29 test weeks which shows that the improvements were consistent across time and not just over few weeks that experienced large revision anomalies.
Candidate models refined by B2F show improvement of over 10% in over 25 states and over 15% in 5 states (NJ, LA, GA, CT, MD).
The improved predictions of CMU-TS and GT-DC (ranked 3rd and 4th in [-COVID-
-]COVID-
Forecast Hub) due to B2F, outperform all the models in the hub (except for E NSEMBLE ) with 7.17% and 4.13% improvements in MAE respectively.
UM ASS -MB, ranked 2nd, is improved by 11.24%.
B2F also improves E NSEMBLE , the current best-performing model of the hub, by 3.6% - 5.18% with over 5% improvement in 38 states, and with IL and TX experiencing over 15% improvement. [Figure] decrease of MAE ( [Figure]
) % improve. in MAE for each week significantly within Rectifying real-time model-evaluation: We evaluate the efficacy of B2F in rectifying the real-time evaluation scores for the L BRP problem.
We noted in Obs 5 that real-time MAE was lower than stable MAE by 9.6 on average.
The difference between B2F rectified estimates MAE stable MAE was reduced to 4.4, a 51.7% decrease (Figure 7).
This results in increased MAE scores across most regions towards stable estimates.
Eg: We reduce the MAE difference in large highly populated states such as GA by 26.1% (from 22.52 to 16.64) and TX by 90% (from 10.8 to 1.04) causing an increase in MAE scores from real-time estimates by 5.88 and 9.4 respectively.
Refinement as a function of data availability: During the initial weeks of the pandemic, we have access to very little revision data both in terms of length and number of BS EQ .
So we evaluate the mean performance improvement for each week across all regions (Figure 6b).
B2F’s performance ramps up and quickly stabilize in just 6 weeks.
Since signals need around 4 weeks (Obs 2) to stabilize, this ramp-up time using small amount of revision data is impressive.
[Figure]
are closer to stable MAE
B2F adapts to anomalies During real-time forecasting, models need to be also robust to rare events of large anomalous data revisions, like during the initial stages of the pandemic when data collection was not fully streamlined.
Consider observation week 5 where there was an abnormally large revision to deaths nationwide (Figure 8a) when the BE RR was 48%.
B2F still provided significant improvements of up to 74.2% for most model predictions (Figure 8b).
B2F refines GDP forecasts To evaluate the extensibility of B2F to other domains that encounter the problem of backfill, we tested on the task of forecasting US National GDP using 25 macroeconomic indicators from past and their revision history for years 2000-2021.
We found that B2F improves predictions of candidate models by 6%-15% and significantly outperforms baselines.
The details of the dataset and results are found in Appendix Sections B, E.2.
C ONCLUSION
[Figure]
We introduced the important and challenging multi-variate backfill problem using COVID-19 and GDP forecasting as examples.
We compiled and released the comprehensive CoVDS dataset to study revision patterns in features and targets as well as aid in
COVID-19 forecasting.
We presented
Back2Future (B2F), the novel deep-learning method to model this phenomenon, which exploits our observations of cross-signal similarities using Graph Recurrent Neural Networks to refine predictions and rectify evaluations for a wide range of models.
Our extensive experiments showed that leveraging similarity among backfill patterns as well as model bias via our proposed method leads to significant 6 - 11% improvements in all the top models.
As future work, our work can potentially help improve data collection and alleviate systematic differences in reporting capabilities across regions.
For example, B2F provides significant gains consistently across time including when there are large anomalies in data.
Therefore, our revision modelling approach can be helpful for anomaly detection (Homayouni et al., 2021).
We can also study how backfill can affect uncertainty calibration in time-series analysis (Yoon et al., 2020).
Adapting to situations where data revisions can occur at different frequencies is another research direction.
E THICS S TATEMENT
The features used in the CoVDS dataset and for GDP forecasting are publicly available and anonymized without any sensitive information.
Our backfill refinement framework and B2F is generalizable to any domain that deals with real-time prediction tasks with feature revisions.
Due to the relevance of our dataset to public health and macroeconomics, prospects for misuse should not be discounted.
The disparities in data collection across features and regions can also have implications on equity of prediction performance and is an interesting direction of research.
R EPRODUCIBLITY S TATEMENT
As described in Section 4, we evaluated our model over 5 runs with different random seeds to show the statistical significance of our method.
We also provide a more extensive description of hyperparameters and data pre-processing in the Appendix.
The code for B2F and the CoVDS dataset is publicly available at https://github.com/AdityaLab/Back2Future .
A CKNOWLEDGEMENTS
This work was supported in part by the NSF (Expeditions CCF-1918770, CAREER IIS-2028586,
RAPID IIS-2027862, Medium IIS-1955883, Medium IIS-2106961, CCF-2115126), CDC MInD program, ORNL, and faculty research awards from Facebook, funds/computing resources from
Georgia Tech.
[-B2F clearly outperforms all baselines and provides similar improvements for COVID-
-]