Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting
Anonymous Author(s)
AffiliationAddress email
Abstract
Transformers have shown great power in time series forecasting due to their globalrange modeling ability.
However, their performance can degenerate terribly onnon-stationary real-world data in which the joint distribution changes over time.
Previous studies primarily adopt stationarization to reduce the non-stationarity oforiginal series for better predictability.
But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting.
This problem, termed over-stationarization in this paper, leads Transformers togenerate indistinguishable temporal attentions for different series and impedesthe predictive capability of deep models.
To tackle the dilemma between seriespredictability and model capability, we propose Non-stationary Transformers as ageneric framework with two interdependent modules: Series Stationarization and
De-stationary Attention.
Concretely, Series Stationarization unifies the statistics ofeach input and converts the output with restored statistics for better predictability.
To address over-stationarization, De-stationary Attention is devised to recover theintrinsic non-stationary information into temporal dependencies by approximatingdistinguishable attentions learned from unstationarized series.
Our Non-stationary
Transformers framework consistently boosts mainstream Transformers by a largemargin, which reduces 49.43% MSE on Transformer, 47.34% on Informer, and46.89% on Reformer, making them the state-of-the-art in time series forecasting.
Introduction
Time series forecasting has become increasingly ubiquitous in real-world applications, such asweather forecasting, energy consumption planning, estimation of future illness cases and financialrisk assessment.
Recently, Transformers [32] have achieved progressive breakthrough on extensiveareas [11, 12, 10, 22].
Especially in time series forecasting, credited to their stacked structure and thecapability of attention mechanisms, Transformers can naturally capture the temporal dependenciesamong time points [36, 17, 20, 34], thereby fitting the series forecasting task perfectly.
Despite the remarkable architectural design, it is still challenging for Transformers to predict realworld time series because of the non-stationarity of data.
Non-stationary time series are characterizedby the continuous change of statistical properties and joint distribution over time [6, 14].
From theaspect of data, non-stationarity makes the time series less predictable [6, 14].
From the view of deeplearning, it is a fundamental but challenging problem to make deep models generalize well on avarying distribution [26, 19, 5].
Therefore, it is generally acknowledged to pre-process time series byelimination of non-stationarity, and the related research has become a hot topic [24, 27, 15]. Thesestudies attenuate the non-stationarity of real-world time series in order to obtain more predictableseries and provide relatively stable data distribution for deep models.
[Figure]
However, non-stationarity is the constitutional property of real-world time series that can be entangledwith essential temporal dependencies for forecasting.
Experimentally, we observe that training onthe stationarized series will undermine the distinction of attentions learned by Transformers.
Asshown in Figure 1(b), even though input series are clearly distinct, Transformers [32] trained on thestationarized series still tends to generate quite similar attentions, and the reason for that is what wecall over-stationarization .
The unexpected side-effect will make Transformers fail to capture eventfultemporal dependencies, limit the model’s predictive ability, and even induce the model to generateoutputs with huge non-stationarity deviation from the ground truth.
Thus, how to increase the seriesstationarity towards better predictability and mitigate the over-stationarization problem for modelcapability simultaneously is the key problem to further improve the performance of forecasting.
In this paper, we explore the effect of stationarization in time series forecasting and propose Nonstationary Transformers as a general framework, which empowers Transformer [32] and its efficientvariants [17, 36, 34] with great predictive ability for real-world time series.
The proposed framework involves two interdependent modules: Series Stationarization to increase the predictabilityof non-stationary series and De-stationary Attention to alleviate over-stationarization.
Technically,
Series Stationarization adopts a simple but effective normalization strategy to unify the statisticsof each series without extra parameters.
And De-stationary Attention approximates the attentionof unstationarized data and restitutes the intrinsic non-stationarity of raw series.
Benefiting fromthe above designs, Non-stationary Transformers can take advantage of the great predictability ofstationarized series and crucial temporal dependencies discovered from original non-stationary data.
Our method achieves state-of-the-art performance on six real-world benchmarks and can generalizeto various Transformers for further improvement on base models.
The contributions lie in three folds:
• We refine that the predictive capability of non-stationary series is essential in real-world
forecasting.
By detailed analysis, we find out that current stationarization approaches will lead to the over-stationarization problem, limiting the predictive capability of Transformers.
• We propose Non-stationary Transformers as a generic framework, including Series Sta- tionarization to make the series more predictable and De-stationary Attention to avoid the
over-stationarization problem by re-incorporating the non-stationarity of original series. • Non-stationary Transformers consistently boosts four mainstream Transformers by a largemargin and achieves state-of-the-art performance on six real-world benchmarks.
Related Work
2.1 Deep Models for Time Series Forecasting
In recent years, deep models with elaboratively designed architectures have achieved great progressin time series forecasting.
RNN-based models [33, 35, 23, 29, 30] are proposed for application in anautoregressive manner for sequence modeling, but the recurrent structure can suffer from modeling long-term dependency.
Soon afterward, Transformer [32] emerges and shows great power in sequencemodeling.
To overcome the quadratic computation growth on sequence length, subsequent worksaim to reduce Self-Attention’s complexity.
Especially in time series forecasting, Informer [36]extends Self-Attention with KL-divergence criterion to select dominant queries.
Reformer [17]introduces local-sensitive hashing (LSH) to approximate attention by allocated similar queries.
Not only improved by reduced complexity, the following models further develop delicate buildingblocks for time series forecasting.
Autoformer [34] fuses the decomposition blocks into a canonicalstructure and develops Auto-Correlation to discover series-wise connections.
Pyraformer [21] designspyramid attention module (PAM) to capture temporal dependencies with different hierarchies.
Otherdeep but Transformer-free models also achieve remarkable performance.
N-BEATS [25] proposesthe explicit decomposition of trend and seasonal terms with strong interpretability.
N-HiTS [9]introduces hierarchical layout and multi-rate sampling for tackling time series with respectivefrequency bands.
In this paper, different from previous works focusing on architectural design, weanalyze the series forecasting task from the basic view of stationarity, which is the essential propertyof time series [6, 14].
It is also notable that as a general framework, our proposed Non-stationary Transformers can be easily applied to various Transformer-based models.
2.2 Stationarization for Time Series Forecasting
While stationarity is the key to the predictability of time series [6, 14], real-world series always presentnon-stationarity.
To tackle this problem, the classical statistical method ARIMA [7, 8] stationarizesthe time series through differencing.
As for deep models, since the distribution-varying problemaccompanied by non-stationarity makes deep forecasting even more intractable, stationarizationmethods are widely explored and always adopted as the pre-processing for deep model inputs.
Adaptive Norm [24] applies z-score normalization for each series fragment by global statistics of asampled set.
DAIN [27] employs a nonlinear neural network to adaptively stationarize time serieswith observed training distribution.
RevIN [15] introduces a two-stage instance normalization [31]that transforms model input and output respectively to reduce the discrepancy of each series.
Incontrast, we find out that directly stationarizing time series will damage the model’s capability ofmodeling specific temporal dependency.
Therefore, unlike previous methods, in addition to thestationarization, Non-stationary Transformers further develops De-stationary Attention to bring theintrinsic non-stationarity of the raw series back to attention.
Non-stationary Transformers
As aforementioned, stationarity is the key to time series predictability.
However, previous “wipingoff non-stationarity” designs obviously neglect inherent properties of real-world series, which willresult in the over-stationarization problem as stated in Figure 1.
To deal with the dilemma, we gobeyond previous works and propose Non-stationary Transformers as a generic framework.
Nonstationary Transformers involves two complementary parts: Series Stationarization to enhance thestationarity of time series for better predictability, and De-stationary Attention to re-incorporate nonstationary information of raw series to prevent over-stationarization.
Empowered by these designs,
Non-stationary Transformers can improve data predictability and model capability simultaneously.
3.1 Series Stationarization
Non-stationary time series make the forecasting task intractable for deep models because it is hardfor them to generalize well on series with changed statistics during inference, typically varied meanand standard deviation.
The pilot work, RevIN [15] applies instance normalization with learnableaffine parameters to each input and restores the statistics to the corresponding output, which makeseach series follow a unified distribution.
Experimentally, we find that this design also works wellwithout learnable parameters.
Thus, we propose a more straightforward but effective design to wrap
Transformers as the base model without extra parameters, naming by Series Stationarization.
As isshown in Figure 2, it contains two corresponding operations: Normalization module at first to dealwith the non-stationary series caused by varied mean and standard deviation, and De-normalizationmodule at the end to transform the model outputs back with original statistics.
Here are the details.
[Table]
quunQTLIKrkoioy6KbLivYB9Qik+m0DebFZFIopbj1B9zqZ4l/oH/hnTEFtYhOSHLm3HPuzL3XSwI/VY7zWrAWFpeWV4qra+sbm1vbpZ3dZhpnkosGj4NYtj2WisCPREP5KhDtRAoWeoFoeXeXOt4aCZn6cXStxonohmwQ+X2fM0VU5yZkashZMKlNb0tlp+KYZc8DNwdl5Ksel15wgx5icGQIIRBBEQ7AkNLTgQsHCXFdTIiThHwTF5hijbwZqQQpGLF39B3QrpOzEe11ztS4OZ0S0CvJaeOQPDHpJGF9mm3imcms2d9yT0xOfbcx/b08V0iswpDYv3wz5X99uhaFPs5NDT7VlBhGV8fzLJnpir65/aUqRRkS4jTuUVwS5sY567NtPKmpXfeWmfibUWpW73muzfCub0kDdn+Ocx40jyvuacW9OilXL/JRF7GPAxzRPM9QRQ11NEzHH/GEZ6tujaypdf8ptQq5Zw/flvXwAXjZk+U=</latexit>
H
Normalization module To guarantee the stationarity of inputs, we conduct normalization on thetemporal dimension by a sliding window over time.
For each input series x = [ x 1 , x 2 , ..., x S ] T ∈ 122 R S × C , we transform it into standard Gaussian distribution by translation and scaling operations andobtain x ′ =
[ x ′ 1 , x ′ 2 , ..., x ′ S ] T ∈ R S × C , where S and C denote the sequence length and variablenumber respectively.
The Normalization module can be formulated as follows:
[Equation]
where µ x , σ x ∈ R C × 1 , 1 σ x means the element-wise division and ⊙ is the Hadamard Product.
Notethat Normalization module eliminates the statistical discrepancy among each input time series, makingthe model insusceptible to varied statistics.
De-normalization module As shown in Figure 2, after the base model H predicting the future valuewith length-
O , we adopt De-normalization to transform the model output y ′ =
[ y ′ 1 , y ′ 2 , ..., y ′
O ] T ∈
R O × C with σ x and µ x and obtain ˆ y =
[ˆ y 1 , ˆ y 2 , ..., ˆ y O ] T as the eventual forecasting results.
The
De-normalization module can be formulated as follows:
[Equation]
By means of the two-stage transformation, the base models will receive stationarized inputs thatfollow a relatively stable distribution and become equivariant to translational and scaling perturbanceof time series, thereby benefiting real-world series forecasting.
3.2 De-stationary Attention
While the statistics of each time series are explicitly restored to the corresponding prediction, thenon-stationarity of the original series can not be fully recovered only by De-normalization.
For aninstance, Series Stationarization can cause the same stationarized input x ′ from distinct time seriesx 1 , x 2 ( i.e. x 2 =
α x 1 + β ), and the base model will get identical attention that fail to capture crucialtemporal dependencies entangled with non-stationarity (Figure 1).
In other words, the underminedeffects caused by over-stationarization happen inside the deep model, especially in the calculation ofattention.
Furthermore, since non-stationary time series are fragmented and normalized into eachinput series following the same distribution, the model is more likely to generate over-stationary anduneventful outputs, which is irreconcilable with the natural non-stationarity of the original series.
To tackle the trivial attention caused by Series Stationarization, we propose a novel De-stationary
Attention mechanism, which can approximate the attention that is obtained without stationarizationand discover the particular temporal dependencies from original non-stationary data.
Analysis of the plain model As mentioned above, the over-stationarization problem is caused bythe vanishment of constitutional non-stationarity information, which will make the base model fail to capture eventful temporal dependencies for forecasting.
Therefore, we try to approximate the attentionlearned from the original non-stationary series.
We start from the formula of Self-Attention [32]:
[Equation]
where Q , K , V ∈ R S ×
d k are length- S queries, keys and values of d k -dimension respectively, and
Softmax( · ) is conducted row by row.
To simplify the analysis, we assume the embedding and feedforward layers f to hold the linear properties 1 and f is conducted separately on each time point,that is, each query token in Q =
[ q 1 , q 2 , ..., q S ] T can be calculated as q i = f
( x i ) with respect tothe input series x =
[ x 1 , x 2 , · · · , x S ] T .
Since it is a convention to conduct normalization on eachtime series variable to avoid certain variable that dominates the scale, we can further assume eachvariable of series x shares the same variance, and thus original σ x ∈ R C × 1 is reduced to a scalar.
After Normalization module, the model receives the stationarized input x ′ =
( x − 1 µ T x ) /σ x , where1 ∈ R S × 1 is an all-ones vector.
Based on the linear property assumption, it can be proved that
the
Attention layer will receive Q ′ =
[ f ( x ′ 1 ) , ..., f ( x ′ S )]
T = ( Q − 1 µ T Q ) /σ x , where µ Q ∈ R d k × 1 is themean of Q along the temporal dimension (See supplementary materials for a detailed proof).
And sois the corresponding transformed K ′ , V ′ .
Without Series Stationarization, the input of Softmax( · ) in
Self-Attention should be QK T / √ d k , while now the attention is calculated based on Q ′ , K ′ :
[Equation]
We find that Q µ K ∈ R S × 1 and µ T Q µ K ∈ R , and they are repeatedly operated on each column andelement of σ 2 x Q ′ K ′ T ∈ R S × S respectively.
Since Softmax( · ) is invariant to the same translation onthe row dimension of input, we have the following equation:
[Equation]
The Equation 5 deduces a direct expression of the attention Softmax (cid:0) QK T / √ d k (cid:1) learned from rawseries x .
Except for the current Q ′ , K ′ from stationarized series x ′ , this expression also requires thenon-stationary information σ x , µ Q , K that are eliminated by Series Stationarization.
De-stationary Attention To recover the original attention on non-stationary series, we attemptto bring the vanished non-stationary information back to its calculation.
Based on the derivation in Equation 5, the key is to approximate the positive scaling scalar τ = σ 2 x ∈ R + and shifting vector∆ =
K µ Q ∈ R S × 1 , which are defined as de-stationary factors .
Considering the deep model scenariounder the real case, de-stationary factors should be disentangled from the statistics of unstationarizedx , Q and K .
As we can only discover limited non-stationary information from current Q ′ , K ′ , theunique and reasonable source to restitute non-stationarity is the original x without being normalized.
Thus, as a direct deep learning implementation of Equation 5, we apply a multi-layer perceptronas the projector to learn de-stationary factors τ, ∆ from the statistics µ x , σ x of unstationarized xindividually.
And the De-stationary Attention is calculated as follows:
[Equation]
where the de-stationary factors τ and ∆ are shared by De-stationary Attention of all layers (Figure 2).
De-stationary Attention mechanism learns the temporal dependencies from both stationarized series
Q ′ , K ′ and non-stationary series x , µ x , σ x , and multiplies by the stationarized values V ′ .
Therefore, it can benefit from the predictability of stationarized series and maintain the inherent temporal dependencies of raw series simultaneously.
Overall architecture Following the prior use of Transformers [36, 34] in time series forecasting, we adopt the standard Encoder-Decoder structure (Figure 2), where the encoder is to extractinformation from past observations, and the decoder is to aggregate past information and refinethe prediction from simple initialization.
The canonical Non-stationary Transformer is wrapped by
Series Stationarization to both the input and output of vanilla Transformer [32], and replacing the
Self-Attention by our proposed De-stationary Attention, which can boost the non-stationary seriespredictive capability of the base model.
For the Transformer variants [17, 36, 34], we transformthe terms inside Softmax( · ) with the de-stationary factors τ , ∆ to re-integrate the non-stationaryinformation (See supplementary materials for the implementation details).
Experiments
We conduct extensive experiments to evaluate the performance of Non-stationary Transformers onsix real-world time series forecasting benchmarks and further validate the generality of the proposedframework on various mainstream Transformer variants.
See supplementary materials for the code.
Datasets Here are the descriptions of the datasets: (1) Electricity [1] records the hourly electricityconsumption of 321 clients from 2012 to 2014.
(2) ETT [36] contains the time series of oil destationary factors and power load collected by electricity transformers from July 2016 to July 2018.
ETTm1 /ETTm2 are recorded every 15 minutes, and ETTh1/ETTh2 are recorded every hour.
Exchange [18] collects the panel data of daily exchange rates from 8 countries from 1990 to 2016.(4) ILI [2] collects the ratio of influenza-like illness patients versus the total patients in one week,which is reported weekly by Centers for Disease Control and Prevention of the United States from2002 and 2021.
(5) Traffic [3] contains hourly road occupancy rates measured by 862 sensors on
San Francisco Bay area freeways from January 2015 to December 2016.
(6) Weather [4] includesmeteorological time series with 21 weather indicators collected every 10 minutes from the Weather
Station of the Max Planck Biogeochemistry Institute in 2020.
Table 1 summarizes overall statisticsof the datasets. We also record the Augmented Dick-Fuller test statistic [13] of each dataset and listthem in ascending order of stationarity.
We follow the standard protocol that divides each dataset intothe training, validation, and testing subsets according to the chronological order.
The split ratio is3:1:1 for the ETT dataset and 7:2:2 for others.
[Table]
Baselines We evaluate the vanilla Transformer [32] equipped by the Non-stationary Tranformers framework in both multivariate and univariate settings to demonstrate its effectiveness.
Formultivariate forecasting, we include six state-of-the-art deep forecasting models: Autoformer [34],
Pyraformer [21], Informer [36], LogTrans [20], Reformer [17] and LSTNet [18].
For univariateforecasting, we include seven competitive baselines: N-HiTS [9], N-BEATS [25], Autoformer [34],
Pyraformer [21], Informer [36], Reformer [17] and ARIMA [7].
In addition, we adopt the proposed framework on both the canonical and efficient variants of Transformers: Transformer [32],
Informer [36], Reformer [17] and Autoformer [34] to validate the generality of our framework.
Implementation details All the experiments are implemented in PyTorch [28] and conducted forthree runs on a single NVIDIA TITAN V 12GB GPU.
Each model is trained by ADAM [16] using
L2 loss with the initial learning rate of 10 − 4 and batch size of 32.
Each Transformer-based modelcontains two encoder layers and one decoder layer.
Considering the efficiency of hyperparameterssearch, we use two-layer perceptron projector with the hidden dimension varying in { 64 , 128 , 256 }in De-stationary Attention.
We report the test MSE/MAE under different prediction lengths as theperformance metric.
A lower MSE/MAE indicates better performance of time series forecasting.
4.1 Main Results
Forecasting results As for multivariate forecasting results, the vanilla Transformer equipped withour framework consistently achieves state-of-the-art performance in all benchmarks and predictionlengths (Table 2).
Notably, Non-stationary Transformer outperforms other deep models impressivelyon datasets characterized by high non-stationarity.
With the prediction length of 336, we achieve17% MSE reduction ( 0 . 509 → 0 . 421 ) on Exchange and 25% ( 2 . 669 → 2 . 010 ) on ILI compared toprevious state-of-the-art results, which indicates that the potential of deep model is still constrainedon non-stationary data.
We also list the univariate results of two typical datasets with differentstationarity in Table 3.
Non-stationary Transformer still realizes remarkable forecasting performance.
[Table] [Table]
[Table] [Table]
ETT
[Table]
[Table]
Framework generality We apply our framework to four mainstream Transformers and report theperformance promotion of each model (Table 4).
Our method consistently improves the forecastingability of different models.
Overall, it achieves averaged 49.43% promotion on Transformer, 47.34%on Informer, 46.89% on Reformer and 10.57% on Autoformer, making each of them surpassprevious state-of-the-art.
Compared to native blocks of the models, there is hardly any parameterand computation increase by applying our framework (See supplementary materials for details),and thereby their computational complexities can be preserved.
It validates that Non-stationary
Transformers is an effective and lightweight framework that can be widely applied to Transformerbased models and enhances their non-stationary predictability to achieve state-of-the-art performance.
4.2 Model Analysis
Quality evaluation To explore the role of each module in our proposed framework, we comparethe prediction results on ETTm2 obtained by three models: vanilla Transformer, Transformer withsolely Series Stationarization, and Non-stationary Transformer.
In Figure 3, we find out that the twomodules strengthen the non-stationary forecasting ability of Transformer from different perspectives.
Series Stationarization focuses on the alignment of statistical properties among each series inputthat benefits Transformer a lot to generalize on out-of-distribution data.
However, as is shown in
Figure 3(b), the over-stationarized circumstance for training makes the deep model more likely tooutput stationary but uneventful series and neglect the nature of non-stationary real-world data.
Withthe aid of De-stationary Attention, the model gives concern back to the constitutional non-stationarityof real-world time series.
It is beneficial for an accurate prediction of the detailed series variation,which is vital in real-world time series forecasting.
[Figure]
Quantitative performance
In addition to the above case study, we also provide quantitativeforecasting performance comparison with stationarization methods: a deep method RevIN [15] and
Series Stationarization (Section 3.1).
As is shown in Table 5, the forecasting results assisted by
RevIN and Series Stationarization are basically the same, which indicates that the parameter-free version of normalization in our framework performs sufficiently to stationarize time series.
Besides,the proposed De-stationary Attention in Non-stationary Transformers further boosts the performanceand achieves the best in all six benchmarks.
The MSE reduction brought by De-stationary Attentionbecomes significant, especially when the dataset is highly non-stationary (Exchange: 0 .
569 → 0 . 461 ,
ETTm2: 0 .
461 → 0 . 306 ).
The comparison reveals that solely stationarizing time series still limitsthe predictive capability of Transformers, and the complementary mechanisms in Non-stationary
Transformers can properly release the models’ potential for non-stationary series forecasting. [Table] Weather
[Table]
Over-stationarization problem To verify the over-stationarization problem from a statistical view
,we train Transformers with the aforementioned methods respectively, arrange all predicted time seriesin chronological order and compare the stationarity with the ground truth (Figure 4).
While modelssolely equipped with stationarization methods tend to output series with high stationarity, the resultsassisted by De-stationary Attention are close to the actual value (relative stationarity ∈ [97% , 103%] ).
Besides, as the stationarity of time series increases, the over-stationarization problem becomesmore significant.
The huge stationarity discrepancy can account for the inferior performance of Transformer with stationarization only.
And it also demonstrates that De-stationary Attention as aninternal renovation of Transformers can alleviate over-stationarization.
[Figure]
Conclusion
This paper addresses time series forecasting from the perspective of stationarity.
Unlike previous studies that solely reduce the non-stationarity of original series, we propose Non-stationary Transformersto increase the predictability of time series and fully exploit the predictive capability of the modelsimultaneously.
Our proposed framework consists of two complementary mechanisms, where Series Stationarization is an efficient implementation to stationarize time series and De-stationary Attentionrenovates the internal mechanism to recover the constitutional non-stationarity of real-world series.
The introduction of our framework boosts the forecasting performance of mainstream Transformersto the state-of-the-art on six real-world benchmarks.
The impressive generality and performance ofthe proposed framework can be interesting to future research for time series forecasting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to 363 answer these questions.
For each question, change the default [TODO] to [Yes] [No] or [N/A] . You arestrongly encouraged to include a justification to your answer either by referencing the appropriate section of 365 your paper or providing a brief inline description. For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note
that
the Checklist 370 section does not count towards the page limit.
In your paper, please delete this instructions block and only keepthe Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Section 1 of main text.
(b) Did you describe the limitations of your work?
[Yes] See Section 7 of supplementary materials.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6 of
supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
[Table]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.2 of
[Table]
main text and Section 1 of supplementary materials.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3.2 of main text
[Table]
and Section 1 of supplementary materials.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes] We have provided the code insupplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5 of supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes] See Section 3 of supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Section 4 of main text.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4 of main text.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] SeeSection 4 of main text.
(d) Did you discuss whether and how consent was obtained from people whose data you’re us-
ing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]