Original: Conformal Prediction with Temporal Quantile Adjustments
Revised: Conformal Prediction with Temporal Quantile Adjustments

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We develop Temporal Quantile Adjustment (TQA), a general method to constructefficient and valid prediction intervals (PIs) for regression on cross-sectional timeseries data.
Revised: We develop Temporal Quantile Adjustment (TQA), a general method to constructefficient and valid prediction intervals (PIs) for regression on cross-sectional timeseries data.

Original: Such data is common in many domains, including econometrics andhealthcare.
Revised: Such data is common in many domains, including econometrics andhealthcare.

Original: A canonical example in healthcare is predicting patient outcomesusing physiological time-series data, where a population of patients composesa cross-section.
Revised: A canonical example in healthcare is predicting patient outcomesusing physiological time-series data, where a population of patients composesa cross-section.

Original: Reliable PI estimators in this setting must address two distinctnotions of coverage: cross-sectional coverage across a cross-sectional slice, andlongitudinal coverage along the temporal dimension for each time series.
Revised: Reliable PI estimators in this setting must address two distinctnotions of coverage: cross-sectional coverage across a cross-sectional slice, andlongitudinal coverage along the temporal dimension for each time series.

Original: Recentworks have explored adapting Conformal Prediction (CP) to obtain PIs in the timeseries context.
Revised: Recentworks have explored adapting Conformal Prediction (CP) to obtain PIs in the timeseries context.

Original: However, none handles both notions of coverage simultaneously.
Revised: However, none handles both notions of coverage simultaneously.

Original: CP methods typically query a pre-specified quantile from the distribution of nonconformity scores on a calibration set.
Revised: CP methods typically query a pre-specified quantile from the distribution of nonconformity scores on a calibration set.

Original: TQA adjusts the quantile to query in CPat each time t , accounting for both cross-sectional and longitudinal coverage in atheoretically-grounded manner.
Revised: TQA adjusts the quantile to query in CPat each time t , accounting for both cross-sectional and longitudinal coverage in atheoretically-grounded manner.

Original: The post-hoc nature of TQA facilitates its use asa general wrapper around any time series regression model.
Revised: The post-hoc nature of TQA facilitates its use asa general wrapper around any time series regression model.

Original: We validate TQA’sperformance through extensive experimentation: TQA generally obtains efficient
Revised: We validate TQA’sperformance through extensive experimentation: TQA generally obtains efficient

Original: PIs and improves longitudinal coverage while preserving cross-sectional coverage.
Revised: PIs and improves longitudinal coverage while preserving cross-sectional coverage.

Original:  Introduction
Revised:  Introduction

Original: The impressive predictive performance of modern “black-box” machine learning methods has startedto make them critical ingredients in various high-stakes decision-making pipelines.
Revised: The impressive predictive performance of modern “black-box” machine learning methods has startedto make them critical ingredients in various high-stakes decision-making pipelines.

Original: It is thus increasingly important to quantify the predictive uncertainty of such models reliably and efficiently, whichremains a fundamental challenge.
Revised: It is thus increasingly important to quantify the predictive uncertainty of such models reliably and efficiently, whichremains a fundamental challenge.

Original: Conformal Prediction (CP), pioneered by Vovk et al.
Revised: Conformal Prediction (CP), pioneered by Vovk et al.

Original: [52] is apowerful framework for quantifying uncertainty under mild assumptions.
Revised: [52] is apowerful framework for quantifying uncertainty under mild assumptions.

Original: The model-agnostic anddistribution-free nature of CP makes it particularly suitable for large neural network models andhas started to attract the attention of the deep learning community [2, 3, 6, 9, 12, 13, 33, 58].
Revised: The model-agnostic anddistribution-free nature of CP makes it particularly suitable for large neural network models andhas started to attract the attention of the deep learning community [2, 3, 6, 9, 12, 13, 33, 58].

Original: Theprimary assumption in most current CP methods is that of data exchangeability.
Revised: Theprimary assumption in most current CP methods is that of data exchangeability.

Original: For instance, onlyassuming exchangeability of the calibration and test data, one can construct 1 − α valid predictionintervals by simply querying the corresponding quantile of nonconformity scores on the calibrationset.
Revised: For instance, onlyassuming exchangeability of the calibration and test data, one can construct 1 − α valid predictionintervals by simply querying the corresponding quantile of nonconformity scores on the calibrationset.

Original: Recent works have started exploring the adaptation of CP in settings that go beyond the usualexchangeability assumption [4, 17, 21, 37, 41, 42, 43, 49, 56], and to more complex data such as timeseries [17, 47, 55, 57].
Revised: Recent works have started exploring the adaptation of CP in settings that go beyond the usualexchangeability assumption [4, 17, 21, 37, 41, 42, 43, 49, 56], and to more complex data such as timeseries [17, 47, 55, 57].

Original: We study the adaptation of CP to the cross-sectional time series regression setting.
Revised: We study the adaptation of CP to the cross-sectional time series regression setting.

Original: More formally,suppose our data comprises of N time series, denoted { S i } Ni =1 , with each S i sampled from anarbitrary distribution P S .
Revised: More formally,suppose our data comprises of N time series, denoted { S i } Ni =1 , with each S i sampled from anarbitrary distribution P S .

Original: Further, each time series S i is a sequence of temporally-dependent randomvariables [ Z i, 1 . . .
Revised: Further, each time series S i is a sequence of temporally-dependent randomvariables [ Z i, 1 . . .

Original: , Z i,t , . . .
Revised: , Z i,t , . . .

Original: , Z i,T ] , with Z i,t =
Revised: , Z i,T ] , with Z i,t =

Original: ( X i,t  Y i,t ) consisting of covariates X · , · ∈ R d andthe response Y · , · ∈ R .
Revised: ( X i,t , Y i,t ) consisting of covariates X · , · ∈ R d andthe response Y · , · ∈ R .

Original: Given data  { Z N +1 ,j } tj =1 until time t for a new time series S N +1 , the time series regression problem entails predicting the response Y N +1 ,t +1 at (unknown) time t + 1 .
Revised: Given data until time t ( { Z N +1 ,j } tj =1 ) and X N +1 ,t +1 for a new time series S N +1 , the time series regression problem entails predicting the response Y N +1 ,t +1 at (unknown)time t + 1 .

Original: We areinterested in quantifying the uncertainty of each prediction by constructing valid prediction intervals(PI).
Revised: We are interested in quantifying the uncertainty of each prediction by constructing validprediction intervals (PI).

Original: That is, given a confidence level 1 − α , we are interested in constructing a prediction interval,ˆ C α,N +1 ,t +1 , that will cover Y N +1 ,t +1 with probability of at least 1 − α .
Revised: That is, given a confidence level 1 − α , we are interested in constructing aprediction interval, ˆ C α,N +1 ,t +1 , that will cover Y N +1 ,t +1 with probability of at least 1 − α .

Original: A crucial requirement in cross-sectional time series regression is to distinguish two notions of validity,longitudinal and cross-sectional , to ensure reliable performance.
Revised: A crucial requirement in cross-sectional time series regression is to distinguish two notions of validity,longitudinal and cross-sectional , to ensure reliable performance.

Original: Longitudinal validity is concernedwith validity along the temporal axis for each time series.
Revised: Longitudinal validity is concernedwith validity along the temporal axis for each time series.

Original: On the other hand, cross-sectional validityis concerned with validity across the populational cross-section of the time series data.
Revised: On the other hand, cross-sectional validityis concerned with validity across the populational cross-section of the time series data.

Original: Figure 1illustrates both notions of validity and a standard real-world occurrence of such a problem setting.
Revised: Figure 1illustrates both notions of validity and a standard real-world occurrence of such a problem setting.

Original: Recently, several research groups have explored adapting CP to the time series setting.
Revised: Recently, several research groups have explored adapting CP to the time series setting.

Original: Some of thiswork [17, 57, 55] focuses only on longitudinal validity, which is extremely difficult without strongdistributional assumptions [4].
Revised: Some of thiswork [17, 57, 55] focuses only on longitudinal validity, which is extremely difficult without strongdistributional assumptions [4].

Original: Furthermore, such methods cannot leverage rich information inherentin the cross-section.
Revised: Furthermore, such methods cannot leverage rich information inherentin the cross-section.

Original: The only work which addresses cross-sectional validity is to due to Stankeviˇci ¯ ut ˙ eet al.
Revised: The only work which addresses cross-sectional validity is to due to Stankeviˇci ¯ ut ˙ eet al.

Original: [47], but it ignores the temporal dependence.
Revised: [47], but it ignores the temporal dependence.

Original: Accounting for both notions of coverage is criticalto obtain reliable performance.
Revised: Accounting for both notions of coverage is criticalto obtain reliable performance.

Original: To remedy the above situation, we propose Temporal Quantile Adjustment (TQA) for CP in thecross-sectional time series regression setting.
Revised: To remedy the above situation, we propose Temporal Quantile Adjustment (TQA) for CP in thecross-sectional time series regression setting.

Original: TQA is the first method that can account for both crosssectional and longitudinal validity simultaneously.
Revised: TQA is the first method that can account for both crosssectional and longitudinal validity simultaneously.

Original: Although TQA can be used as a wrapper aroundany time series regression model, we focus on neural networks, as our main inspiration comes fromcomplicated time series regression problems in healthcare 1 .
Revised: Although TQA can be used as a wrapper aroundany time series regression model, we focus on neural networks, as our main inspiration comes fromcomplicated time series regression problems in healthcare 1 .

Original: Neural networks are particularly suitedfor such tasks, which can involve modeling the evolution of heterogeneous entities such as diagnosticand drug codes, patient and physician embeddings and regressing over a target of interest.
Revised: Neural networks are particularly suitedfor such tasks, which can involve modeling the evolution of heterogeneous entities such as diagnosticand drug codes, patient and physician embeddings and regressing over a target of interest.

Original: Takinginspiration from [17], TQA adjusts the quantile to query at each time step in a theoretically-groundedmanner.
Revised: Takinginspiration from [17], TQA adjusts the quantile to query at each time step in a theoretically-groundedmanner.

Original: Based on the nature of quantile adjustment, we also propose two variants of TQA, whichfurther shed light on the generality of our method.
Revised: Based on the nature of quantile adjustment, we also propose two variants of TQA, whichfurther shed light on the generality of our method.

Original: The ability of TQA to handle both cross-sectionaland longitudinal validity is borne out in extensive experimentation, where it significantly outperformscompeting methods.
Revised: The ability of TQA to handle both cross-sectionaland longitudinal validity is borne out in extensive experimentation, where it significantly outperformscompeting methods.

Original:  Related Work
Revised:  Related Work

Original: Our work falls squarely within the Conformal Prediction (CP) framework.
Revised: Our work falls squarely within the Conformal Prediction (CP) framework.

Original: The original formulationof CP was in a purely transductive setting [45, 46, 52], and was computationally inefficient.
Revised: The original formulationof CP was in a purely transductive setting [45, 46, 52], and was computationally inefficient.

Original: Moreefficient variants dubbed as Inductive Conformal Prediction (ICP)
Revised: Moreefficient variants dubbed as Inductive Conformal Prediction (ICP)

Original: [38, 39, 51] were proposed soonafter and were more broadly popularized by followup works in statistics [29, 31, 32].
Revised: [38, 39, 51] were proposed soonafter and were more broadly popularized by followup works in statistics [29, 31, 32].

Original: A similar idea,often referred to as Split Conformal [30], is now more or less used interchangeably with ICP, and is fundamental to our paper.
Revised: A similar idea,often referred to as Split Conformal [30], is now more or less used interchangeably with ICP, and is fundamental to our paper.

Original: The model-agnostic and distribution-free nature of split conformal makesit suitable for large black-box models, and thus it has seen adoption in deep learning-based pipelines(e.g.
Revised: The model-agnostic and distribution-free nature of split conformal makesit suitable for large black-box models, and thus it has seen adoption in deep learning-based pipelines(e.g.

Original: One of the mainstays of the CP framework is the assumption of exchangeability between calibrationand test data.
Revised: One of the mainstays of the CP framework is the assumption of exchangeability between calibrationand test data.

Original: Extensions of CP “beyond exchangeability” have attracted relatively little attentionuntil recently.
Revised: Extensions of CP “beyond exchangeability” have attracted relatively little attentionuntil recently.

Original: A key publication in the area is [49] which used the notion of weighted exchangeabilityto handle covariate shift.
Revised: A key publication in the area is [49] which used the notion of weighted exchangeabilityto handle covariate shift.

Original: Several notable works that address various aspects of the covariate shiftproblem include [21, 41, 56, 43].
Revised: Several notable works that address various aspects of the covariate shiftproblem include [21, 41, 56, 43].

Original: More recent work [4, 17] handles gradual distribution drifts.
Revised: More recent work [4, 17] handles gradual distribution drifts.

Original: [4]additionally proposes an extension of CP when the data-points cannot be treated in a symmetricmanner.
Revised: [4]additionally proposes an extension of CP when the data-points cannot be treated in a symmetricmanner.

Original: Extensions of the split conformal method to a broad class of dependent processes such as stationary β -mixing processes was proposed by [37].
Revised: Extensions of the split conformal method to a broad class of dependent processes such as stationary β -mixing processes was proposed by [37].

Original: These works provide some general methodologicalinsights to model temporal dependence in our case, but are otherwise not directly related.
Revised: These works provide some general methodologicalinsights to model temporal dependence in our case, but are otherwise not directly related.

Original: In particular,the online adaptive method of [17], served as a major source of inspiration for the development of avariant of TQA (dubbed TQA-E).
Revised: In particular,the online adaptive method of [17], served as a major source of inspiration for the development of avariant of TQA (dubbed TQA-E).

Original: The most related works have a focus on generating valid intervalsin time series regression [17, 57, 4, 55].
Revised: The most related works have a focus on generating valid intervalsin time series regression [17, 57, 4, 55].

Original: These works end up ignoring cross-sectional aspects, whichis understandable given the tasks they study.
Revised: These works end up ignoring cross-sectional aspects, whichis understandable given the tasks they study.

Original: On the other extreme is [47], which focuses only oncross-sectional coverage, ignoring the temporal dimension in constructing PIs.
Revised: On the other extreme is [47], which focuses only oncross-sectional coverage, ignoring the temporal dimension in constructing PIs.

Original: Various other techniques, outside the ambit of CP, have also been extended to quantify uncertainty intime series forecasting as well.
Revised: Various other techniques, outside the ambit of CP, have also been extended to quantify uncertainty intime series forecasting as well.

Original: For instance, approximate Bayesian methods [8, 53, 36, 34, 25, 15, 28]are quite popular for uncertainty quantification, and have been extended to RNNs [14, 7].
Revised: For instance, approximate Bayesian methods [8, 53, 36, 34, 25, 15, 28]are quite popular for uncertainty quantification, and have been extended to RNNs [14, 7].

Original: Finally,one may also use the idea of directly predicting the quantiles (as opposed to the point estimate) inregression tasks [48, 27], and applying it to time series forecasting [54, 16].
Revised: Finally,one may also use the idea of directly predicting the quantiles (as opposed to the point estimate) inregression tasks [48, 27], and applying it to time series forecasting [54, 16].

Original: However, such methodsusually require changing the base model and typically do not come with coverage guarantees.
Revised: However, such methodsusually require changing the base model and typically do not come with coverage guarantees.

Original:  Preliminaries
Revised:  Preliminaries

Original: This section builds foundation for our exposition of TQA.
Revised: This section builds foundation for our exposition of TQA.

Original: We begin by expounding further onlongitudinal and cross-sectional validity, followed by presenting the exchangeability assumption.
Revised: We begin by expounding further onlongitudinal and cross-sectional validity, followed by presenting the exchangeability assumption.

Original: Finally, we discuss the use of split conformal prediction to construct (cross-sectionally) valid PIs.
Revised: Finally, we discuss the use of split conformal prediction to construct (cross-sectionally) valid PIs.

Original: 3.1 Cross-sectional Validity vs Longitudinal Validity
Revised: 3.1 Cross-sectional Validity vs Longitudinal Validity

Original: Cross-sectional validity is the more common type of validity encountered in CP, being the only typeof validity in non-time-series settings.
Revised: Cross-sectional validity is the more common type of validity encountered in CP, being the only typeof validity in non-time-series settings.

Original: More formally:
Revised: More formally:

Original: Definition 1.
Revised: Definition 1.

Original: Prediction interval ˆ C · , · is 1 − α cross-sectionally valid if, for any t ,
Revised: Prediction interval ˆ C · , · is 1 − α cross-sectionally valid if, for any t ,

Original: [Equation]
Revised: [Equation]

Original: P S N +1 means the probability is taken over the randomness of S N +1 .
Revised: P S N +1 means the probability is taken over the randomness of S N +1 .

Original: If ˆ C N +1 ,t is random (e.g.depends on { S i } Ni =1 ) then the probability is taken over the randomness of ˆ C N +1 ,t as well.
Revised: If ˆ C N +1 ,t is random (e.g.depends on { S i } Ni =1 ) then the probability is taken over the randomness of ˆ C N +1 ,t as well.

Original: Note thatif we consider the case where every time series only consists of one step ( T = 1 ), then we recoverthe usual definition of marginal validity.
Revised: Note thatif we consider the case where every time series only consists of one step ( T = 1 ), then we recoverthe usual definition of marginal validity.

Original: Cross-sectional validity translates to high-probability incoverage for a randomly drawn time series .
Revised: Cross-sectional validity translates to high-probability incoverage for a randomly drawn time series .

Original: As we will see later, cross-sectional validity is easier toachieve, since we can assume inter -time-series exchangeability.
Revised: As we will see later, cross-sectional validity is easier toachieve, since we can assume inter -time-series exchangeability.

Original: Longitudinal validity , on the other hand, is concerned with coverage along the temporal axis for aparticular TS.
Revised: Longitudinal validity , on the other hand, is concerned with coverage along the temporal axis for aparticular TS.

Original: We use the following definition:
Revised: We use the following definition:

Original: Definition 2.
Revised: Definition 2.

Original: Prediction interval ˆ C · , · is 1 − α longitudinally valid if for almost every time-series
Revised: Prediction interval ˆ C · , · is 1 − α longitudinally valid if for almost every time-series

Original: S N +1 ∼ P S there exists a T 0 such that:
Revised: S N +1 ∼ P S there exists a T 0 such that:

Original: [Equation]
Revised: [Equation]

Original: Here, the qualifier “almost every” means that the set of time series’ for which such coverage may failis of measure zero (under P S ).
Revised: Here, the qualifier “almost every” means that the set of time series’ for which such coverage may failis of measure zero (under P S ).

Original: The threshold T 0 allows some “time” for ˆ C to potentially adapt to thetemporal information in a particular TS.
Revised: The threshold T 0 allows some “time” for ˆ C to potentially adapt to thetemporal information in a particular TS.

Original: It should be clear that longitudinal validity is harder to attain,because we can no longer marginalize the probability over randomly drawn time series.
Revised: It should be clear that longitudinal validity is harder to attain,because we can no longer marginalize the probability over randomly drawn time series.

Original: 3.2 Conformal Prediction for Cross-sectionally Valid PI
Revised: 3.2 Conformal Prediction for Cross-sectionally Valid PI

Original: In this section we explain how to use conformal prediction to construct cross-sectionally valid PIs.
Revised: In this section we explain how to use conformal prediction to construct cross-sectionally valid PIs.

Original: Wefirst introduce exchangeability assumption, which is the central assumption in conformal prediction,and slightly weaker than the standard i.i.d assumption.
Revised: Wefirst introduce exchangeability assumption, which is the central assumption in conformal prediction,and slightly weaker than the standard i.i.d assumption.

Original: More formally:
Revised: More formally:

Original: Definition 3.
Revised: Definition 3.

Original: ( Exchangeability [52])
Revised: ( Exchangeability [52])

Original: A sequence of random variables, Z 1 , Z 2 , . . .
Revised: A sequence of random variables, Z 1 , Z 2 , . . .

Original: , Z n ∈ Z areexchangeable if for any permutation π : { 1 , 2 , . . . , n } → { 1 , 2 , . . . , n } , and every measurable set
Revised: , Z n ∈ Z areexchangeable if for any permutation π : { 1 , 2 , . . . , n } → { 1 , 2 , . . . , n } , and every measurable set

Original: E ⊆ Z n , we have
Revised: E ⊆ Z n , we have

Original: [Equation]
Revised: [Equation]

Original: Definition 3 can naturally be extended to a sequence of randomly drawn time series:
Revised: Definition 3 can naturally be extended to a sequence of randomly drawn time series:

Original: Definition 4.
Revised: Definition 4.

Original: ( Exchangeable Time Series )
Revised: ( Exchangeable Time Series )

Original: Given time series S 1 , S 2 , . . .
Revised: Given time series S 1 , S 2 , . . .

Original: , S n where S
Revised: , S n where S

Original: i =[
Revised: i =[

Original: Z i, 1 , . . .
Revised: Z i, 1 , . . .

Original: , Z i,T , . .
Revised: , Z i,T , . .

Original: . ] , denote Z i, { t j } mj =1 as the concatenated random variable of ( Z i,t 1 , . . .
Revised: . ] , denote Z i, { t j } mj =1 as the concatenated random variable of ( Z i,t 1 , . . .

Original: , Z i,t m ) .
Revised: , Z i,t m ) .

Original: Time series S 1 , S 2 , . .
Revised: Time series S 1 , S 2 , . .

Original: . , S n are exchangeable if, for any finitely many t 1 < · · · < t m , the randomvariables Z 1 , { t j } mj =1 , . .
Revised: . , S n are exchangeable if, for any finitely many t 1 < · · · < t m , the randomvariables Z 1 , { t j } mj =1 , . .

Original: , Z n, { t j } mj =1 are exchangeable.
Revised: , Z n, { t j } mj =1 are exchangeable.

Original: As a concrete example, suppose we randomly pick 100 patients from a hospital’s EHR database forpredicting readmission risk.
Revised: As a concrete example, suppose we randomly pick 100 patients from a hospital’s EHR database forpredicting readmission risk.

Original: It is fairly reasonable to assume that these time series are exchangeable,despite the obvious strong temporal dependence within each time series.
Revised: It is fairly reasonable to assume that these time series are exchangeable,despite the obvious strong temporal dependence within each time series.

Original: Throughout this paper, wewill assume S 1 , . . .
Revised: Throughout this paper, wewill assume S 1 , . . .

Original: , S N +1 are exchangeable time series.
Revised: , S N +1 are exchangeable time series.

Original: We now explain how to construct cross-sectionally valid PIs.
Revised: We now explain how to construct cross-sectionally valid PIs.

Original: To construct a PI for Y i,t , we firstsplit ourtraining data { S i } Ni =1 into a proper training set and a calibration set [40].
Revised: To construct a PI for Y i,t , we firstsplit our data { S i } Ni =1 into a proper training set and a calibration set [40].

Original: The trainingset is used to train models for the nonconformity score function, and the calibration set is used tocollect such nonconformity scores (denoted as V ( · ) ).
Revised: The training set isused to train models for the nonconformity score function, and the calibration set is used to collectsuch nonconformity scores (denoted as V ( · ) ).

Original: For example, one may train a mean estimatorˆ µ (e.g. an RNN) on the training set, and use the absolute residual v i,t ← | y i,t − ˆ y i,t | , whereˆ y i,t
Revised: For example, one may train a mean estimator ˆ µ(e.g. an RNN) on the training set, and use the absolute residual v i,t ← | y i,t − ˆ y i,t | , where ˆ y i,t

Original: = ˆ µ ( X i,t ; S i, : t − 1 ) , as the nonconformity score.
Revised: =ˆ µ ( X i,t ; S i, : t − 1 ) , as the nonconformity score.

Original: Here S i, : t − 1 denotes [ Z i, 1 , . . .
Revised: Here S i, : t − 1 denotes [ Z i, 1 , . . .

Original: , Z i,t − 1 ] .
Revised: , Z i,t − 1 ] .

Original: The136 idea  behind the split conformal method is that the scoring function (e.g. ˆ µ ) is only fit on theproper training set, implying that nonconformity scores on the calibration set and v N +1 ,t are alsoexchangeable .
Revised: The idea 136 behind the split conformal method is that the scoring function (e.g. ˆ µ ) is only fit on the proper trainingset, implying that nonconformity scores on the calibration set and v N +1 ,t are also exchangeable .

Original: Here on, for the sake of simplicity of notation, we will use { S i } Ni =1 to denote thecalibration set only , assuming all necessary models have already been trained.
Revised: Here on, for the sake of simplicity of notation, we will use { S i } Ni =1 to denote the calibration set only ,assuming all necessary models have already been trained.

Original: Given a nonconformity score V ( · ) (possibly using some trained model ˆ µ ) and a set of exchangeabletime series { S i } N +1 i =1 , the split conformal method can be used to generate the following 1 − α 142 cross-sectionally valid prediction interval:
Revised: Given a nonconformity score V ( · ) (possibly using some trained model ˆ µ ) and a set of exchangeabletime series { S i } N +1 i =1 , the split conformal method can be used to generate the following 1 − α 142 cross-sectionally valid prediction interval:

Original: [Equation]
Revised: [Equation]

Original: Here, Q ( β ; A ) means the β -quantile for the set A .
Revised: Here, Q ( β ; A ) means the β -quantile for the set A .

Original: As a concrete example, if we let v i,t = | y i,t − ˆ y i,t | , 144 and employ the standard trick that replaces v
Revised: As a concrete example, if we let v i,t = | y i,t − ˆ y i,t | , 144 and employ the standard trick that replaces v

Original: N +1 ,t +1 with ∞ to avoid plugging in (uncountably) 145 many values for y [4], the prediction interval for Y N +1 ,t +1 becomes:ˆ C splitN
Revised: N +1 ,t +1 with ∞ to avoid plugging in (uncountably) 145 many values for y [4], the prediction interval for Y N +1 ,t +1 becomes:ˆ C splitN

Original: +1 ,t +1 :=
Revised: +1 ,t +1 :=

Original: [ˆ y − ˆ v, ˆ y
Revised: [ˆ y − ˆ v, ˆ y

Original: + ˆ v ] where ˆ v := Q (cid:16) 1 − α ; {| y i,t +1 − ˆ y i,t +

Revised: + ˆ v ] where ˆ v := Q (cid:16) 1 − α ; {| y i,t +1 − ˆ y i,t +

Original: |} Ni =1 ∪ {∞}} (cid:17) (5)
Revised: |} Ni =1 ∪ {∞}} (cid:17) (5)

Original: Assuming exchangeability, we can easily show the cross-sectional validity of ˆ C split :
Revised: Assuming exchangeability, we can easily show the cross-sectional validity of ˆ C split :

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: ˆ C split is 1 − α cross-sectionally valid (Def. 1).
Revised: ˆ C split is 1 − α cross-sectionally valid (Def. 1).

Original: The intuition behind the proof is that the exchangeability of the time series translates to the exchangeability of the nonconformity scores, which means the rank of v N +1 ,t +1 among { v i,t +1 } N +

Revised: The intuition behind the proof is that the exchangeability of the time series translates to the exchangeability of the nonconformity scores, which means the rank of v N +1 ,t +1 among { v i,t +1 } N +

Original: i =1 followsa uniform distribution.
Revised: i =1 followsa uniform distribution.

Original: The coverage guarantee in Theorem 3.1 then follows.
Revised: The coverage guarantee in Theorem 3.1 then follows.

Original: Note that with a finitecalibration set, the 1 − α -quantile could be ambiguously defined, and in practice one would use⌈ (1 − α )( N +1)
Revised: Note that with a finitecalibration set, the 1 − α -quantile could be ambiguously defined, and in practice one would use⌈ (1 − α )( N +1)

Original: ⌉ N + to get a slightly more conservative PI, or “flip a (biased) coin” to choose between⌈ (1 − α )
Revised: ⌉ N + to get a slightly more conservative PI, or “flip a (biased) coin” to choose between⌈ (1 − α )

Original: ( N +1)
Revised: ( N +1)

Original: ⌉ N + and ⌊ (1 − α )
Revised: ⌉ N + and ⌊ (1 − α )

Original: ( N +1) ⌋ N + for a precise 1 − α coverage (e.g. the “smoothed” ICP in [52] or thetie-breaking trick in [2]).
Revised: ( N +1) ⌋ N + for a precise 1 − α coverage (e.g. the “smoothed” ICP in [52] or thetie-breaking trick in [2]).

Original: In Section 4, we assume the precise coverage PI for the ease of discussion.
Revised: In Section 4, we assume the precise coverage PI for the ease of discussion.

Original: While cross-sectionally valid, ˆ C split ignores the temporal dependence in the nonconformity scorescompletely.
Revised: While cross-sectionally valid, ˆ C split ignores the temporal dependence in the nonconformity scorescompletely.

Original: We will explain in Section 4 how to adapt to the temporal dependence by “quantileadjustment”, thus improving longitudinal coverage as well.
Revised: We will explain in Section 4 how to adapt to the temporal dependence by “quantileadjustment”, thus improving longitudinal coverage as well.

Original:  Temporal Quantile Adjustment (TQA)
Revised:  Temporal Quantile Adjustment (TQA)

Original: In Section 3 we discussed a classical conformal prediction method and also highlighted its inherentlimitations in the time series setting.
Revised: In Section 3 we discussed a classical conformal prediction method and also highlighted its inherentlimitations in the time series setting.

Original: In this section we will formally introduce Temporal Quantile
Revised: In this section we will formally introduce Temporal Quantile

Original: Adjustment (TQA), which queries quantiles differently than in the aforementioned split conformalprocedures.
Revised: Adjustment (TQA), which queries quantiles differently than in the aforementioned split conformalprocedures.

Original: We first explicate the goals and motivations of TQA in Section 4.1.
Revised: We first explicate the goals and motivations of TQA in Section 4.1.

Original: Then, in Section 4.2and 4.3 we propose two principled adjustment methods along with theoretical analyses.
Revised: Then, in Section 4.2and 4.3 we propose two principled adjustment methods along with theoretical analyses.

Original: 4.1 Improving Longitudinal Coverage
Revised: 4.1 Improving Longitudinal Coverage

Original: Although it is tempting to directly pursue distribution-free finite-sample PI estimator that achieveslongitudinal coverage guarantee (Def. 2), it is likely too optimistic due to the fact that [ Z i, 1 , . . .
Revised: Although it is tempting to directly pursue distribution-free finite-sample PI estimator that achieveslongitudinal coverage guarantee (Def. 2), it is likely too optimistic due to the fact that [ Z i, 1 , . . .

Original: , Z i,T ]are not exchangeable and we cannot characterize them meaningfully without imposing (strong)distributional assumptions.
Revised: , Z i,T ]are not exchangeable and we cannot characterize them meaningfully without imposing (strong)distributional assumptions.

Original: For example, in [4], the bound for coverage gap—which captures theloss in coverage compared to what is achievable under exchangeability—essentially becomes 1 asthe data becomes time-dependent.
Revised: For example, in [4], the bound for coverage gap—which captures theloss in coverage compared to what is achievable under exchangeability—essentially becomes 1 asthe data becomes time-dependent.

Original: In fact, one could view Def. 2 as a “conditional validity” (where
Revised: In fact, one could view Def. 2 as a “conditional validity” (where

Original: S N +1 , : t as a whole is viewed as the input), and it is well-known that distribution-free finite-sampleconditional validity is impossible to achieve in a meaningful way [5, 32].
Revised: S N +1 , : t as a whole is viewed as the input), and it is well-known that distribution-free finite-sampleconditional validity is impossible to achieve in a meaningful way [5, 32].

Original: Nevertheless, longitudinal coverage may still be empirically improved if we can adapt to the temporaldependence.
Revised: Nevertheless, longitudinal coverage may still be empirically improved if we can adapt to the temporaldependence.

Original: From our EHR example, suppose ˆ µ has been giving high-error predictions for a patientfor 8 out of the past 10 days, we might suspect high error going forward for this patient as well.
Revised: From our EHR example, suppose ˆ µ has been giving high-error predictions for a patientfor 8 out of the past 10 days, we might suspect high error going forward for this patient as well.

Original: Ifthis is indeed the case, then naively applying ˆ C split can only attain low coverage for this patient, nomatter how long a history we observe.
Revised: Ifthis is indeed the case, then naively applying ˆ C split can only attain low coverage for this patient, nomatter how long a history we observe.

Original: To address this, we propose to query different quantiles basedon the partially observed time series.
Revised: To address this, we propose to query different quantiles basedon the partially observed time series.

Original: From now on, we use ˆ C a i,t ,i,t to denote a PI for Y i,t with a pre-specified target coverage of 1 − a i,t ,in order to emphasize dependence on the “quantile to query”.
Revised: From now on, we use ˆ C a i,t ,i,t to denote a PI for Y i,t with a pre-specified target coverage of 1 − a i,t ,in order to emphasize dependence on the “quantile to query”.

Original: We let a i,t
Revised: We let a i,t

Original: =
Revised: =

Original: α − ˆ δ
Revised: α − ˆ δ

Original: i,t , where ˆ δ i,t is thequantile adjustment .
Revised: i,t , where ˆ δ i,t is thequantile adjustment .

Original: Classical split conformal PIs (e.g. [47]) entail a special case: ∀ i, ∀ t, ˆ δ i,t ≡ 0 .
Revised: Classical split conformal PIs (e.g. [47]) entail a special case: ∀ i, ∀ t, ˆ δ i,t ≡ 0 .

Original: We refer to this method as Temporal Quantile Adjustment (TQA).
Revised: We refer to this method as Temporal Quantile Adjustment (TQA).

Original: Now, denote the random variable R i,t as the rank/quantile of V i,t among { V j,t } N +1 j =1 :
Revised: Now, denote the random variable R i,t as the rank/quantile of V i,t among { V j,t } N +1 j =1 :

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: For example, if v i,t is the smallest among { v j,t } N +1 j =1 , then r i,t = 0 .
Revised: For example, if v i,t is the smallest among { v j,t } N +1 j =1 , then r i,t = 0 .

Original: Intuitively, we would like to usea more conservative (smaller) a N +1 ,t when we believe S N +1 as a whole is less “conformal” and
Revised: Intuitively, we would like to usea more conservative (smaller) a N +1 ,t when we believe S N +1 as a whole is less “conformal” and

Original: R N +1 ,t is likely high.
Revised: R N +1 ,t is likely high.

Original: A crucial observation is that if there is no actual temporal dependence betweenthe nonconformity scores (i.e. we are just adjusting a t based on some “noise”), then we do not loseany coverage as long as the expected adjustment is zero.
Revised: A crucial observation is that if there is no actual temporal dependence betweenthe nonconformity scores (i.e. we are just adjusting a t based on some “noise”), then we do not loseany coverage as long as the expected adjustment is zero.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: If the nonconformity score’s rank ( R N +1 ,t ) is independent of the quantile adjustment(ˆ δ N +1 ,t ) , then
Revised: If the nonconformity score’s rank ( R N +1 ,t ) is independent of the quantile adjustment(ˆ δ N +1 ,t ) , then

Original: P S N +

Revised: P S N +

Original: { Y N +1 ,
Revised: { Y N +1 ,

Original: t ∈
Revised: t ∈

Original: ˆ C a N +1 ,t ,N +1 ,t } ≥ 1 − α + E
Revised: ˆ C a N +1 ,t ,N +1 ,t } ≥ 1 − α + E

Original: [ˆ δ N +1 ,t ] .
Revised: [ˆ δ N +1 ,t ] .

Original: All proofs are deferred to the Appendix.
Revised: All proofs are deferred to the Appendix.

Original: Remark : The assumption in Theorem 4.1 is not that S N +1 itself is not temporally dependent, nor isit the slightly weaker assumption, that of the temporal independence of either { R · ,t } Tt =1 or { V · ,t } Tt =1 .
Revised: Remark : The assumption in Theorem 4.1 is not that S N +1 itself is not temporally dependent, nor isit the slightly weaker assumption, that of the temporal independence of either { R · ,t } Tt =1 or { V · ,t } Tt =1 .

Original: The assumption in Theorem 4.1 only suggests that there is no temporal pattern in the prediction errorsthat ˆ δ can capture.
Revised: The assumption in Theorem 4.1 only suggests that there is no temporal pattern in the prediction errorsthat ˆ δ can capture.

Original: This could happen, for example, when the RNN captures the underlying datagenerating process fully, but only misses the random noise (aleatoric uncertainty).
Revised: This could happen, for example, when the RNN captures the underlying datagenerating process fully, but only misses the random noise (aleatoric uncertainty).

Original: This could alsohappen if our quantile adjustments ( ˆ δ ) are pure noise.
Revised: This could alsohappen if our quantile adjustments ( ˆ δ ) are pure noise.

Original: Unfortunately, although it can be tempting to conclude that E [ˆ δ N +1 ,t ] = 0 implies finite-sample crosssectional validity, this conclusion would be incorrect, because it ignores the dependence between1 { Y N +1 ,t ∈
Revised: Unfortunately, although it can be tempting to conclude that E [ˆ δ N +1 ,t ] = 0 implies finite-sample crosssectional validity, this conclusion would be incorrect, because it ignores the dependence between1 { Y N +1 ,t ∈

Original: ˆ C a N +1 ,t ,N +1 ,t } and a N +1 ,t .
Revised: ˆ C a N +1 ,t ,N +1 ,t } and a N +1 ,t .

Original: However, we will next discuss how to perform quantile 201 adjustment, and why it typically improves coverage.
Revised: However, we will next discuss how to perform quantile 201 adjustment, and why it typically improves coverage.

Original: [Figure]
Revised: [Figure]

Original: 4.2 Quantile Budgeting (TQA-B)
Revised: 4.2 Quantile Budgeting (TQA-B)

Original: Theorem 4.1 provides an interesting constraint that we should consider while designing ˆ δ .
Revised: Theorem 4.1 provides an interesting constraint that we should consider while designing ˆ δ .

Original: That is,we should let E [ˆ δ ] = 

Revised: That is,we should let E [ˆ δ ] =

Original: so as to keep the same coverage when we cannot predict the quantiles, buthopefully improving coverage when we can.
Revised: so as to keep the same coverage when we cannot predict the quantiles, buthopefully improving coverage when we can.

Original: This suggests a design of ˆ δ that we refer to as a typeof “budgeting”.
Revised: This suggests a design of ˆ δ that we refer to as a typeof “budgeting”.

Original: Although it is possible to directly predict a good ˆ δ i,t +1 , we adopt a more principledtwo-step approach:
Revised: Although it is possible to directly predict a good ˆ δ i,t +1 , we adopt a more principledtwo-step approach:

Original: (i)
Revised: (i)

Original: We predict quantile ˆ r i,t +1 which estimates r i,t +1 .
Revised: We predict quantile ˆ r i,t +1 which estimates r i,t +1 .

Original: (ii)
Revised: (ii)

Original: We use a pre
Revised: We use a pre

Original: -defined mapping g to define the quantile adjustment ˆ δ i,t +1 ← g (ˆ r i,t +1 ; α ) .
Revised: -defined mapping g to define the quantile adjustment ˆ δ i,t +1 ← g (ˆ r i,t +1 ; α ) .

Original: This also permits research to improve each component independently.
Revised: This also permits research to improve each component independently.

Original: We introduce one alternativefor each step in the Appendix.(i) Quantile Prediction : The quantile prediction ˆ r i,t +1 is estimated by a function of the formf ( S i, : t ; { S j, : t } N +1 j =1 ) .
Revised: We introduce one alternativefor each step in the Appendix.(i) Quantile Prediction : The quantile prediction ˆ r i,t +1 is estimated by a function of the formf ( S i, : t ; { S j, : t } N +1 j =1 ) .

Original: Since ˆ r i,t +1 is supposed to predict r i,t +1 , the rank of the nonconformity score,we impose the constraint that ˆ r i,t +1 should follow a uniform distribution over { jN } Nj =0 .
Revised: Since ˆ r i,t +1 is supposed to predict r i,t +1 , the rank of the nonconformity score,we impose the constraint that ˆ r i,t +1 should follow a uniform distribution over { jN } Nj =0 .

Original: We willfocus on a simple rank prediction method stated below:
Revised: We willfocus on a simple rank prediction method stated below:

Original: [Equation]
Revised: [Equation]

Original: Here, ϵ i,t is the exponentially decayed mean residual of time series i up to time t , and we useβ = 0 . 8 .
Revised: Here, ϵ i,t is the exponentially decayed mean residual of time series i up to time t , and we useβ = 0 . 8 .

Original: Note that taking the rank with Q − 1 achieves the uniformity requirement, and ϵ · ,t could bereplaced by any scoring function that takes into account the temporal information.(ii)
Revised: Note that taking the rank with Q − 1 achieves the uniformity requirement, and ϵ · ,t could bereplaced by any scoring function that takes into account the temporal information.(ii)

Original: Budgeting :
Revised: Budgeting :

Original: Given prediction ˆ r i,t , we propose the following adjustment ˆ δ i,t :=
Revised: Given prediction ˆ r i,t , we propose the following adjustment ˆ δ i,t :=

Original: g B (ˆ r i,t ; α ) :
Revised: g B (ˆ r i,t ; α ) :

Original: [Equation]
Revised: [Equation]

Original: The particular coefficient designused ensures that the following property holds:
Revised: 221 Denote a TQA − B N +1 ,t as the quantile to query by TQA-B. Our particular coefficient design ensures that:

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Using ˆ r ms · , · and g B , {+∀ t, E S N +
+}
Revised: Using ˆ r ms · , · and g B , {+∀ t, E S N +
+}

Original: [Equation]
Revised: [ a TQA − B N +1 ,t ] = α . (Recall a TQA − B N +1 ,t = α − g B ( · ) )

Original: Thus, by Theorem 4.1, TQA-B will not lose coverage if ˆ r and r are independent.
Revised: Thus, by Theorem 4.1, TQA-B will not lose coverage if ˆ r and r are independent.

Original: The followingtheorem provides a worst-case cross-sectional coverage guarantee, regardless of how “bad” ˆ r is:
Revised: The followingtheorem provides a worst-case cross-sectional coverage guarantee, regardless of how “bad” ˆ r is:

Original: Theorem 4.
Revised: Theorem 4.3.

Revised: P

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: The worst-case loss term is typically small: about 0 .
Revised: The worst-case loss term is typically small: about 0 .

Original: 012 for α = 0 .
Revised: 012 for α = 0 .

Original: 1 and N = 100 , although it canalso be also be high for a large α like 0 .
Revised: 1 and N = 100 , although it canalso be also be high for a large α like 0 .

Original: In practice, it is unlikely that ˆ r is worse than a random guess; the coverage is typically greater than 1 − α , as we will see in the experiments and illustrated in
Revised: In practice, it is unlikely that ˆ r is worse than a random guess; the coverage is typically greater than 1 − α , as we will see in the experiments and illustrated in

Original: Figure 2.
Revised: Figure 2.

Original: In the Appendix, we present a more aggressive quantile adjustment function g that providesa weaker guarantee than Theorem 4.3, but empirically performs better.
Revised: In the Appendix, we present a more aggressive quantile adjustment function g that providesa weaker guarantee than Theorem 4.3, but empirically performs better.

Original: Implementation Details To avoid creating infinitely-wide PIs, we could also let ˆ δ = λg (ˆ r ; α ) soa t is bounded away from 0 (In our experiments we choose λ such that a t ≥ 0 . 01 ).
Revised: Implementation Details To avoid creating infinitely-wide PIs, we could also let ˆ δ = λg (ˆ r ; α ) soa t is bounded away from 0 (In our experiments we choose λ such that a t ≥ 0 . 01 ).

Original: Moreover, thespecific form of C presented in this section depends on the concrete distribution of ˆ r .
Revised: Moreover, thespecific form of C presented in this section depends on the concrete distribution of ˆ r .

Original: For example,
Revised: For example,

Original: C would take a different form if ˆ r is defined to be uniform over { jN +1 } N +1 j =1 rather than { jN } Nj =0 .
Revised: C would take a different form if ˆ r is defined to be uniform over { jN +1 } N +1 j =1 rather than { jN } Nj =0 .

Original: Practically, we can simply let C =
Revised: Practically, we can simply let C =

Original: α 2 (1 − α ) − 2 regardless of N .
Revised: α 2 (1 − α ) − 2 regardless of N .

Original: The additional loss in Theorem 4.3will become α 2 / (1 − α ) , and the change in Theorem 4.2 is negligible for a reasonable value of N .
Revised: The additional loss in Theorem 4.3will become α 2 / (1 − α ) , and the change in Theorem 4.2 is negligible for a reasonable value of N .

Original: 4.3 Error-based adjustment (TQA-E)
Revised: 4.3 Error-based adjustment (TQA-E)

Original: Another simple quantile adjustment approach is using a heuristic that depends on the past “errors”:
Revised: Another simple quantile adjustment approach is using a heuristic that depends on the past “errors”:

Original: Define err t = 1 { Y t
Revised: Define err t = 1 { Y t

Original: ̸∈ ˆ C a t } , and increase δ t +1 (conservative) if we see too many errors in { err t ′ } t ′ <
Revised: ̸∈ ˆ C a t } , and increase δ t +1 (conservative) if we see too many errors in { err t ′ } t ′ <

Original: t +1 239 compared with α , and vice versa.
Revised: t +1 240 compared with α , and vice versa.

Original: Since this approach does not depend on the cross-section, we dropthe subscript · N +1 for simplicity.
Revised: Since this approach does not depend on the cross-section, we dropthe subscript · N +1 for simplicity.

Original: We use the following update rule (with ˆ δ 0 = 0 ) inspired by [17] 2 :
Revised: We use the following update rule (with ˆ δ 0 = 0 ) inspired by [17] 2 :

Original: [Equation]
Revised: [Equation]

Original: Note that we do not explicitly impose the restriction that α − ˆ δ t = a t ∈ [0 , 1] , which means the PIcould have infinite width .
Revised: Note that we do not explicitly impose the restriction that α − ˆ δ t = a t ∈ [0 , 1] , which means the PIcould have infinite width .

Original: However, infinite-wide PI means no error, so ˆ δ t +1 will decrease and weresume to a finite PI gradually.
Revised: However, infinite-wide PI means no error, so ˆ δ t +1 will decrease and weresume to a finite PI gradually.

Original: As ˆ δ t depends on the entire error history, it is not immediately clear whether TQA-E is still valid withthe assumption in Theorem 4.1.
Revised: As ˆ δ t depends on the entire error history, it is not immediately clear whether TQA-E is still valid withthe assumption in Theorem 4.1.

Original: Below we state a “no-worse” type theorem for TQA-E:
Revised: Below we state a “no-worse” type theorem for TQA-E:

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: If we assume the nonconformity score’s rank has no temporal dependence, then
Revised: If we assume the nonconformity score’s rank has no temporal dependence, then

Original: [Equation]
Revised: [Equation]

Original: Thus, ˆ C TQA − E is finite-sample cross-sectional 1 − α valid following Theorem 4.1.
Revised: Thus, ˆ C TQA − E is finite-sample cross-sectional 1 − α valid following Theorem 4.1.

Original: Finally, an asymptotic longitudinal validity result can also be shown for long time series 3 :
Revised: Finally, an asymptotic longitudinal validity result can also be shown for long time series 3 :

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: (Asymptotic Longitudinal Coverage)
Revised: (Asymptotic Longitudinal Coverage)

Original: For any time series S , lim T →∞ (cid:80) T − 1 t =0 err t T ≤ α .
Revised: For any time series S , lim T →∞ (cid:80) T − 1 t =0 err t T ≤ α .

Original: Remarks : Although Theorem 4.5 seems to suggest some sort of longitudinal validity, it does notcontradict the hardness claim in Section 4.1, because TQA-E achieves this via infinitely-wide PIs.
Revised: Remarks : Although Theorem 4.5 seems to suggest some sort of longitudinal validity, it does notcontradict the hardness claim in Section 4.1, because TQA-E achieves this via infinitely-wide PIs.

Original: Wealso refer interested readers to [57] as an example of an different heuristic based on errors of a singletime series.
Revised: Wealso refer interested readers to [57] as an example of an different heuristic based on errors of a singletime series.

Original: However, it will require further modifications for finite-sample cross-sectional validity.
Revised: However, it will require further modifications for finite-sample cross-sectional validity.

Original:  Experiments
Revised:  Experiments

Original: In this section, our goal is to to verify the following empirically:
Revised: In this section, our goal is to to verify the following empirically:

Original: TQA maintains cross-sectional coverage and achieves competitive PI efficiency.
Revised: TQA maintains cross-sectional coverage and achieves competitive PI efficiency.

Original: Ignoring temporal dependence in naive split conformal prediction leads to low longitudinal
Revised: Ignoring temporal dependence in naive split conformal prediction leads to low longitudinal

Original: coverage for some TS.
Revised: coverage for some TS.

Original: TQA improves longitudinal coverage.
Revised: TQA improves longitudinal coverage.

Original: Baselines : We use the following state-of-the-art baselines for PI construction: Conformal forecasting
Revised: Baselines : We use the following state-of-the-art baselines for PI construction: Conformal forecasting

Original: RNN (CFRNN (Split)) [47]), a direct application of split-conformal prediction [52] 4 ; Quantile RNN2 Despite the similarity on the surface, [17] has no notion of cross-section.
Revised: RNN (CFRNN (Split)) [47]), a direct application of split-conformal prediction [52] 4 ; Quantile RNN2 Despite the similarity on the surface, [17] has no notion of cross-section.

Original: The update rule as shown in Eq.
Revised: The update rule as shown in Eq.

Original: 9 creates an asymmetry to accommodate for the asymptotic guarantee in Theorem 4.5, which is why the expectation in Theorem 4.4 is not an equality but an inequality.
Revised: 9 creates an asymmetry to accommodate for the asymptotic guarantee in Theorem 4.5, which is why the expectation in Theorem 4.4 is not an equality but an inequality.

Original: If asymptotic coverage is not a concern (because T is small), one could use (1 − γ )
Revised: If asymptotic coverage is not a concern (because T is small), one could use (1 − γ )

Original: ˆ δ t when α − ˆ δ t = a t < 0 as well, in which case Theorem 4.4 becomes an equality, as we discuss in the proof for Theorem 4.4 in the Appendix.
Revised: ˆ δ t when α − ˆ δ t = a t < 0 as well, in which case Theorem 4.4 becomes an equality, as we discuss in the proof for Theorem 4.4 in the Appendix.

Original: In practice (as we observe in our experiments), the difference in behavior is negligible
Revised: In practice (as we observe in our experiments), the difference in behavior is negligible

Original: (QRNN) [54], which directly predicts the two endpoints (represented by two quantiles) of the PI;
Revised: (QRNN) [54], which directly predicts the two endpoints (represented by two quantiles) of the PI;

Original: RNN with Monte-Carlo Dropout (DP-RNN) [15]; Conformalized Quantile Regression with QRNN(CQRNN) [44], which, as the name suggests is a conformalized version of quantile regression;
Revised: RNN with Monte-Carlo Dropout (DP-RNN) [15]; Conformalized Quantile Regression with QRNN(CQRNN) [44], which, as the name suggests is a conformalized version of quantile regression;

Original: Locally adaptive split conformal prediction (LASplit) [29], which uses a normalized absolute error asthe nonconformity score (we follow the implementation in [44]).
Revised: Locally adaptive split conformal prediction (LASplit) [29], which uses a normalized absolute error asthe nonconformity score (we follow the implementation in [44]).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: TQA-E has an natural advantage by using infinitely-wide PIs.
Revised: TQA-E has an natural advantage by using infinitely-wide PIs.

Original: However, even TQA-B still significantly improves the longitudinal coverage rate over all baselines.
Revised: However, even TQA-B still significantly improves the longitudinal coverage rate over all baselines.

Original: [Figure]
Revised: [Figure]

Original: y | ) for t = T − 10 .
Revised: y | ) for t = T − 10 .

Original: Each dataset is normalized so the mean of the residuals is 1.
Revised: Each dataset is normalized so the mean of the residuals is 1.

Original: To cover extreme values, even if allowed to “sacrifice” some less extreme values, the PI on average is expected to get much wider.
Revised: To cover extreme values, even if allowed to “sacrifice” some less extreme values, the PI on average is expected to get much wider.

Original: It is thus surprising that TQA-B could improve both efficiency and the tail coverage.
Revised: It is thus surprising that TQA-B could improve both efficiency and the tail coverage.

Original: Datasets We test our methods and baselines onthe following datasets: Electronic health recordsdata for white blood cell counts (WBCC) prediction( MIMIC [23, 18, 22]), COVID-19 cases prediction( COVID [10]), Electroencephalography trajectory prediction after visual stimuli ( EEG [50]), energy load forecasting ( GEFCom [20]), and healthcare claim amountprediction ( CLAIM ) using data from a large Americanhealthcare data provider.
Revised: Datasets We test our methods and baselines onthe following datasets: Electronic health recordsdata for white blood cell counts (WBCC) prediction( MIMIC [23, 18, 22]), COVID-19 cases prediction( COVID [10]), Electroencephalography trajectory prediction after visual stimuli ( EEG [50]), energy load forecasting ( GEFCom [20]), and healthcare claim amountprediction ( CLAIM ) using data from a large Americanhealthcare data provider.

Original: Among these, we mostly follow [47] in preparing MIMIC , COVID and EEG .
Revised: Among these, we mostly follow [47] in preparing MIMIC , COVID and EEG .

Original: Note that
Revised: Note that

Original: GEFCom is originally a single time series (hourly observations for years).
Revised: GEFCom is originally a single time series (hourly observations for years).

Original: Therefore, we treat each day as asingle TS, and perform a strict temporal splitting (testdata is preceded by calibration data, which is precededby the training data), which means exchangeability isbroken .
Revised: Therefore, we treat each day as asingle TS, and perform a strict temporal splitting (testdata is preceded by calibration data, which is precededby the training data), which means exchangeability isbroken .

Original: We also include a GEFCom-R (andom) versionthat preserves the exchangeability by ignoring the temporal order in data splitting.
Revised: We also include a GEFCom-R (andom) versionthat preserves the exchangeability by ignoring the temporal order in data splitting.

Original: Table 1 provides a briefsummary of the data.
Revised: Table 1 provides a briefsummary of the data.

Original: Due to space constraints, thedetails for each dataset are relegated to the Appendix.
Revised: Due to space constraints, thedetails for each dataset are relegated to the Appendix.

Original: Evaluation Metrics and Experiment Setup We use
Revised: Evaluation Metrics and Experiment Setup We use

Original: RNN as the base point estimator due to its flexibilityand for comparison with [47].
Revised: RNN as the base point estimator due to its flexibilityand for comparison with [47].

Original: We use α = 0 . 1 , anda LSTM ([19]) similar to that in [47] (full implementation details in the Appendix).
Revised: We use α = 0 . 1 , anda LSTM ([19]) similar to that in [47] (full implementation details in the Appendix).

Original: For TQA-E, we useγ =
Revised: For TQA-E, we useγ =

Original: 005 following [17].
Revised: 005 following [17].

Original: For each dataset, we repeatthe prediction task 50 times, and report the mean andstandard deviation of the average coverage rate, tailcoverage rate, and inverse coverage efficiency for thelast 20 time-steps.
Revised: For each dataset, we repeatthe prediction task 50 times, and report the mean andstandard deviation of the average coverage rate, tailcoverage rate, and inverse coverage efficiency for thelast 20 time-steps.

Original: Here, tail coverage rate means theaverage coverage rate of the least-covered 10% of thetime series.
Revised: Here, tail coverage rate means theaverage coverage rate of the least-covered 10% of thetime series.

Original: A high tail coverage rate thus implies better longitudinal coverage.
Revised: A high tail coverage rate thus implies better longitudinal coverage.

Original: Inverse coverage efficiencyis measured by the average PI width divided by themarginal coverage rate (the smaller the better).
Revised: Inverse coverage efficiencyis measured by the average PI width divided by themarginal coverage rate (the smaller the better).

Original: Since
Revised: Since

Original: TQA-E could create infinite PIs, we replace ∞ with 2xthe widest finite PI.
Revised: TQA-E could create infinite PIs, we replace ∞ with 2xthe widest finite PI.

Original: We also include in the Appendix results on full time series, and with Linear
Revised: We also include in the Appendix results on full time series, and with Linear

Original: Regression instead of LSTM to show the model-agnostic nature of TQA.
Revised: Regression instead of LSTM to show the model-agnostic nature of TQA.

Original: Note that although we use equal-length time series in these experiments, for data with variable lengths(such as CLAIM or MIMIC ), one could filter the calibration set before querying the quantile.
Revised: Note that although we use equal-length time series in these experiments, for data with variable lengths(such as CLAIM or MIMIC ), one could filter the calibration set before querying the quantile.

Original: As longas we assume exchangeability conditioning on length, all theoretical analysis still holds.
Revised: As longas we assume exchangeability conditioning on length, all theoretical analysis still holds.

Original: Results The results are presented in Tables 2, 3, and 4.
Revised: Results The results are presented in Tables 2, 3, and 4.

Original: In Table 2, we verify that conformal predictionmethods - CFRNN (Split), CQRNN, LASplit and TQA- are empirically cross-sectionally valid.
Revised: In Table 2, we verify that conformal predictionmethods - CFRNN (Split), CQRNN, LASplit and TQA- are empirically cross-sectionally valid.

Original: Thenon-conformal methods (QRNN and DPRNN) have unreliable coverage.
Revised: Thenon-conformal methods (QRNN and DPRNN) have unreliable coverage.

Original: In Table 3, we show that
Revised: In Table 3, we show that

Original: TQA can greatly improve the (longitudinal) average coverage rate for the worst TS.
Revised: TQA can greatly improve the (longitudinal) average coverage rate for the worst TS.

Original: (TQA-E isconsistently better than TQA-B due to the presence of infinitely-wide PIs.)
Revised: (TQA-E isconsistently better than TQA-B due to the presence of infinitely-wide PIs.)

Original: This is also visualized in
Revised: This is also visualized in

Original: Figure 3.
Revised: Figure 3.

Original: Note that although CQRNN and LASplit do not perform quantile adjustment, they model uncertainty directly, which also helps improve the longitudinal coverage but is less robust.
Revised: Note that although CQRNN and LASplit do not perform quantile adjustment, they model uncertainty directly, which also helps improve the longitudinal coverage but is less robust.

Original: In Table 4,we verify that TQA did not achieve better coverage simply by using very wide PIs (which is howeverthe case for DPRNN on GEFCom ).
Revised: In Table 4,we verify that TQA did not achieve better coverage simply by using very wide PIs (which is howeverthe case for DPRNN on GEFCom ).

Original: This is somewhat surprising because from Figure 4, the marginalgain in coverage decreases fast as a t decreases.
Revised: This is somewhat surprising because from Figure 4, the marginalgain in coverage decreases fast as a t decreases.

Original: The PIs for TQA-B should be wider due to the slightover-coverage, and the convexity (with any quantile adjustment).
Revised: The PIs for TQA-B should be wider due to the slightover-coverage, and the convexity (with any quantile adjustment).

Original: This suggests that TQA-B performsthe budgeting very efficiently to cancel out both effects.
Revised: This suggests that TQA-B performsthe budgeting very efficiently to cancel out both effects.

Original: The efficiency of TQA-E seems low due tothe infinitely-wide PIs (replaced by 2x maximum finite width in this computation), but we will seethat it generates mostly finite PIs, and the median width is still competitive.
Revised: The efficiency of TQA-E seems low due tothe infinitely-wide PIs (replaced by 2x maximum finite width in this computation), but we will seethat it generates mostly finite PIs, and the median width is still competitive.

Original: Finally, we would like to also emphasize that any nonconformity scores could theoretically becombined with TQA.
Revised: Finally, we would like to also emphasize that any nonconformity scores could theoretically becombined with TQA.

Original: In this paper, and in our experiments, we mostly tried to combine the simplestnonconformity scores used in CFRNN (Split) with TQA.
Revised: In this paper, and in our experiments, we mostly tried to combine the simplestnonconformity scores used in CFRNN (Split) with TQA.

Original: The question of how to combine TQA withother nonconformity scores (such as those used in CQRNN or LASplit) is left for future research.
Revised: The question of how to combine TQA withother nonconformity scores (such as those used in CQRNN or LASplit) is left for future research.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we proposed Temporal Quantile Adjustment, or TQA, to quantify uncertainty (createprediction intervals) in time series forecasting with a cross-section.
Revised: In this paper, we proposed Temporal Quantile Adjustment, or TQA, to quantify uncertainty (createprediction intervals) in time series forecasting with a cross-section.

Original: TQA belongs to the framework ofconformal prediction, and the main idea is to adjust the quantile to query using temporal informationcollected so far.
Revised: TQA belongs to the framework ofconformal prediction, and the main idea is to adjust the quantile to query using temporal informationcollected so far.

Original: This allows TQA to work with any model and any nonconformity score design .
Revised: This allows TQA to work with any model and any nonconformity score design .

Original: TQA theoretically is “no-worse” in cross-sectional coverage than vanilla split conformal as long asthe expected value of adjustment is zero, and empirically improves the coverage.
Revised: TQA theoretically is “no-worse” in cross-sectional coverage than vanilla split conformal as long asthe expected value of adjustment is zero, and empirically improves the coverage.

Original: We also proposedtwo variants, TQA-B and TQA-E, both of which significantly outperform baselines in improvingtemporal/longitudinal coverage across many real world datasets.
Revised: We also proposedtwo variants, TQA-B and TQA-E, both of which significantly outperform baselines in improvingtemporal/longitudinal coverage across many real world datasets.

Original: We hope that this work will serve asa foundation for the future design of PIs with both high cross-sectional and temporal coverage.
Revised: We hope that this work will serve asa foundation for the future design of PIs with both high cross-sectional and temporal coverage.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section...•
Revised: [Yes] See Section...•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.•
Revised: [No] The code and the data are proprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] TQA could theoretically lead to worse(although empirically better) coverage, as discussed in Section 4.(c)
Revised: [Yes] TQA could theoretically lead to worse(although empirically better) coverage, as discussed in Section 4.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We do notforesee any potential negative societal impact of this work.(d)
Revised: [N/A] We do notforesee any potential negative societal impact of this work.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.
Revised: Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Apart from exchangeability of time series (which we assume throughout this paper), any assumption is stated rightbefore the relevant theorem.(b)
Revised: [Yes] Apart from exchangeability of time series (which we assume throughout this paper), any assumption is stated rightbefore the relevant theorem.(b)

Original: Did you include complete proofs of all theoretical results?
Revised: Did you include complete proofs of all theoretical results?

Original: [Yes] In the Appendix.3.
Revised: [Yes] In the Appendix.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Original: [Yes] In the supplemental materialand will be published if accepted.
Revised: [Yes] In the supplemental materialand will be published if accepted.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?

Original: [Yes] In the Appendix. (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
Revised: [Yes] In the Appendix.(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Original: [Yes] We repeat all experiments 50 times.
Revised: [Yes] We repeat all experiments 50 times.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?

Original: [Yes] In Appendix.
Revised: [Yes] In Appendix.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] In Appendix.
Revised: [Yes] In Appendix.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] Most datasets are publicly available, except for CLAIM which is
Revised: [Yes] Most datasets are publicly available, except for CLAIM which is

Original: proprietary and provided to us by a company.
Revised: proprietary and provided to us by a company.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] They have been de-identified if applicable.
Revised: [Yes] They have been de-identified if applicable.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applica-
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applica-

Original: ble?
Revised: ble?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on

Original: participant compensation?
Revised: participant compensation?

Original: [N/A]
Revised: [N/A]