O N THE BENEFITS OF MAXIMUM LIKELIHOOD ESTIMATION FOR R EGRESSION AND F ORECASTING
Anonymous authors
Paper under double-blind review
A BSTRACT
We advocate for a practical Maximum Likelihood Estimation (MLE) approach towards designing loss functions for regression and forecasting, as an alternative to the typical approach of direct empirical risk minimization on a speciﬁc target metric.
The MLE approach is better suited to capture inductive biases such as prior domain knowledge in datasets, and can output post-hoc estimators at inference time that can optimize different types of target metrics.
We present theoretical results to demonstrate that our approach is competitive with any estimator for the target metric under some general conditions.
In two example practical settings, Poisson and Pareto regression, we show that our competitive results can be used to prove that the MLE approach has better excess risk bounds than directly minimizing the target metric.
We also demonstrate empirically that our method instantiated with a well-designed general purpose mixture likelihood family can obtain superior performance for a variety of tasks across time-series forecasting and regression datasets with different data distributions.
I NTRODUCTION
The task of ﬁtting a regression model for a response variable y against a covariate vector x ∈ R d is ubiquitous in supervised learning in both linear and non-linear settings (Lathuilière et al., 2020;
Mohri et al., 2018) as well as non-i.i.d settings like multi-variate forecasting (Salinas et al., 2020;
Wang et al., 2019).
The end goal in regression and forecasting problems is often to use the resulting model to obtain good performance in terms of some target metric of interest on the population level (usually measured on a previously unseen test set).
The mean-squared error or the mean absolute deviation are examples of common target metrics.
In this paper, our focus is on the choice of loss function that is used to train such models, which is an important question that is often overlooked, especially in the deep neural networks context where the emphasis is more on the choice of network architecture (Lathuilière et al., 2020).
Perhaps the most common method used by practitioners for choosing the loss function for a particular regression model is to directly use the target metric of interest as the loss function for empirical risk minimization (ERM) over a function class on the training set.
We denote this approach for choosing a loss function as Target Metric Optimization (TMO).
This is especially more common with the advent of powerful general purpose function optimizers like deep networks and has also been rigorously analyzed for simpler function classes (Mohri et al., 2018).
Target Metric Optimization seems like a reasonable approach - if the practitioner knows about the target metric of interest for prediction using the model, it seems intuitive to optimize for the same objective on the training data.
Prior work (both theoretical and applied) has both advocated for and argued against TMO for regression problems.
Many prominent works on regression (Goldberger et al., 1964; Lecue & Mendelson, 2016) use the TMO approach, though most of them assume that the data is well behaved (e.g. sub-Gaussian noise).
In terms of applications, many recent works on time-series forecasting (Wu et al., 2020; Oreshkin et al., 2019; Sen et al., 2019) also use the TMO approach directly on the target metric.
On the other hand, the robust regression literature has long advocated for not using the target metric directly for ERM in the case of contamination or heavy tailed response/covariate behaviour (Huber, 1992; Hsu & Sabato, 2016; Zhu & Zhou, 2021;
Lugosi & Mendelson, 2019a; Audibert et al., 2011; Brownlees et al., 2015) on account of its suboptimal high-probability risk bounds.
However, as noted in (Prasad et al., 2020), many of these methods are either not practical (Lugosi & Mendelson, 2019a; Brownlees et al., 2015) or have sub-optimal empirical performance (Hsu & Sabato, 2016).
Even more practical methods such as (Prasad et al., 2020) would lead to sufﬁciently more computational overhead over standard TMO.
Another well known approach for designing a loss function is Maximum Likelihood Estimation (MLE).
Here one assumes that the conditional distribution of y given x belongs to a family of distributions p ( y | x ; θ ) parameterized by θ ∈ Θ (McCullagh & Nelder, 2019).
Then one can choose the negative log likelihood as the loss function to optimize using the training set, to obtain an estimate ˆ θθθ mle .
This approach is sometimes used in the forecasting literature (Salinas et al., 2020; Davis & Wu, 2009) where the choice of a likelihood can encode prior knowledge about the data.
For instance a negative binomial distribution can be used to model count data.
During inference, given a new instance x (cid:48) , one can output the statistic from p ( y | x (cid:48) ; ˆ θθθ mle ) that optimizes the target metric, as the prediction value (Gneiting, 2011).
MLE also seems like a reasonable approach for loss function design - it is folklore that the MLE is asymptotically optimal for parameter estimation, in terms of having the smallest asymptotic variance among all estimators (Heyde, 1978;
Rao, 1963), when the likelihood is well-speciﬁed.
However, much less is known about ﬁnite-sample, ﬁxed-dimension analysis of MLE, which is the typical regime of interest for the regression problems we consider in this paper.
An important practical advantage for MLE is that model training is agnostic to the choice of the target metric - the same trained model can output estimators for different target metrics at inference time.
Perhaps the biggest argument against the MLE approach is the requirement of knowing the likelihood distribution family.
We address both these topics in Section 5.
Both TMO and MLE can be viewed as offering different approaches to selecting the loss function for a given regression model.
In this paper, we argue that in several settings, both from a practical and theoretical perspective, MLE might be a better approach than TMO.
This result might not be immediately obvious apriori - while MLE does beneﬁt from prior knowledge of the distribution,
TMO also beneﬁts from prior knowledge of the target metric at training time.
Our main contributions are as follows:
Competitiveness of MLE:
In Section 3, we prove that under some general conditions on the family of distributions and a property of interest, the MLE approach is competitive with any estimator for the property.
We show that this result can be applied to ﬁxed design regression problems in order to prove that MLE can be competitive (up to logarithmic terms) with any estimator in terms of excess square loss risk, under some assumptions.
Example Applications:
In Section 4.1, we apply our general theorem to prove an excess square loss bound for the the MLE estimator for Poisson regression with the identity link (Nelder & Wedderburn, 1972; Lawless, 1987).
We show that these bounds can be better than those of the TMO estimator, which in this case is least-squares regression.
Then in Section 4.2, we show a similar application in the context of Pareto regression i.e y | x follows a Pareto distribution.
We show that MLE can be competitive with robust estimators like the one in (Hsu & Sabato, 2016) and therefore can be better than TMO (least-squares).
Empirical Results: We propose the use of a general purpose mixture likelihood family (see
Section 5) that can capture a wide variety of prior knowledge across datasets, including zero-inﬂated or bursty data, count data, sub-Gaussian continuous data as well as heavy tailed data, through different choices of (learnt) mixture weights.
Then we empirically show that the MLE approach with this likelihood can outperform ERM for many different commonly used target metrics like WAPE, MAPE and RMSE (Hyndman & Koehler, 2006) for two popular forecasting and two regression datasets.
Moreover the MLE approach is also shown to have better probabilistic forecasts (measured by quantile losses (Wen et al., 2017)) than quantile regression (Koenker & Bassett Jr, 1978; Gasthaus et al., 2019; Wen et al., 2017) which is the TMO approach in this case.
P RIOR W ORK ON MLE
Maximum likelihood estimators (MLE) have been studied extensively in statistics starting with the work of Wald (1949); Redner (1981), who showed the maximum likelihood estimates are asymptotically consistent for parametric families.
Fahrmeir & Kaufmann (1985) showed asymptotic normality for MLE for generalized linear models.
It is also known that under some regularity assumptions, MLE achieves the Cramer-Rao lower bound asymptotically (Lehmann & Casella, 2006).
However, we note that none of these asymptotic results directly yield ﬁnite sample guarantees.
Finite sample guarantees have been shown for certain problem scenarios.
Geer & van de Geer (2000);
Zhang (2006) provided uniform convergence bounds in Hellinger distance for maximum likelihood estimation.
These ideas were recently used by Foster & Krishnamurthy (2021) to provide algorithms for contextual bandits.
There are other works which study MLE for non-parametric distributions e.g.,
Dümbgen & Ruﬁbach (2009) showed convergence rates for log-concave distributions.
There has been some works (Sur & Candès, 2019; Bean et al., 2013; Donoho & Montanari, 2016; El Karoui, 2018) that show that MLE can be sub-optimal for high dimensional regression i.e when the dimension grows with the number of samples.
In this work we focus on the setting where the dimension does not scale with the number of samples.
Our MLE results differ from the above work as we provide ﬁnite sample competitiveness guarantees.
Instead of showing that the maximum likelihood estimator converges in some distance metric, we show that under some mild assumptions it can work as well as other estimators.
Hence, our methods are orthogonal to known well established results in statistics.
Perhaps the closest to our work is the competitiveness result of Acharya et al.
(2017), who showed that MLE is competitive when the size of the output alphabet is bounded and applied to proﬁle maximum likelihood estimation.
In contrast, our work applies for unbounded output alphabets and can provide stronger guarantees in many scenarios.
C OMPETITIVENESS OF MLE
In this section, we will show that under some reasonable assumptions on the likelihood family, the
MLE is competitive with any estimator in terms of estimating any property of a distribution from the family.
We will then show that this result can be applied to derive bounds on the MLE in some ﬁxed design regression settings that can be better than that of TMO.
We will ﬁrst setup some notation.
Notation: Given a positive semi-deﬁnite symmetric matrix M , (cid:107) x (cid:107) M := x T Mx is the matrix norm of the vector x .
λ ( M ) denotes an eigenvalue of a symmetric square matrix M ; speciﬁcally λ max ( M ) and λ min ( M ) denote the maximum and minimum eigenvalues respectively.
The letter f is used to denote general probability densities.
We use p to denote the conditional probability density of the response given the covariate.
(cid:107)·(cid:107) 1 will be overloaded to denote the (cid:96)
1 norm between two probability distributions i.e (cid:107) p − p (cid:48) (cid:107) 1 :=
(cid:82) | p ( z ) − p (cid:48)
( z ) | dz .
D KL ( p 1 ; p 2 ) will be used to denote the
KL-divergence between the two distributions.
If Z is a set equipped with a norm (cid:107)·(cid:107) , then N ( (cid:15), Z ) will denote an (cid:15) -net
i.e any point z ∈ Z has a corresponding point z (
cid:48) ∈
N ( (cid:15), Z ) s.t (cid:107)
z − z (cid:48) (cid:107) ≤ (cid:15) .
B dr denotes the ball centered at the origin with radius r and S d − 1 r denotes its surface.
We deﬁne [ n ] := { 1 , 2 , · · · , n } .
| · | denotes the size of the enclosed set.
General Competitiveness:
We ﬁrst consider a general family of distributions F over the space
Z .
For a sample z ∼ f (for z ∈ Z and f ∈ F ), the MLE distribution is deﬁned as f z = argmax f ∈F f ( z ) .
We are interested in estimating a property π : F → W of these distributions from an observed sample.
The following deﬁnition will be required to impose some joint conditions on the distribution family and the property being estimated, that are needed for our result.
Deﬁnition 1.
The tuple ( F , π ) , where F is a set of distributions and π : F → W a property of those distributions, is said to be ( T, (cid:15), δ 1 , δ 2 )
-approximable, if there exists a set of distributions ˜ F ⊆ F s.t | ˜ F| ≤ T and for every f ∈ F , there exists a ˜ f such that (cid:107) f − ˜ f
(cid:107) 1 ≤ δ 1 and
Pr z ∼ f (cid:16)(cid:13)(cid:13)(cid:13) π ( f z ) − π ( ˜ f z ) (cid:13)(cid:13)(cid:13) ≥ (cid:15) (cid:17) ≤ δ 2 , where ˜ f
z = argmax ˜ f ∈ ˜
F ˜ f ( z ) and W has a norm (cid:107)·(cid:107) .
The above deﬁnition states that the set of distributions F has a ﬁnite δ -cover, ˜ F in terms of the (cid:96)distance.
Moreover the cover is such that solving MLE on the cover and applying the property π on the result of the MLE is not too far from π applied on the MLE over the whole set F .
This property is satisﬁed trivially if F is ﬁnite.
We note that it is also satisﬁed by some commonly used set of distributions and corresponding properties.
Now we state our main result.
Theorem 1.
Let ˆ π be an estimator such that for any f ∈ F and z ∼ f , Pr( (cid:107) π
( f ) − ˆ π
( z ) (cid:107) ≥ (cid:15) )
≤ δ .
Let F f be a subset of F that contains f such that with probability at least 1 − δ , f z ∈ F f and ( F f , π ) is ( T, (cid:15), δ 1 , δ 2 ) -approximable.
Then the MLE satisﬁes the following bound,
[Equation]
We provide the proof of Theorem 1 in Appendix A.
We also provide a simpler version of this result for ﬁnite distribution families as Theorem 3 in Appendix A for the beneﬁt of the reader.
Competitiveness in Fixed Design Regression:
Theorem 1 can be used to show that MLE is competitive with respect to any estimator for square loss minimization in ﬁxed design regression.
We will ﬁrst formally introduce the setting.
Consider a ﬁxed design matrix X ∈ R n × d where n is the number of samples and d the feature dimension.
We will work in a setting where n (cid:29) d .
The target vector is a random vector given by y n ∈ R n .
Let y i be the i -th coordinate of y n and x i denote the i -th row of the design matrix.
We assume that the target is generated from the conditional distribution given x i such that,
[Equation]
We are interested in optimizing a target metric (cid:96) ( · , · ) given an instance of the random vector y n .
The ﬁnal objective is to optimize,
[Equation]
where H is a class of functions.
In this context, we are interested in comparing two approaches.
TMO (see (Mohri et al., 2018)).
This is standard empirical risk minimization on the target metric where given an instance of the random vector y n one outputs the estimator ˆ h = min h ∈H 1 n (cid:80)
[Equation]
(cid:96) ( y i , h ( x i )) .
MLE and post-hoc inference (see (Gneiting, 2011)).
In this method one ﬁrst solves for the parameter in the distribution family that best explains the empirical data by MLE i.e.,
[Equation]
n
[Equation]
θ ) , where L ( y n ; θ ) :
[Equation]
p
[Equation]
y
[Equation]
x i
[Equation]
θ )
Then during inference given a sample x i the predictor is deﬁned as, ˜ h
( x i ) :
=
argmin ˆ y E y ∈ p ( ·| x i ;ˆ θθθ mle ) [ (cid:96) ( y, ˆ y )] or in other words we output the statistic from the MLE distribution that optimizes the loss function of interest.
For instance if (cid:96) is the square loss, then ˜ h
( x i ) will be the mean of the conditional distribution p ( y | x i ; ˆ θθθ mle ) .
We will prove a general result using Theorem 1 when the target metric (cid:96) is the square loss and H is a linear function class.
Moreover, the true distribution p ( ·| x i ; θθθ ∗ ) is such that E [ y i ]
=
(cid:104) θθθ ∗ , x i (cid:105) for all i ∈ [ n ] i.e we are in the linear realizable setting.
In this case the quantity of interest is the excess square loss risk given by,
[Equation]
where Σ :=
( (cid:80) i x i x Ti ) /n is the normalized covariance matrix, and θθθ ∗ is the population minimizer of the target metric over the linear function class.
Now we are ready to state the main result.
Theorem 2.
Consider a ﬁxed design regression setting where the likelihood family is parameterized by θ ∈ Θ ⊆ B dw and |N ( (cid:15), Θ ∩ B dw ) | ≤ |N ( (cid:15), B dw ) | for a small enough (cid:15) .
Further the following conditions hold,
D KL ( p ( y i ; θ ); p ( y i ; θ (cid:48) ))
≤ L (cid:107) θ − θ (cid:48) (cid:107) 2 . 2.
The negative log-likelihood L ( y n ; θ ) as a function of θ is α -strongly convex and β -smooth, w.p.
at least 1 − δ .
Further suppose there exists an estimator θ est such that E ( θ est ) ≤ ( c 1 + c 2 log(1 /δ ))
η /n , where c 1 , c 2 are problem dependent quantities and η > 0 .
Then the MLE estimator also satisﬁes,
[Equation]
We provide the proof in Appendix C.
The proof involves proving the conditions in Theorem 1 and bounding the size of the cover T .
In order to better understand Theorem 2, let us consider a typical case where there exists a possibly complicated estimator such that E ( θ est )
=
O (( d + log(1 /δ )) /n ) .
In this case the above theorem implies that MLE will be competitive with this estimator up to a log n factor.
In many cases the MLE might be much simpler to implement than the original estimator but would essentially match the same error bound.
We now provide speciﬁc examples in subsequent sections.
A PPLICATIONS OF C OMPETITIVENESS R ESULT
In this section we will specialize to two examples, Poisson regression and Pareto regression, where we show that MLE can be better than TMO through the use of our competitive result in Theorem 2.
4. P OISSON R EGRESSION
We work in the ﬁxed design setting in Section 3 and assume that the conditional distribution of y | x is
Poisson i.e,
[Equation]
for all i ∈ [ n ] .
Poisson regression is a popular model for studying count data regression which naturally appears in many applications like demand forecasting (Lawless, 1987).
Note that here we study the version of Poisson regression with the identity link function (Nelder & Wedderburn, 1972), while another popular variant is the one with exponential link function (McCullagh & Nelder, 2019).
We choose the identity link function for a fair comparison of the two approaches as it is realizable for both the approaches under the linear function class i.e the globally optimal estimator in terms of population can be obtained by both approaches.
The exponential link function would make the problem non-realizable under a linear function class for the TMO approach.
We make the following natural assumptions.
Let Σ =
( (cid:80) ni
=1 x i x Ti ) /n be the design covariance matrix as before and M =
( (cid:80) ni
=1 µ i u i u Ti ) /n , where u i
= x i / (cid:107) x i (cid:107) 2 .
Let χ and ζ be the condition numbers of the matrices M and Σ respectively.
Assumption 1.
The parameter space Θ and the design matrix
X satisfy the following,
• (A1)
The parameter space Θ = {
θ ∈ R d : (cid:107) θ (cid:107) 2 ≤ w, min( (cid:107) θ (cid:107) 2 2 , (cid:104) θ , x i (cid:105) ) ≥ γ > 0 , ∀ i ∈ [ n ] } .
• (A2)
The design matrix is such that λ min (Σ) > 0 and (cid:107) x i (cid:107) 2 ≤ R for all i ∈ [ n ] .
• (A3) Let λ min ( M ) ≥ R 2 4 nγ 2 ( d log(24 χ ) + log(1 /δ )) and (cid:112) λ max ( M )( d log(24 χ )
+ log(1 /δ ))
≤ √ nλ min ( M ) / 16 , for a small δ ∈ (0 , 1) 1 .
The above assumptions are fairly mild.
For instance λ min is ˜Ω(1 /d ) for random Gaussian covariance matrices (Bai & Yin, 2008).
The other part merely requires that λ min ( M ) = ˜Ω
( (cid:112) dλ max ( M ) /n ) .
We are interested in comparing MLE with TMO for the square loss which is just the least-squares estimator i.e ˆ θθθ ls := argmin θ ∈ Θ 1 n (cid:107) y n − Xθ (cid:107) 22 .
Note that it is apriori unclear as to which approach would be better in terms of the target metric because on one hand the MLE method knows the distribution family but on the other hand TMO is explicitly geared towards minimizing the square loss during training.
Least squares analysis is typically provided for regression under sub-Gaussian noise.
By adapting existing techniques (Hsu et al., 2012), we show the following guarantee for Poisson regression with least square loss.
We provide a proof in Appendix D for completeness.
Lemma 1.
Let µ max = max i µ i .
The least squares estimator ˆ θθθ ls satisﬁes the following loss bounds w.p. at least 1 − δ ,
[Equation]
otherwise Now we present our main result in this section which uses the competitiveness bound in Theoremcoupled with the existence of a superior estimator compared to TMO, to show that the MLE estimator can have a better bound than TMO.
In Theorem 4 (in Appendix F), under some mild assumptions on the covariates, we construct an estimator θ est with the following bound for the Poisson regression setting,
[Equation]
The construction of the estimator is median-of-means tournament based along the lines of (Lugosi &
Mendelson, 2019a) and therefore the estimator might not be practical.
However, this immediately gives the following bound on the MLE as a corollary of Theorem 2.
Corollary 1.
Under assumption 1 and the conditions of Theorem 4, the MLE estimator for the
Poisson regression setting satisﬁes w.p. at least 1 − δ ,
[Equation]
The bound in Corollary 1 can be better than the bound for ˆ θθθ ls in Lemma 1.
In the sub-Gaussian region, the bound in Lemma 1 scales linearly with µ max which can be prohibitively large even when a few covariates have large norms.
The bound for the MLE estimator in Corollary 1 has no such dependence.
Further, in the sub-Exponential region the bound in Lemma 1 scales as ˜ O
( d 2 /n ) while the bound in Corollary 1 has a ˜ O ( d/n ) dependency, up to log-factors.
In Appendix G, we also show that when the covariates are one-dimensional, an even sharper analysis is possible, that shows that the MLE estimator is always better than least squares in terms of excess risk.
In Appendix I.8, we perform a simulated experiment that shows a linear growth of the test error w.r.t λ max (Σ) , further showing the efﬁcacy of our theoretical bounds.
4. P ARETO R EGRESSION
Now we will provide an example of a heavy tailed regression setting where it is well-known that
TMO for the square loss does not perform well (Lugosi & Mendelson, 2019a).
We will consider the
Pareto regression setting given by,
[Equation]
) provided (cid:104) θθθ ∗ , x i (cid:105) > γ for all i ∈ [ n ] .
Thus y i is Pareto given x i and E [ y i | x i ]
= µ i := (cid:104) θθθ ∗ , x i (cid:105) .
We will assume that b > 4 such that 4 + (cid:15) -moment exists for (cid:15) > 0 .
As in the Poisson setting, we choose this parametrization for a fair comparison between TMO and MLE i.e in the limit of inﬁnite samples θθθ ∗ lies in the linear solution space for both TMO (least squares) and MLE.
As before, to apply Theorem 2 we need an estimator with a good risk bound.
We use the estimator in
Theorem 4 of (Hsu & Sabato, 2016), which in the ﬁxed design pareto regression setting yields,
[Equation]
Note that the above estimator might not be easily implementable, however this yields the following corollary of Theorem 2, which is a bound on the performance of the MLE estimator.
Corollary 2.
Under assumptions of our Pareto regression setting, the MLE estimator satisﬁes w.p. at least 1 − δ ,
[Equation]
The proof is provided in Appendix H. It involves verifying the two conditions in Theorem 2 in the
Pareto regression setting.
The above MLE guarantee is expected to be much better than what can be achieved by TMO which is least-squares.
It is well established in the literature (Hsu & Sabato, 2016;
Lugosi & Mendelson, 2019a) that ERM on square loss cannot achieve a O (log(1 /δ )) dependency in a heavy tailed regime; instead it can achieve only a O ( (cid:112) 1 /δ ) rate.
C HOICE OF L IKELIHOOD AND I NFERENCE M ETHODS
In this section we discuss some practical considerations for MLE, such as adapting to a target metric of interest at inference time, and the choice of the likelihood family.
Inference for different target metrics:
In most practical settings, the trained regression model is used at inference time to predict the response variable on some test set to minimize some target metric.
For the MLE based approach, once the distribution parameters are learnt, this involves using an appropriate statistic of the learnt distribution at inference time (see Section 3).
For mean squared error and mean absolute error, the estimator corresponds to the mean and median of the distribution, but for several other commonly used loss metrics in the forecasting domain such as Mean Absolute
Percentage Error (MAPE) and Relative Error (RE) (Gneiting, 2011; Hyndman & Koehler, 2006), this estimator corresponds to a median of a transformed distribution (Gneiting, 2011).
Please see
Appendix I for more details.
This ability to optimize the estimator at inference time for different target metrics using a single trained model is another advantage that MLE based approaches have over TMO models that are trained individually for speciﬁc target metrics.
Mixture Likelihood: An important practical question when performing MLE-based regression is to decide which distribution family to use for the response variable.
The goal is to pick a distribution family that can capture the inductive biases present in the data.
It is well known that a misspeciﬁed distribution family for MLE might adversely affect generalization error of regression models (Heagerty & Kurland, 2001).
At the same time, it is also desirable for the distribution family to be generic enough to cater to diverse datasets with potentially different types of inductive biases, or even datasets for which no distributional assumptions can be made in advance.
A simple approach that we observe to work particularly well in practice with regression models using deep neural networks is to assume the response variable comes from a mixture distribution, where each mixture component belongs to a different distribution family and the mixture weights are learnt along with the parameters of the distribution.
More speciﬁcally, we consider a mixture distribution of k components p ( y | x ; θ 1 , . .
, θ k , w 1 , . . .
, w k )
= (cid:80) kj =
1 w j p j ( y | x ; θ j ) , where each p j characterizes a different distribution family, and the mixture weights w j and distribution parameters θ j are learnt together.
We would like to have a mixture distribution that can handle different situations like sparse data, sub-Exponential and sub-Gaussian tails, count data as well as heavy tailed data.
Moreover it should be applicable to continuous valued datasets in general.
We use a three component mixture of ( i ) the constant 0 (zero-inﬂation for dealing with bi-modal sparse data), ( ii ) a continuous version of negative binomial where n and p are learnt and ( iii ) a
Pareto distribution where the scale parameter is learnt.
We provide more details about the continuous version of negative binomial distribution in Appendix I.2.
Our experiments in Section 6 show that this mixture shows promising performance on a variety of datasets.
This will increase the number of parameters and the resulting likelihood might require non-convex optimization.
However, we empirically observed that with sufﬁciently over-parameterized networks and gradient-based optimizers, this is not a problem in practice (Fig. 4 shows a convergence curve).
E MPIRICAL R ESULTS
We present empirical results on two time-series forecasting problems and two regression problems using neural networks.
We will ﬁrst describe our models and baselines.
Our goal is to compare the
MLE approach with the TMO approach for three target metrics per dataset.
[Table]
[Table]
Common Experimental Protocol: Now we describe the common experimental protocol on all the datasets (we get into dataset related speciﬁcs and architectures subsequently).
For a fair comparison the architecture is kept the same for TMO and MLE approaches.
For each dataset, we tune the hyper-parameters for the TMO(MSE) objective.
Then these hyper-parameters are held ﬁxed for all models for that dataset i.e only the output layer and the loss function is modiﬁed.
We provide all the details in Appendix I.
For the MLE approach, the output layer of the models map to the MLE parameters of the mixture distribution introduced in Section 5, through link functions.
The MLE output has 6 parameters, three for mixture weights, two for negative binomial component and one for the scale parameter in Pareto.
The choice of the link functions and more details are speciﬁed in Appendix I.3.
The loss function used is the negative log-likelihood implemented in Tensorﬂow Probability (Dillon et al., 2017).
Note that for the MLE approach only one model is trained per dataset and during inference we output the statistic that optimizes the target metric in question.
We refer to our MLE based models that employ the mixture likelihood from Section 5 as MLE(ZNBP) loss, where ZNBP refers to the mixture components Z ero, N egative- B inomial and P areto.
For TMO, the output layer of the models map to ˆ y and we directly minimize the target metric in question.
Note that this means we need to train a separate model for every target metric .
Thus we have one model each for target metrics in {’MSE’, ’MAE’, ’MAPE’}.
Further we also train a model using the Huber loss 2 .
In order to keep the number of parameters the same as that of MLE, we add an additional 6 neurons to the TMO models.
6. E XPERIMENTS ON F ORECASTING D ATASETS
We perform our experiments on two well-known forecasting datasets used in Kaggle competitions.
The M5 dataset (M
consists
of time series
data
of product sales from 10 Walmart stores in three US states
The data consists of two different
hierarchies:
the
product hierarchy and
store location hierarchy.
For simplicity
, in our experiments
we use only
the product hierarchy consisting of
3K individual time-series and 1.8K time steps.
The Favorita dataset (Favorita, 2017) is a similar dataset, consisting of time series data from Corporación Favorita, a South-American grocery store chain.
As above, we use the product hierarchy, consisting of 4.5k individual time-series and 1.7k time steps.
The task is to predict the values for the last 14 days all at once.
The preceding 14 days are used for validation.
We provide more details about the dataset generation for reproducibility in Appendix I.
[Table] [Table]
The base architecture for the baselines as well as our model is a seq-2-seq model (Sutskever et al., 2014).
The encoders and decoders both are LSTM cells (Hochreiter & Schmidhuber, 1997).
The architecture is illustrated in Figure 1 and described in more detail in Appendix I
We present our experimental results in Table 1.
On both the datasets the MLE model with the appropriate inference-time estimator for a metric is always better than TMO trained on the same target metric, except for WAPE in M5 where MLE’s performance is only marginally worse.
Note that the MLE model is always the best or second best performing model on all metrics, among all
TMO models.
For TMO the best performance is not always achieved for the same target metric.
For instance, TMO (MAE) performs better than TMO (MSE) for the RMSE metric on the Favorita dataset.
In Table 4 we perform an ablation study on the Favorita dataset, where we progressively add mixture components resulting in three MLE models: Negative Binomial, Zero-Inﬂated Negative Binomial and ﬁnally ZNBP.
This shows that each of the components add value in this dataset.
6. E XPERIMENTS ON R EGRESSION D ATASETS
We perform our experiments on two standard regression datasets,
The Bicyle Share dataset (Bicycle, 2017) has daily counts of the total number of rental bikes.
The features include time features as well as weather conditions such as temperature, humidity, windspeed etc.
A random 10% of the dataset is used as test and the rest for training and validation.
The dataset has a total of 730 samples.
The Gas Turbine dataset (Kaya et al., 2019) has 11 sensor measurements per example (hourly) from a gas turbine in Turkey.
We consider the level of NOx as our target variable and the rest as predictors.
There are 36733 samples in total.
We use the ofﬁcial train/test split.
A randomly chosen 20% of the training set is used for validation.
The response variable is continuous.
For all our models, the model architecture is a fully connected DNN with one hidden layer that has 32 neurons.
Note that for categorical variables, the input is ﬁrst passed through an embedding layer (one embedding layer per feature), that is jointly trained.
We provide further details like the shape of the embedding layers in Appendix I.
The architecture is illustrated in Figure 2.
We present our results in Table 2.
On the Bicycle Share dataset, the MLE(ZNBP) based model performs optimally in all metrics and often outperforms the TMO models by a large margin even though TMO is a separate model per target metric.
On the Gas Turbine dataset, the MLE based model is optimal for WAPE and RMSE, however it does not perform that well for the MAPE metric.
In Table 3, we compare the MLE based approach versus quantile regression (TMO based) on the
Bicycle Share dataset, where the metric presented is the normalized quantile loss (Wang et al., 2019).
We train the TMO model for the corresponding quantile loss directly and the predictions are evaluated on normalized quantile losses as shown in the table.
The MLE based model is trained by minimizing the negative log-likelihood and then during inference we output the corresponding empirical quantile from the predicted distribution. MLE(ZNBP) outperforms TMO (Quantile) signiﬁcantly.
Discussion: We compare the approaches of direct ERM on the target metric (TMO) and MLE followed by post-hoc inference time optimization for regression and forecasting problems.
We prove a general competitiveness result for the MLE approach and also show theoretically that it can be better than TMO in the Poisson and Pareto regression settings.
Our empirical results show that our proposed general purpose likelihood function employed in the MLE approach can uniformly perform well on several tasks across four datasets.
Even though this addresses some of the concerns about choosing the correct likelihood for a dataset, some limitations still remain for example concerns about the non-convexity of the log-likelihood.
We provide a more in-depth discussion in Appendix J.
A P ROOF FOR COMPETITIVENESS OF MLE
Theorem 3.
Let ˆ π be an estimator for a property π , such that for any f ∈ F and z ∼ f , Pr( (cid:107) π
( f ) − ˆ π
( z ) (cid:107) ≥ (cid:15) )
≤ δ .
Then the MLE based estimator satisﬁes the following bound,
[Equation]
Proof of Theorem 3.
By triangle inequality and by the properties,
[Equation]
Hence,
[Equation]
We now take expectation of both LHS and RHS of the above equation with respect to the distribution f .
For the LHS
[Equation]
observe that
[Equation]
For the ﬁrst term in the RHS
[Equation]
by the assumption in the theorem.
Combining the above three equations yield that
[Equation]
We now prove that E [1 (cid:107) ˆ π ( z ) − π ( ˜ f z ) (cid:107)≥ (cid:15) ] ≤ |F| δ .
[Equation]
where ( a ) uses the fact that f z is the MLE estimate over set F and hence f z ( z ) ≥ f
( z ) .
( b ) follows from Fubini’s theorem.
Now we prove an extension of our last theorem that can deal with inﬁnite likelihood families.
Proof of Theorem 1.
Let ˜ F be the cover of F f that satisﬁes assumptions in Deﬁnition 1.
By triangle inequality and by the properties of ˜ F , with probability at least 1 − δ 2 − δ ,
[Equation]
π ( f ) − ˆ π ( z ) (cid:107) + (cid:107) ˆ π
( z ) − π
( ˜ f z ) (cid:107) + (cid:107) π
( ˜ f z ) − π ( f z ) (cid:107) ≤ (cid:107) π ( f ) − ˆ π
( z ) (cid:107) + (cid:107) ˆ
π
( z ) − π
( ˜ f z ) (cid:107) + (cid:15)
We have used the fact that f z ∈ F f w.p at least 1 − δ .
Hence with probability at least 1 − δ 2 − δ ,
[Equation]
We now take expectation of both LHS and RHS of the above equation with respect to the distribution f .
For the LHS, observe that
[Equation]
For the ﬁrst term in the RHS
[Equation]
by the assumption in the theorem.
Combining the above three equations yield that
[Equation]
We now prove that E [1 (cid:107) ˆ π ( z ) − π (
[Equation]
f z
[Equation]
] ≤ Tδ + δ 1 .
Let ˜ f be the distribution in ˜ F f that is at most δaway from f .
Then,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the ( a ) follows from the deﬁnition of (cid:96) 1 distance between distributions, ( b ) follows from the properties of the cover, ( c ) uses the fact
that ˜ f
z is the MLE estimate over set ˜ F and hence ˜
f
z ( z ) ≥ ˜ f
( z ) .
( d ) follows from Fubini’s Theorem.
B O THER GENERAL RESULTS FOR THE MLE
In this section, we prove that the log likelihood of the MLE distribution at the observed data point is is close to the log-likelihood of the ground truth at the data point, upto an additive factor that depends on the Shtarkov sum of the family.
Lemma 2.
Let z ∼ f .
The maximum likelihood estimator satisﬁes the following inequality with probability at least 1 − δ ,
[Equation]
where (cid:80) z ∈Z
f z ( z ) is the Shtarkov sum of the family F .
Furthermore, if F is ﬁnite, then
[Equation]
Further is the distribution family is ﬁnite, then
[Equation]
Proof.
Let K = 1 δ (cid:80)
[Equation]
f
[Equation]
z
[Equation] [Equation]
where ( a ) follows from Markov’s inequality.
The ﬁrst part of the lemma follows by taking the logarithm and substituting the value of K .
For the second part, observe that if F is ﬁnite then,
[Equation]
We now prove a result of Shtarkov’s sum, which will be useful in other scenarios.
Lemma 3.
Let w =
π
( z ) for some property π .
[Equation]
Proof.
[Equation]
C G ENERAL F IXED D ESIGN R ESULT
In this section, we will prove Theorem 2 which is an application of Theorem 1 to the ﬁxed design regression setting with the square loss as the target metric.
The theorem holds under some reasonable assumptions.
We will ﬁst prove some intermediate lemmas.
Lemma 4.
Let f ( · ; θ ) =
(cid:81) ni
=1 p ( ·| x i ; θ ) in the ﬁxed design setting.
If (cid:107) θ − θ (cid:48) (cid:107) 2 ≤ δ then,
[Equation]
under the assumptions of Theorem 2.
Proof.
By Pinskers’ inequality we have the following chain,
[Equation]
Lemma 5.
Assume the conditions of Theorem 2 holds.
Let ˜ θθθ
mle := argmax θ ∈N (˜ (cid:15), Θ) L ( y n , θ ) , in the Poisson regression setting.
Then with probability at least 1 − δ we have,
[Equation]
Proof.
We assume that the event in (2) of Theorem 2 holds.
[Equation]
where θ c is the closest point in N (˜ (cid:15), Θ) to ˆ θθθ mle .
Therefore, by deﬁnition
[Equation]
y
[Equation]
˜
[Equation]
mle
[Equation]
By virtue of the strong-convexity we have,
[Equation]
Now by triangle inequality we have,
[Equation]
Now we can use the fact | a − b | ≤ 2 max( √ a, √ b ) |√ a − √ b | and that Θ ⊆ B dw to conclude,
[Equation]
ˆ
[Equation]
Proof of Theorem 2.
Consider the net N ( φ, θ ) .
If φ = δ 2 / (2 nL ) , then by Lemma 4 the net forms a δ cover on F .
Note that under the conditions of Theorem 2 |N ( φ, θ ) | ≤ (3 w/φ ) d .
Next note that in the application of Theorem 1, we should set
[Equation]
Thus if we set φ in place of ˜ (cid:15) in Lemma 5 we would need the following to apply Theorem 1,
[Equation]
Thus, φ can be such that,
[Equation]
From above we can apply Theorem 1 with the above (cid:15) , δ 1 = δ 2 = δ and T = (3 w/φ ) d .
Therefore, we get
[Equation]
We can set δ = δ/ ( T + 5) to then conclude that w.p at least 1 − δ ,
[Equation]
Substituting the bound on log 1 φ yields the result.
D L EAST S QUARES FOR P OISSON
We begin by restating the following general statement about OLS.
Lemma 6 (see Theorem 2.2 in (Rigollet, 2015)) .
Suppose r = rank
( X T X ) and φφφ ∈ R n × r be an orthonormal basis for the column-span of X .
Then we have the following result,
[Equation]
) where ˜ (cid:15)(cid:15)(cid:15) = φφφ T (cid:15)(cid:15)(cid:15) .
Here, (cid:15) =
y n − X θθθ ∗ for the given response sample y n .
Now are at a position to prove Lemma 1.
Proof of Lemma 1.
We are interested in tail bounds on the RHS of (6).
We will ﬁrst analyze tail bounds for a ﬁxed u ∈ S r − 1.
We have the following chain,
[Equation]
where µ i = (cid:104) θθθ ∗ , x i (cid:105) .
Now bounding each term separately we have,
[Equation]
We have, n (cid:89)
i
[Equation]
where we have used the fact that || v || p ≤ || v ||
= 1 for all p ≥ 2 .
Thus we have,
[Equation]
This means that the RV (cid:104) u , ˜ (cid:15)(cid:15)(cid:15) (cid:105) is sub-exponential with parameter ν 2 = 2 µ max and α = 0 .
Thus if t ≤ 4 µ max then we have that for a ﬁxed u ,
P (
[Equation]
4 µ max
[Equation]
Thus by a union bound over the (cid:15) -net with (cid:15) = 2 , we have that wp 1 − δ ,
[Equation]
Thus we get the risk bound wp 1 − δ ,
[Equation]
For the sub-exponential region when t ≥ µ max we get the bound,
[Equation]
We get the bound by setting r = d .
E C OMPETITIVENESS FOR P OISSON R EGRESSION
Lemma 7.
Let f ( · ; θ ) =
(cid:81) ni
=1 p ( ·| x i ; θ ) where p is deﬁned in Eq.
If (cid:107) θ − θ (cid:48) (cid:107) 2 ≤ δ then,
[Equation]
Proof.
By Pinskers’ inequality we have the following chain,
[Equation]
Notice that the absolute value of the derivative of a log x − x wrt x , is upper bounded by a/γ −
if x ≥ γ > 0 and also a ≥ γ .
Therefore, following from above we have,
[Equation]
thus concluding the proof.
Now we prove high probability bounds on the strong convexity and smoothness of the likelihood in the context of Poisson regression.
Lemma 8.
Let χ be the condition number of the matrix M , where u i = x i / (cid:107) x i (cid:107) 2 .
Then with probability at least 1 − 2 δ , we have
[Equation]
provided nλ min ( M ) ≥ 14 γ 2 ( d log(24 χ ) + log(1 /δ )) and (cid:112) λ max ( M )( d log(24 χ ) + log(1 /δ )) ≤ √ nλ min ( M ) / 16 .
Proof.
We start with the expression of the Hessian of L ( y n ; θ ) ,
[Equation]
where u i = x i / (cid:107) x i (cid:107) 2 .
For a ﬁxed unit vector u ∈ S d − 1let us deﬁne,
L ( u , θ ) :
= n (cid:88) i =
y
i u T u
i u Ti u (cid:107)
θ (cid:107):= n (cid:88)
i =
y i v i , where v i = (cid:104) u
i , u (cid:105) 2 / (cid:107) θ (cid:107) 2 .
Following the deﬁnition of sub-exponential RV in (Rinaldo, 2019),
[Equation]
where ν i = 2 µ i v 2 i .
This implies that wp atleast 1 − 2 δ/C we have,
[Equation]
if
[Equation]
Note that the above condition is mild and is satisﬁed when λ min (cid:0)(cid:80) i µ i u
i u Ti (cid:1) ≥ log( C/δ ) / 4 γ 2 , as min u (cid:80) i µ i
v i ≥ λ min (cid:0)(cid:80)
i µ i u i u
Ti (cid:1) .
Here, C is a problem dependent constant that will be chosen later.
Now consider an (cid:15) -net
in (cid:96) 2 norm over the surface unit sphere denoted by N ( (cid:15), d ) .
It is well known that |N ( (cid:15), d ) | ≤ (3 /(cid:15) )
d .
Now any z ∈ S d − 1can be written as z = x + u where x ∈ N ( (cid:15), d ) and u ∈ B
d − 1 (cid:15) .
Therefore, we have that
[Equation]
A similar argument as above gives us,
[Equation]
Thus by an union bound over the net we have wp 1 − 2 δ and other conditions,
[Equation]
if the conditions above hold with C = (3 /(cid:15) )
d .
We can now set (cid:15)
=
1 / (8 ∗ χ ) where χ is the condition number of the matrix (cid:80) i µ i u i u Ti .
Now by virtue of that fact that (cid:112) λ max ( (cid:80) µ i u i u Ti ) log( C/δ ) ≤ λ min (cid:0)(cid:80) i µ i u i u Ti (cid:1) /
16 we have the result.
Now we will prove a result on the smoothness of the negative log likelihood for the Poisson regression setting.
Lemma 9.
With probability at least 1 − 2 δ , we have
[Equation]
if λ min ( M ) ≥ R 2 ( d log 6 + log(1 /δ )) / 4 nγ 2 .
Proof.
We start by writing out the Hessian again but this time upper bounding it in semi-deﬁnite ordering,
[Equation]
For a ﬁxed unit vector u ∈ S d − 1let us deﬁne,
[Equation]
where v i =
R 2 (cid:104) u
i , u (cid:105) 2 /γ 2 .
Proceeding as in Lemma 8 we conclude that, (cid:88)
[Equation]
(13) where ν
i = 2 µ i v 2 i .
This implies that wp atleast 1 − 2 δ/C we have,
[Equation]
if,
[Equation]
Note that the above condition is mild and is satisﬁed when λ min (cid:0)(cid:80) i µ i u
i u Ti (cid:1) ≥ R 2 log( C/δ ) / 4 γ 2 , as min u (cid:80) i µ i
v i ≥ λ min (cid:0)(cid:80)
i µ i u i u
Ti (cid:1) .
Here, C is a problem dependent constant that will be chosen later.
Now consider an 1 / 2 -net in (cid:96) 2 norm over the surface unit sphere denoted by N (0 . 5 , d ) .
It is well known that |N ( (cid:15), d ) | ≤ (3 /(cid:15) )
d .
Now any z ∈ S d − 1can be written as z = x + u where x ∈ N (0 . 5 , d ) and u ∈ B d − 1 0 .
Therefore, we have that,
[Equation]
Thus we set C = 6 d and obtain wp at least 1 − 2 δ ,
[Equation]
provided 4 (cid:112) λ max ( M ) log( C/
[Equation]
≤ λ max ( M
[Equation]
/ √ n .
Proof of Corollary 1.
We show that the conditions of Theorem 2 hold.
Creating Nets: First we need to create an (cid:15) -net over the parameter space Θ .
We start by creating an (cid:15) -net over the sphere with radius w .
Now suppose, (cid:15) <
γ/ 2 R .
Then we remove all centers θ c if ∃ i s.t (cid:104) θ c , x i (cid:105) <
γ/ 2 .
This is a valid (cid:15) -net over Θ as all net partitions that are removed do not have any points lying in Θ .
In subsequent section, we will always follow this strategy to create (cid:15) -nets over subsets of Θ .
Strong Convexity and Smoothness:
From Lemma 8 and 9 we have that,
[Equation]
with probability 1 − O ( δ ) .
KL Divergence: The L in Theorem 2 is bounded by 2 wR 2 /γ
according to Lemma 7.
Combining the above into Theorem 2 and using the estimator in Section F we get our result.
F M EDIAN OF M EANS E STIMATOR FOR P OISSON
In this section we will design a median of means estimator for the Poisson regression model based on the estimator proposed in the work of Lugosi & Mendelson (2019b).
Recall that we have a ﬁxed design matrix X ∈ R n × d with rows x 1 , . . .
, x n , and for each i ∈ [ n ] , y i is drawn from a Poisson distribution with mean µ i = θθθ ∗ · x i .
We will further assume that the design matrix is chosen from an ( L, 4) hyper-contractive distribution.
Mathematically this implies that for any unit vector u
[Equation]
For simplicity we will assume that L =
O (1) .
In the above deﬁnition note that E is the empirical expectation over teh ﬁxed design.
This is a benign assumption and for instance would be satisﬁed if the design matrix is drawn from a sub-Gaussian distribution.
In the general case, the obtained bounds will scale with L .
We have the following guarantee associated with Algorithm 1.
Algorithm 1: Median of Means Estimator
Input: Samples S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } , conﬁdence parameter δ .
Step
1: Compute Σ = E [ xx (cid:62) ] .
Form S (cid:48)
= { x (cid:48)
= y
Σ
− 12 x 1 ,
, x (cid:48) n = y n Σ − 12 x n
Step 2: Randomly partition S (cid:48) into k blocks of size n/k each where k = 20 (cid:100) log( 1 δ ) (cid:101) .
Step 3:
Feed in the k blocks to the median-of-means estimator of Lugosi & Mendelson (2019b) to get ˆ v .
Step 4:
Return ˆ θ = Σ − 12 ˆ v .
Theorem 4.
There is an absolute constant c > 0 such that with probability at least 1 − δ , Algorithmoutputs ˆ θ such that
[Equation]
Proof.
The proof is exactly along the lines of the proof of Theorem 1 in the work of (Lugosi &
Mendelson, 2019b) that we repeat here for the sake of completeness since the original theorem is not explicitly stated for a ﬁxed design setting.
To begin with notice that
[Equation]
Hence if ˆ v is the output of the median of the means estimator in Step 3 of Algorithm 1, then the least squares error of ˆ θ is exactly (cid:107) ˆ v − Σ 12 θθθ ∗ (cid:107) 2 .
For convenience deﬁne µ (cid:48) =
Σ 12 θθθ ∗ and Σ (cid:48)
=
E [ x (cid:48) x (cid:48)(cid:62) ] − Σ 12 θθθ ∗ θθθ ∗(cid:62) Σ 12 .
Exactly as in Lugosi & Mendelson (2019b)
our goal is to show that µ (cid:48) beats any other vector v in the median of means tournament if v is far away from µ (cid:48) .
To quantify this deﬁne
[Equation]
For a ﬁxed vector v of length r , and block B j , µ (cid:48) beats v if
[Equation]
Let us denote by σ i,j ∈ { 0 , 1 } a random variable representing whether data point i is in block j or not.
By Chebychev’s inequality we get that with probability at least 9 / 10 ,
[Equation]
Here p is the probability of a point belonging to block k .
Noting that np = Θ( n/k ) we get that with probability at least 9 / 10 ,
[Equation]
Applying binomial tail estimates we get that with probability at least 1 − e − k/ 180 , µ (cid:48) beats v on at least 8 / 10 of the blocks.
By applying the covering argument verbatim as in the proof of Theoremin Lugosi & Mendelson (2019b) we get that with probability at least 1 − δ , ˆ v will satisfy
[Equation]
Finally, it remains to bound the spectrum of Σ (cid:48) .
We have
[Equation]
[Equation] [Equation]
23) where
[Equation] [Equation]
To bound T 1 , T 2 we note that for any function m ( x ) we have
[Equation]
Using the above inequality and the fact that the design matrix is ( O (1)
[Equation]
4) hyper-contractive we get that
[Equation]
Combining the above we get that
[Equation]
G 1-D P OISSON R EGRESSION
When the covariates are one dimensional, a sharper analysis can actually be performed to show that the MLE dominates TMO in all regimes in the Poisson regression setting considered above.
Lemma 10.
There exists an absolute constant c ≥ 2 such that for any δ ∈ [ 1 n c , 1) , it holds with probability at least 1 − δ that,
[Equation]
The bound above is also tight
i
[Equation]
e with constant probability it holds that
[Equation]
We provide the proofs in later in the section.
Having established the bound for least squares estimator, we next prove the following upper bound on the mean squared error achieved by the MLE.
Theorem 5.
There exists an absolute constant c ≥ 2 such that for any δ ∈ [ 1 n c , 1) , it holds with probability at least 1 − δ that,
[Equation]
It is easy to see that the covariate dependent term in the bound on the mean squared error achieved by ˆ θ mle (deﬁned in Eq.
32) is always better the corresponding term in the bound achieved by ˆ θ ls (deﬁned in Eq. 33).
To see this notice that,
[Equation]
Furthermore
[Equation]
in many cases the bound achieved by ˆ θ mle can be signiﬁcantly better than the one achieved by ˆ θ ls .
As an example consider a skewed data distribution where √ n of the x i ’s take a large value of √ n , while the remaining data points take a value of n (cid:15) , where (cid:15) is a small constant.
In this case we have that,
[Equation]
Proof of Lemma 10.
Notice that
[Equation]
Hence, it is enough to bound the parameter distance, i.e., (ˆ θ ls − θ ∗ ) 2 .
In order to do that we ﬁrst notice that y i
x
i is a sub-exponential random variable with parameters
ν 2 i , α where where ν 2 i = 2 µ i x 2 i , µ i = θ ∗ x i , and α i = 0 .
56 x i (Rinaldo, 2019).
In other words,
[Equation] [Equation]
Thus from the bound on a sum of independent sub-exponential variables we have that,
[Equation]
Thus we have that,
[Equation]
t ≤
[Equation]
otherwise
This means, that w.p at least 1 − 2 δ , |
[Equation]
if
[Equation]
The condition above is satisﬁed under our assumptions on x i and δ , thereby leading to the bound
[Equation]
The bound on the MSE claimed in the lemma then follows.
Now we prove the lower bound.
Again it is enough to show a lower bound on | ˆ θ ls − θ ∗ | .
Notice that
[Equation]
Deﬁne the random variable Z = (cid:80) ni =
( y i − µ i ) x i .
We will show anti-concentration for Z by computing the hyper-contractivity of the random variable.
Recall that a random varibale Z is η -HC (hyper-contractive) if E [ Z 4 ] ≤ η 4 E [ Z 2 ] 2 .
Next we have
[Equation] [Equation]
Hence Z is η -HC with η 4 = (cid:16)
[Equation]
From anti-concentration of hyper-contractive random variables (O’Donnell, 2014) we have
[Equation]
Hence we get that
[Equation] [Equation]
Next notice that since µ i ≥ γ for all i (Assumption 1), we have that 1 + 3 µ i ≤ (3 + 1 γ ) µ i .
This implies that ∆ ≤ (3 + 1 γ ) (cid:16) (cid:80) ni =1 µ i x 2 i (cid:17) 2 .
Hence if γ is a constant then with probability at least 1 3+ 1 γ = Ω(1) we have
[Equation]
Proof of Theorem 5.
Recall that ˆ θ mle is deﬁned as
[Equation]
Setting the gradient of the objective to zero, we get the following closed form expression for ˆ θ mle .
[Equation]
Next, we note that Z = (cid:80) ni
=
y i is a poission random random variable with parameter µ = (cid:80) ni
=1 θ ∗ x i .
From tail bounds for Poisson random variables (Klar, 2000) we have that for any (cid:15) > 0 ,
P [ |
[Equation]
Taking (cid:15)
=
c √ µ log( 1 δ ) , we get that with probability at least 1 − δ , |
[Equation]
(cid:88)
[Equation]
(47) provided that log( 1 δ ) <
µ (that holds for our choice δ , once n is large enough).
Hence we conclude that with probability at least 1 − δ , the mean squared error of ˆ θ mle is bounded by
[Equation]
The bound on the mean squared error follows from the above.
H C OMPETITIVENESS FOR P ARETO R EGRESSION
We verify the conditions of Theorem 2 for the Pareto regression setting.
Lemma 11.
Let f ( · ; θ ) =
(cid:81) ni
=1 p ( ·| x i ; θ ) where p is deﬁned in Eq.
If
[Equation] [Equation]
Proof.
By Pinskers’ inequality we have the following chain,
[Equation]
The second inequality follows from the fact that log x is Lipschitz with parameter L if x > 1 /L .
The last inequality follows from Cauchy-Schwarz and the norm bound on x i ’s.
Now we prove the rest of the conditions
[Equation]
Lemma 12.
We have the following smoothness bound,
[Equation]
Proof.
We start by writing out the Hessian,
[Equation]
Using the fact that (cid:104) θ , x i (cid:105) ≥ γ gives us the result.
Lemma 13.
We have the following strong convexity bound,
[Equation]
Proof.
It follows from the expression of the Hessian in Lemma 12 and using the fact (cid:104) θ , x i (cid:105) ≤ (cid:107) θ (cid:107) 2 (cid:107) x i (cid:107)
Proof of Corollary 2.
We show that the conditions of Theorem 2 hold.
Creating Nets: First we need to create an (cid:15) -net over the parameter space Θ .
We start by creating an (cid:15) -net over the sphere with radius w .
Now suppose, (cid:15) <
γ/ 2 R .
Then we remove all centers θ c if ∃ i s.t (cid:104) θ c , x i (cid:105) <
γ/ 2 .
This is a valid (cid:15) -net over Θ as all net partitions that are removed do not have any points lying in Θ .
In subsequent section, we will always follow this strategy to create (cid:15) -nets over subsets of Θ .
Strong Convexity and Smoothness:
From Lemma 13 and 12 we have that,
[Equation]
KL Divergence: The L in Theorem 2 is bounded by 2 bR/γ according to Lemma 11.
Combining the above into Theorem 2 and using the estimator in (Hsu & Sabato, 2016) we get our result.
I M ORE ON E XPERIMENTS
We provide more experimental details in this section.
I. M ETRICS
The metrics and loss functions used are as follows:
MSE:
The metric is
[Equation]
[Table]
MAE
: The metric is
[Equation]
WAPE:
The metric is
[Equation]
MAPE
: The metric is
n
[Equation]
Quantile Loss: The reported metrics in Table 3 are the normalized quantile losses deﬁned as,
[Equation]
During training the unnormalized version is used for quantile regression.
Huber Loss: The loss is given by,
[Equation]
I. M ORE DETAILS ABOUT THE MIXTURE DISTRIBUTION
We use a mixture distribution between zero, a continuous extension of negative binomial (NB) distribution and Pareto.
The continuous extension of negative binomial is such that the p.d.f at y is proportional to,
[Equation]
given parameters n and p .
That is, the p.m.f of a regular discrete NB distribution is written in terms of Gamma functions and then we extend that to non-integral points up to proportionality, such that the measure sums to 1.
This deﬁnition of NB is the standard implementation in Tensorﬂow (Abadi et al., 2016; Dillon et al., 2017), in order to support both discrete and continuous data .
It has been used in many regression datasets before, especially in the ﬁeld of genomics where the collected data is a discrete continuous mixture (Robinson & Smyth, 2008; Chen et al., 2016; McCarthy et al., 2012).
Why not use a discrete-continuous mixture of zero, discrete NB and Pareto?
A mixture of these three distributions is a discrete continuous mixture, whose CDF is well-deﬁned.
It is possible to deﬁne a likelihood in a standard manner such that it has a component proportional to the pdf of Pareto at all points and to this we add dirac delta functions proportional to magnitude of the negative-binomial pmf at non-negative integral points.
We have an extra mass at zero to account for the zero component.
The sum of the dirac masses along with the integral of the Pareto density sums to one.
However, such a likelihood is not very useful in practice.
For instance, if the data is mostly continuous but not heavy tailed, the log-likelihood would mostly have the Pareto component (because non-integral points have no contribution from the other components) which is not a desirable outcome, as a NB distribution can better model sub-Gaussian and sub-Exponential data in terms of moments.
I. M APPING OF OUTPUTS FOR ZNBP
For the ZNBP model we require an output dimension size of 6.
The ﬁrst three dimensions are mapped through a softmax layer (Goodfellow et al., 2016) to mixture weights.
The fourth dimension is mapped to the ’n’ in negative-binomial likelihood through the link function,
[Equation]
The ﬁfth dimension is mapped to ’p’ of the negative-binomial though the sigmoid function.
The last dimension is mapped to the scale parameter for the Pareto component using the φ ( x ) link function above.
I. H ARDWARE
We use the Tesla V100 architecture GPU for our experiments.
We use Intel Xeon Silver 4216 16-Core, 32-Thread, 2.1 GHz (3.2 GHz Turbo) CPU and our post-hoc inference for MLE is parallelized over all the cores.
I. M ORE DETAILS ABOUT I NFERENCE FOR MLE
We follow the approach of monte-carlo sampling.
For each inference sample x (cid:48) , we generate 10k samples from the learnt distribution p ( ·| x (cid:48) ; ˆ θθθ mle ) .
Then we compute the correct statistics.
For MSE,
RMSE the statistic is just the mean and for WAPE, MAE it is the median.
For a quantile, it is the corresponding quantile from the empirical distribution.
For any loss of the form,
[Equation]
[Figure]
the optimal statistic is the median from the distribution proportional to y β p ( y | x (cid:48) ; ˆ θθθ mle ) (Gneiting, 2011).
Note that the MAPE falls under the above with β = − 1 and relative error corresponds to β = 1 .
The statistic can be computed by importance weighing the empirical samples.
I. M ODELS , H YPERPARAMETERS AND T UNING
[Figure]
For the time-series datasets, the model is a seq-2-seq model with one hidden layer LSTMs for both the encoder and the decoder.
The hidden layer size is h = 256 .
The output layer of the LSTM is connected to the ﬁnal output layer through one hidden layer also having h neurons.
Note that h was tuned in the set [8, 16, 32, 64, 128, 256] and for all datasets and models 256 was chosen.
In order to be keep the number of parameters exactly the same, in the TMO models we add an extra layer with
ReLU with 6 neurons before the output.
We tuned the learning rate for Adam optimizer in the range [1e-5, 1e-1] in log-scale.
The batch-size was also tuned in [64, 128, 256, 512] and the Huber- δ in [ 2 i for i in range(-8, 8)].
The learning rate was eventually chosen as 2.77e-3 for both datasets, as it was close to the optimal values selected for all baseline models.
The batch-size was chosen to be 512 and the Huber-
δ was 32 and 64 for M5 and
Favorita respectively.
For the regression datasets the model is a DNN with one hidden layer of size 32.
For the Bicycle dataset the categorical features had there own embedding layer.
The features [season, month, weekday, weathersit] had embedding layer sizes [2,4,4,2].
We tuned the learning rate for Adam optimizer in the range [1e-5, 1e-1] in log-scale.
The batch-size was also tuned in [64, 128, 256, 512] and the Huber- δ in [ 2 i for i in range(-8, 8)].
The learning rate was eventually chosen as 3e-3 for the gas turbine dataset based on the best perforamce of the baseline models and 1.98e-3 for the gas turbine dataset.
The batch-size was chosen to be 512 and the Huber-
δ was 128 and 32 for Bicyle Share and Gas turbine respectively.
For the ZNBP model we also tune the α parameter in the pareto component between [3 , 4 , 5] .
The value of 3 was selected for all datasets, except for gas turbine where we used α = 5 .
[Figure]
MLE
TMO
[Figure]
[Table]
We used a batched version of GP-UCB (Srinivas et al., 2009) to tune the hyper-parameters.
We used Tensorﬂow (Abadi et al., 2016) to train our models. In Table 5, we provide the number of samples and the dimension of the last layer in each of our datasets. I. D ATASETS
For the Favorita and M5 dataset we used the product hierarchy over the item-level time series.
Along with the item-level (leaf) time-series, we also add all the higher-level (parent) time-series from the product hierarchy (i.e. we add family and class level time-series for Favorita, and department and category level time series for M5).
The time-series for a parent time-series is obtained as the mean of the time-series of its children.
This is closer to a real forecasting setting in practice where one is interested in all levels of the hierarchy.
The metrics reported are over all the time-series (both parents and leaves) treated equally.
The history length for our predictions is set to 28.
For the M5 dataset the validation scores are computed using the predictions from time stepsto 1899, and test scores on steps 1900 to 1913.
For the Favorita dataset the validation scores are computed using the predictions from time steps 1660 to 1673, and test scores on steps 1674 to 1687.
The train test splits are as mentioned in the main paper.
For the Gas turbine dataset we use the ofﬁcial train test split.
For the Bicycle share data there is no ofﬁcial split, but we use a randomly chosen ﬁxed 10% as the test set for all our experiments.
I. A DDITIONAL E XPERIMENTS AND F IGURES
In order to show the dependency of λ max (Σ) in the bound in Corollary 1 we perform a simulated experiment.
We generate a dataset with n = 5000 and d = 10 such that each coordinate of x is distributed i.i.d from a uniform distribution between [0 , U ] .
In out experiment, we vary U in { 1 , 2 , · · · , 9 } .
Then y is genearted from a Poisson distribution with rate (cid:104)
θθθ ∗ , x (cid:105) for a ﬁxed θθθ ∗ .
This varies λ max (Σ) which is equal to U 2 / 12 .
We train and validate on 2500 samples with early stopping and plot the squared loss achieved on the test by the ˆ θθθ mle based estimator in Figure 3 versus λ max (Σ) .
We can clearly see a linear relationship which further validates our theoretical results.
In Figure 4, we plot the average training loss as a function of training iterations for the
MLE(ZNBP) model.
We can see that the loss converges to a minima.
J E XTENDED D ISCUSSION AND L IMITATIONS
We advocate for MLE with a suitably chosen likelihood family followed by post-hoc inference tuned to the target metric of interest, in favor of ERM directly with the target metric.
On the theory side, we prove a general result that shows competitiveness of MLE with any estimator for a target metric under fairly general conditions.
Application of the bound in the case of MSE for Poisson regression and Pareto regression is shown.
We believe that our general result is of independent interest and can be used as a tool to prove competitiveness of MLE for a wide variety of problems.
Such applications can be an interesting direction of future work.
On the empirical side we show that a well designed mixture likelihood like the one from Section 5 can adapt quite well to different datasets as the mixture weights are trained.
As we have mentioned before, the MLE log-likelihood loss in such cases can be non-convex which might lead to some limitations in terms of optimization.
However, we observed that this is usually not a problem in practice and the solutions that can be reached by mini-batch SGD can be quite good in terms of performance.
In conclusion we would recommend the following protocol for a practitioner based on our theoretical and empirical observations:
If the overall problem is convex for TMO but introducing a MLE loss makes the problem nonconvex, then the gains from the MLE approach might be neutralized by the added hardness of the non-convexity introduced.
An example of such a situation is TMO for minimizing square loss on a linear function class, which is just least-squares linear regression, but introduction of a mixture likelihood like the one in Section 4 makes the problem non-convex.
In this case it might be better to stick with TMO or at least proceed with caution with the MLE approach.
Note that if the chosen
MLE retains the convexity of the problem, for example Poisson MLE in Section 4.1, then we would still recommend going with the MLE approach.
However, in many practical scenarios when training using a deep network, the TMO approach is non-convex to begin with, even when the target metric itself is something simple and convex like the square loss.
In such a case we would recommend the MLE approach with a likelihood class that can capture inductive biases about the dataset.
This is because both TMO and MLE are non-convex and it is better to capitalize on the potential gains from the MLE approach.
Finally, note that the user can always choose between TMO and even between different likelihood classes through cross-validation in a practical setting.
If the practitioner would like to forgo the decision making in choosing the likelihood class, we recommend using a versatile likelihood like the mixture likelihood in Sections 5.
We do not anticipate this work to have any negative social or ethical impact.