Modeling Extremes with d -max-decreasing Neural Networks
Abstract
We propose a neural network architecture that enables non-parametric calibration and generation of multivariate extreme value distributions (MEVs).
MEVs arise from Extreme Value Theory (EVT) as the necessary class of models when extrapolating a distributional fit over large spatial and temporal scales based on data observed in intermediate scales.
In turn, EVT dictates that d -maxdecreasing, a stronger form of convexity, is an essential shape constraint in the characterization of
MEVs.
As far as we know, our proposed architecture provides the first class of non-parametric estimators for MEVs that preserve these essential shape constraints.
We show that the architecture approximates the dependence structure encoded by
MEVs at parametric rate.
Moreover, we present a new method for sampling high-dimensional MEVs using a generative model.
We demonstrate our methodology on a wide range of experimental settings, ranging from environmental sciences to financial mathematics and verify that the structural properties of MEVs are retained compared to existing methods.
INTRODUCTION
Modeling the occurrence of extreme events is an important task in many disciplines such as medicine, environmental science, engineering, and finance.
For example, understanding the probability of a patient having an adverse reaction to medication or the distribution of economic shocks is critical to mitigating the associated effects of these events [Dey and
Yan, 2016].
However, these events are rare in occurrence and therefore are often difficult to characterize with traditional statistical tools.
This has been the primary focus of extreme value theory (EVT), which describes how to exFigure trapolate the occurrence of rare events outside the range of available data.
In the one-dimensional case, EVT provides remarkably simple models for the asymptotic distribution of the maximum of an infinite number of independent and identically distributed (i.i.d.)
random variables, which is due to the celebrated Fisher-Tippet-Gnedenko theorem [Embrechts et al., 1997].
These are known as the generalized extreme value (GEV) distributions [de Haan and Ferreira, 2010].
Perhaps more relevant to practical use-cases is to consider simultaneous extremes in the multi-dimensional scenario.
For example, how are extreme weather patterns related in geographical areas or how do extremes of different financial instruments relate?
Unlike the one-dimensional case, multivariate extreme value (MEV) distributions generally do not endow simple analytical forms of the underlying density.
This leads to difficulties in performing inference tasks using conventional methods.
Instead, MEV distributions are characterized by tail dependence functions embedded in extreme value copulas [Pickands, 1981, Gudendorf and
Segers, 2010].
BACKGROUND: EXTREME VALUE COPULAS
We start with a brief overview of multivariate EVT and provide additional background material in Appendix A. Let ∆ d − 1 denote the unit d − dimensional simplex.
Let X i =
( X ( i ) 1 , . . .
, X ( i ) d ) ∈ R d for i ∈ { 1 , . . . , n } be a sample of i.i.d.
random vectors with common continuous probability distribution F , marginals F 1 , . .
F d and copula C F .
The copula C F : [0 , 1] d → [0 , 1] is a function that satisfies:
[Equation]
Let the vector of component-wise maxima be given by: M ( n
[Equation]
, where M ( n ) k
[Equation]
max i =1 ,...,n X ( i ) k for k ∈ { 1 , . . .
, d } .
Let C n be the copula of ¯ M ( n ) given by: ¯ M ( n
[Equation]
where each component-wise maxima M ( n ) k is normalized with sequences of real numbers a ( n ) k > 0 and b ( n ) k such that the corresponding limiting marginal is non-degenerate.
Then the following property known as max-stability holds:
[Equation]
We are interested in finding the limiting copula C of C n as n → ∞ .
The limiting copula is then called an extreme value copula and we say that C F is in the maximum domain of attraction of C , denoted as C F ∈ MDA ( C ) .
The limiting extreme value copula C has the form [Segers, 2012]:
[Equation]
where A is known as a Pickands dependence function that defines the joint dependence of a MEV.
Definition 1 (Pickands dependence function) .
A function
A : ∆ d − 1 →
[1 /d, 1] is called a Pickands dependence function if it satisfies the following properties:
A
is homogeneous of order
and
d -max
-decreasing where d is the dimension;
A satisfies
all
∆ d − 1 .
A ( e k ) = 1 where e k is the k th canonical basis vector.
We give the functional definition of d -max-decreasing in
Appendix B 1 and instead give the spectral correspondence of A here.
Definition 2 (Spectral form of Pickands dependence function) .
For any Pickands dependence function A , there exists a Borel measure (spectral measure) Λ on ∆ d − 1 satisfying (cid:82) ∆ d
[Equation]
1 s k dΛ( s ) = 1 for k ∈ { 1 , . . .
, d } such that
[Equation]
The equality (cid:82)
[Equation]
s k dΛ( s ) = 1 is only used as a convention to standardize the margins, and is not essential in maintaining the d -max decreasing property [Fougères et al., 2013].
To provide some intuition on the aims of this paper, Figure 1 illustrates the relationship between different equivalent representations for a canonical parametric MEV – the symmetric logistic distribution with dependence parameter α = 0 .
05 leaning towards complete dependence and α = 0 .
999 leaning towards complete independence.
The proposed methods estimates the Pickands function (right most column) and recovers the spectral measure (middle column) which enables sampling MEVs (left most column).
Related Work.
A number of techniques have been developed to estimate extreme value copulas from data.
The most relevant to the present work is that by Pickands [1981] where a non-parametric estimator of the Pickands function was first proposed.
Following works such as Capéraà et al.
[1997] and Bücher et al.
[2011] describe alternative takes on estimating the dependence function.
The above methods, however, do not guarantee that the estimate completely satisfy the conditions of a valid Pickands dependence function.
In Marcon et al.
[2017], the authors consider a projection of a nonparametric estimator to a convex function represented as a Bernstein polynomial.
However, the number of parameters required significantly increases with both the amount of data and the dimensionality, making it difficult for higher dimensional problems or problems with many data points.
Finally, a number of Pickands estimators were compared and described in Vettori et al.
[2018], and notably none of the estimators reviewed satisfied all requirements of the Pickands function in cases where d > 2 .
For additional details, please refer to the review on extreme value copulas in Gudendorf and Segers [2012].
A theoretical review of dmax-decreasing functions and their applications to copulas is given in Ressel [2019].
Our Contributions.
We present d -max-decreasing neural networks, an ar-
chitecture constrained to represent Pickands dependence functions of MEVs.
We prove that, in the limit, the proposed architecture can approximate arbitrary Pickands functions.
We propose a generative neural network representation of the spectral density of Pickands functions.
We propose an extension of the Pickands estimator to train neural networks.
NEURAL REPRESENTATIONS OF EXTREME VALUE DISTRIBUTIONS
Our main results propose two architectures for representing
MEVs: a deterministic method for representing the Pickands dependence function, and a stochastic method for representing the spectral measure.
While both represent equivalent quantities, each is more suited for a particular task.
The deterministic representation is more suitable for estimating exceedance probabilities whereas the spectral representation is more suitable for sample generation.
2.1 D-MAX-DECREASING NEURAL NETWORKS
We are interested in finding a flexible parameterization of
A that enforces all the properties given in Definition 1.
The most difficult property to enforce is being d -max-decreasing.
To that end, we propose a new architecture inspired by
Maxout Networks [Goodfellow et al., 2013] and Input
Convex Neural Networks (ICNNs)
[Amos et al., 2017].
The proposed architecture, dubbed d -max Neural Networks (dMNNs) , has additional restrictions to fulfill the conditions of the Pickands dependence function.
Theorem 1 ( d -max-decreasing Neural Architecture) .
Let
A ( m ) θ ( w ) be a function defined as:
[Equation]
where
[Equation]
· ⊙ h ( i − 1) (cid:17) k , h
( i − 1) =
ℓ ( i − 1) ◦ · · · ◦ ℓ (1) ( w ) ,
L ( e ) =
( L ( e 1 ) , . . .
, L ( e d )) T , m is the number of layers, n i is the width of the i th layer, Θ ( i
) ∈
R n i × n i − 1 + are the weights of the i th layer, constrained to be all positive, and e i is the
i th canonical basis vector.
⊙ denotes component-wise multiplication.
Then, A ( m ) θ ( w ) is a d -max-decreasing function.
Moreover,
A ( m ) θ ( w ) represents a valid Pickands dependence function.
Intuition of proof.
The proof uses the idea that
E s [max k =1 ,...,d ( w k s k )] , s ∈ ∆ d − 1 is d -max-decreasing and certain compositions of this function retain this property.
The full proof is given in Appendix C.1.
For notational convenience, we drop the ( m ) unless needed.
To get an intuition behind the structure of the architecture, note that in the single layer case in the limit as n 1 → ∞ , the weights θ correspond to samples of the spectral measure in Definition 2 and the expectation is computed empirically.
While the proposed architecture is guaranteed to enforce the properties of the Pickands function, and is thus d -maxdecreasing, we are also interested in seeing how well it can approximate an arbitrary Pickands dependence function.
We present results in the following theorem:
Theorem 2 (Uniform Convergence) .
Suppose that θ are samples from the true spectral measure and A is the true
Pickands function.
The empirical process
[Equation]
converges to a zero mean Gaussian process as n → ∞ where A (1) θ is a single layer d MNN of width n .
Intuition of proof.
We first establish pointwise convergence.
Then we show A is Lipschitz over a bounded set whose covering number grows in accordance with functions that are P − Donsker.
The full proof is given in Appendix C.2.
The result in Theorem 2 has many implications on the properties of the proposed network since it, for example, allows us to quantify the uncertainty associated with our function estimates.
Using the proposed architecture, we mitigate issues faced by previous estimators, such as [Bücher et al., 2011, Capéraà et al., 1997, Marcon et al., 2017], in enforcing the d -max-decreasing property, inequalities, and endpoints of the function.
2.2 A GENERATIVE MODEL FOR THE SPECTRAL MEASURE
While the spectral measure can be computed from the weights of the proposed d MNN, we propose an alternative representation of the spectral measure using a generative neural network.
We model y ∼ Λ in (2) as the output of a generative neural network G ( · ; ϕ ) ∈ R d + with parameters ϕ , i.e. y =
G ( z ; ϕ ) which maps input samples
z ∼ p z to y , where p z is a distribution that is easy to sample from (such as a multivariate Gaussian distribution).
This leads us to a representation of A in terms of the generator:
[Equation]
where E [ y k ] = 1 .
The expectation is taken empirically with a large number of samples from G .
Remark 1.
The function given by (4) satisfies all the necessary conditions for a valid Pickands function.
Following Remark 1, we informally note that it follows from the universal approximation theorem of neural networks that if G is sufficiently expressive then (4) can represent an arbitrary Pickands dependence function.
Use Cases of Each Representation.
The difference between the representation given by the d MNN (3) and the generative neural network (4) is: in the d MNN case the spectral measure is modeled by a discrete number of elements as dictated by the d MNN architecture, while in the generator case the implicit distribution of the spectral measure is modeled.
The d MNN is useful in representing probabilistic quantities since it provides a deterministic representation of the CDF and therefore it does not exhibit the variance of the generative representation.
On the other hand, the generative model is capable of simulating many realizations of the
MEV, particularly useful for sampling applications.
PARAMETER ESTIMATION
Fitting data to high dimensional copulas is often a difficult task since the probability density function (PDF) is not directly modeled.
In general, specific parametric families are used to make the process easier, such as in Archimedean copulas.
While it is theoretically possible to first obtain the underlying PDF via differentiating the CDF and then fit the d MNN with Maximum Likelihood Estimation (MLE), the procedure is computationally complex, especially in high dimensions.
The main drawback of such a method lies in the need to differentiate the d − variate CDF, since nested differentiation with existing automatic differentiation methods may result in numerical errors [Margossian, 2019].
Instead, we use specific properties of MEVs to transform the parameter fitting procedure into MLE over univariate random variables.
We additionally present the analogs for survival distributions in Appendix D.
3.1 FITTING THE DEPENDENCE FUNCTION
Let F k denote the univariate marginal CDF (which can be fitted using MLE as in Embrechts et al.
[1997] or the
L -moments method of Hosking [1990]) of the k th normalized component wise maxima ¯ M ( n ) k
[Equation]
k
[Equation]
[Equation] [Equation]
d } .
In addition, let w =
[Equation]
. , w d
[Equation]
∈ ∆ d − 1 .
We introduce the transformation on ¯ M ( n ) k :
[Equation]
Then, we have: P [ Z w > z ] =
e − zA ( w ) (for the full derivation, see Section 3 of Gudendorf and Segers [2012]).
This transformation casts the original multi-dimensional distribution into the new variables Z w that are exponentially distributed with rate parameter given by the Pickands dependence function A ( w ) .
From this transformation, we can fit the model
A θ ( w ) to samples Z w using MLE.
This can be done by training the model
A θ ( w ) with stochastic gradient descent (SGD) to match the data points Z w as follows:
[Equation]
where
[Equation]
Alternative losses could be considered by reformulating the loss with respect to the estimators defined in Bücher et al.
[2011] and Capéraà et al.
We empirically found that the MLE approach described in (7) provides the best performance, and it follows naturally from the original formulation of Pickands [1981].
The training procedure is summarized in Algorithm 1.
Algorithm 1 Fitting the Pickands- d MNN to Data
[Equation] [Equation]
[Equation]
3.2 FITTING THE GENERATOR
Recall that we have an equivalent representation of A given by A G in (4) where G ( · ; ϕ ) is a function, with parameters ϕ , of random variables.
We fit the parameters ϕ of the generator by solving the following optimization problem:
[Equation]
with L now defined using the representation of A G in (4):
[Equation]
where y =
( y 1 , . .
. , y d ) =
G ( z ; ϕ ) , y ∈ R d + and z ∈ R
k ∼ p
z with η > 0 as a regularization factor.
Note that the second expectation in (9) is only needed to enforce the margins.
It need not be strictly enforced, enforcing approximately only results in minor changes in the tail index.
The expectations with respect to y in (9) are approximated using the sample mean with samples from the generator.
To summarize the parameter estimation section, we bypass the need to differentiate the CDF and use properties of
MEVs to estimate the parameters of the distribution from data.
Both representations of the Pickands function presented can be used with this technique.
SAMPLING
While learning MEV distributions from data is important for computing probabilities, it is also useful to simulate possible scenarios by sampling from an estimated MEV distribution.
We introduce a sampling technique using the proposed architectures to efficiently sample from arbitrary
MEVs.
To the best of our knowledge, there are no general sampling methods for arbitrary extreme value copula that scale to high dimensions.
This is because MEV sampling algorithms assume knowledge of the spectral measure, and do not consider sampling when given only the Pickands function.
It then becomes necessary to recover the spectral measure from a given Pickands function or from data, which we previously described two methods for doing so.
We additionally note that the traditional method of conditional sampling for copulas is ineffective since it requires both computing high order derivatives and using numerical root-finding techniques.
We base our sampling procedure on algorithms for the infinite dimensional analogue of MEV distributions known as max-stable processes [Dombry et al., 2016].
Max-stable processes have the property that finite dimensional marginals are MEVs and have a spectral representation in terms of the spectral measure Λ for stationary processes.
This ultimately allows us to recast MEV sampling in terms of prior work on sampling from max-stable processes, where established methods exist.
4.1 MARGINS OF MAX-STABLE PROCESSES AS MEV DISTRIBUTIONS
A stationary max-stable process has the form:
[Equation]
where ξ i is the i th realization of a Poisson point process with intensity ξ − 2 d ξ .
y i is the i th sample from the spectral measure.
Additionally, E [ y ( x )] = 1 , x ∈ X is generally assumed to enforce unit Frechet margins.
For a finite number d of { x j } dj =1 , this corresponds to a d -dimensional spectral measure with the same properties as in Definition 2.
The key idea is to use the representation in (10) to sample from the full MEV distribution with only knowledge of the spectral measure.
We use the algorithm mentioned in Hofert et al.
[2018, Algorithm 1] for sampling from the full distribution given samples of the spectral measure.
We give the details of the algorithm in Appendix J Algorithm 4.
4.2 SAMPLING FROM THE DMNN
Suppose we fit a single layer d MNN using Algorithmwith weights given by θ ∈ R w × d + where w is the width of the network and d is the data dimension.
Consider the transformation ˆ θ i,j = θ i,j / (cid:80) dj
=1 θ i,j where we transform the weights of the network to the unit simplex ∆ d − 1 , and i, j refer to the row and column indices.
We then choose a number N and compute max
[Equation]
} ) where ξ i is defined as per (10).
While this method is effective in sampling, a possible issue is the finite number of ˆ θ dictated by the width w of the network.
The generative model on the other hand allows for unlimited generation of samples of the spectral measure.
4.3 SAMPLING FROM THE GENERATIVE MODEL
Suppose we fit a generative model G ( z ; ϕ ) to data following the optimization procedure in (9).
Then sampling proceeds similarly to the case with the d MNN except in this case we do not use the weights of the network explicitly, but sample from the model:
[Equation]
where the notation is maintained as above with p ( z ) defining an easy to sample prior distribution.
As a final note regarding the sampling methods, one particularly useful way of combining the methods is to first estimate
A θ from data using an estimator such as the d MNN.
Then,
[Figure] [Figure]
[Figure] [Figure]
taking the mean squared error (MSE) between the two representations, i.e.
[Equation]
This provides a simple way to recover the spectral density of any given EVC and thus an effective way to sample from arbitrary MEVs.
We detail this algorithm in Appendix J
Algorithm 3.
RESULTS
In this section, we provide numerical results that compare the estimation capabilities of the proposed d MNNbased model with well-known estimators from the literature:
Pickands [Pickands, 1981], CFG [Capéraà et al., 1997], and the estimator described in [Bücher et al., 2011] which we refer to as BDV.
These estimators are described in greater detail in Appendix G. We start by evaluating the performance for estimating survival probabilities on known parametric models, followed by real data.
We conclude with experiments on sampling from a MEV, where we use the proposed generative model for high dimensional data with different dependence structures.
To align with the results in Theorem 2, for the experiments presented in this section, we use a single layer d MNN with a width of 512 .
Additional experiments with two different architectures are presented in Appendix E. Code for experiments is available at 2 .
M SE
CFGPickandsBDVProposed
[Figure]
[Figure] [Figure]
[Figure] [Figure]
( w ) =
(cid:16)(cid:80)
[Equation]
α α ∈ (0 , 1]
A
[Equation]
w
[Equation]
(cid:80)
[Equation]
(cid:18) (cid:80)
[Equation]
( λ i,
[Equation]
w i ) 1 /α
[Equation]
(cid:19)
[Equation]
b α b ∈ (0 , 1] λ i,b ∈ [0 , 1] (cid:80) i ∈
b λ i,b =
Synthetic data.
We consider two canonical families of extreme value distributions known as the symmetric logistic ( A SL ) and the asymmetric logistic ( A ASL ) families where the underlying Pickands function is given by Gudendorf and
Segers [2010] listed in Table 1.
α ∈ (0 , 1] is the parameter modeling the degree of dependence between variables ranging from complete dependence ( α = 0 ) to complete independence ( α = 1 ).
Exact sampling from distributions of this type are described in Stephenson [2003].
Note that for both the symmetric and asymmetric copulas, the marginals are distributed according to the standard Fréchet distribution.
We start by comparing the MSE of survival probabilities for d = 2 where the true Pickands dependence function is given by the symmetric or asymmetric model described above for different degrees of dependence α .
We compute the exact values of the survival probability and consider survival probabilities associated with margins above the 75th percentile.
As shown in Figures 3a and 3b, the proposed
Pickands-
d MNN estimator achieves the lowest MSE performance for most degrees of dependence α for the symmetric logistic model and all the degrees of the asymmetric logistic model.
The proposed method performs worse comparatively in the full dependence case of the symmetric logistic (when all components of the vector are the same) which we suspect is due to difficulties in the optimization procedure of the d MNN.
We additionally showcase the ability of the proposed method to model high dimensional extreme value distributions.
To do this, we train the Pickands-
d MNN with data for d = 256 with α ∈ { 0 . 25 , 0 .
75 , 1 . 0 } and for d = { 256 , 512 , 728 , 1024 } with α = 0 .
Then, we com-
[Figure] [Figure]
pute the MSE between the Pickands-
d MNN and the true
Pickands function via Monte Carlo with 10,000 uniformly sampled points in ∆ d − 1 .
The results are illustrated for varying α in Figures 4a and 4b and for α = 0 .
5 in Figures 4c and 4d.
While all hyperparameters were fixed at the beginning and not fine-tuned, we note that performance may improve if additional fine-tuning is performed using a validation set.
[Figure]
Real data.
We test the proposed estimator with real data on extreme ozone levels ( d = 4) , wind gusts ( d = 10) , commodity prices ( d = 10) , cryptocurrencies to USD conversion rates ( d = 100) , S&P 500 components with sufficient history ( d = 418) , and county-level COVID-19 case counts for California ( d = 58) , New York ( d = 58) and North
Carolina ( d = 100) .
We provide details for each dataset in
Appendix H. For environmental datasets, we compute the maximum over the different sampling periods, while for the financial data we compute the maximum drawdown.
The maximum drawdown is defined as the difference between the minimum and maximum values over a time period normalized by the maximum value.
For the COVID-19 data, we compute the change in case counts over different time scales.
All margins were fit with GEVs using the scipy implementation, which computes the a n , b n normalizing constants.
The main challenge associated with real data is the lack of a ground truth for comparison purposes.
It is extremely difficult to accurately compare different estimators on real data because we can never observe the true distribution of extremes.
Since the purpose of EVT is to extrapolate to the tails from observations not necessarily in the tails, we consider extreme events on different time scales.
If we fit based on extreme observations on shorter time scales and test on extreme observations on longer time scales, we will obtain an estimate of how well the different methods extrapolate to tail probabilities, since longer time scales will have more extreme events.
We compute the accuracy of the different estimators with respect to the empirical estimate on held out data over longer time scales.
Specifically, we choose a series of quantiles where we observe data and compute the difference between the estimated survival probabilities and the empirical estimate calculated from observed data.
This is quantified as:
[Equation]
, where
[Equation]
is the d − dimensional vector of point-wise maxima (or point-wise maximum drawdown over a period of interest), P θ is the estimated survival probability, and Q is a set of thresholds to consider.
We choose Q to be all quantiles such that the empirical probability is greater than 0.
This measures how well the proposed method can extrapolate to greater extremes over longer time scales.
The results are presented in Table 2 and suggest that while most estimators perform similarly, the proposed method most consistently performs the best in terms of the evaluation metric.
We would like to emphasize that empirical evaluation on real data is very challenging, and the high variances prevent
[Table]
[Figure] [Figure]
us from making meaningful statements on the efficacy of any of the methods.
However, from Figures 2d and 5d, we see that our proposed estimator is the only one that satisfies the necessary properties of the Pickands function, which is the main purpose of the proposed method.
Specifically, if we consider the properties of convexity and bounds, the proposed estimator is the only one that retains these.
The other estimators are not convex and achieve values greater than 1, which leads to incorrect probabilities when considering conditional probabilities.
Additional figures in Appendix I showcase this property on additional datasets and
Appendix E Figures 5 to 10 compares these for different architectures.
It is critical that these properties are satisfied so that downstream tasks such as conditional probabilities can be computed.
From the state-of-the-art estimators, the properties are not satisfied and thus the applicability of the estimators is severely limited.
Conditional Prediction.
One important task is computing the conditional survival probability of a random variable.
Suppose we have a d -dimensional EVC and we are interested in computing the probability that the i th component exceeds a threshold conditioned on some subset of the other components, C ⊆ { 1 , . . .
, d } /i .
We can compute this through the relation:
[Equation]
For performance evaluation, we can cast this as a classification problem where we consider features X j , j ∈ C with a positive class associated if the combination of { X i , X j } , j ∈ C appears in the held out data.
Therefore we only have examples of positive classes since all the examples in the held out data are realizations that did occur.
Since we cannot observe the examples that do not occur, we must evaluate how well the method is performing based on the examples that do.
This classification problem with a single class was studied in Lee and Liu [2003], where they propose a metric that behaves similarly to the F1 score in binary classification.
This metric is defined as
[Equation]
11) where r = 1 N
[Equation]
) ≥ 0 .
5 } is the proportion of correctly classified examples on the held out data.
The denominator is approximated by taking an empirical average over the space [0 , 1] |C| +1 .
The score has a range of [0 , ∞ ) where larger values indicate better performance.
Table 3 presents the results of classification on held out data for the COVID datasets where the top 5 most populous counties are used to predict the probability of the 6 th county having the change in case counts at or greater than the observed value.
We consider greater than or equal to due to case counts often being underestimated due to lack of testing.
The results in Table 3 suggest that the method is an effective tool for computing conditional probabilities necessary for classification tasks.
[Table]
Sampling from the copula.
Finally, to determine the efficacy of sampling from an arbitrary Pickands copula, we consider two synthetic examples using the previously described MEV distributions in Table 1.
In this experiment, we train the generator G ( · ; ϕ ) in (9) based on 1000 samples from the target distribution.
We represent G ( · ; ϕ ) as a 2 layer 256 width multi-layer perceptron with ReLU activation functions and set η = 1 .
Since the Pickands function completely determines the dependency of the random variables, we compare the CFG estimate of the Pickands function from generated samples to the true Pickands function as a measure of sampling quality.
We use the CFG estimator due to its ubiquity in the literature and its highly regarded status as a standard estimator for the Pickands dependence function.
The results for generating 225 dimensional samples with varying dependence α ∈ [0 , 1] are shown in Figures 6a and 6b.
The figures suggest that the generative model performs comparatively well for both distributions considered, with the worst performance occurring in the nearly independent cases ( α = 1 ).
This is expected, since independence implies a spectral measure with delta functions on the corners of the simplex, which is difficult to learn (see the bottom row of Figure 1 as an example).
The figures additionally suggest that sampling using the learned weights of the d MNN has lower variance (since the spectral measure in this is a finite discrete approximation) but does not perform as well in sampling as the generative model.
The error of the CFG estimate for the proposed sampling methods (blue and orange) and the exact sampling (green) follow very similar trends in errors, suggesting that both sampling methods are recovering the true spectral measure.
CONCLUDING REMARKS
We introduced a new neural network architecture for modeling MEV distributions while enforcing all the properties of the distribution.
We additionally show that the architecture can approximate any Pickands function, which allows for precise representations of MEV distributions.
Finally, we present a generative model for recovering the spectral representation.
Numerical results are provided to empirically demonstrate the effectiveness of the methods in their respective tasks.
However, there are some limitations of the proposed methods.
Limitations of Pickands- d MNNs and Generative Model.
The main challenge associated with modeling using d MNNs are optimization and architectural choices.
Choosing appropriate hyperparameters is a difficult and opaque task that requires additional care.
This is a case where non-parametric methods are advantageous, at the cost of being unable to guarantee the necessary properties of the function.
In general, we suggest using a wide architecture with single depth, since this is the architecture that most of the theory builds upon.
Additional progress on understanding the training of deep neural networks should improve the representational capabilities of the d MNNs, given its theoretical potential to approximate any Pickands functions to arbitrary precision.
Optimization of the generative model suffers from the same issues.
Furthermore, since the proposed method requires training a neural network for estimation, the non-parametric methods have a significant computational advantage.
In practice, this is not a major issue since these estimators are generally fit once and the proposed method takes only a few seconds on a GPU to fit.
Future Work.
The proposed methods have possible applications in a variety of modeling situations.
One possibility is to extend the application on estimating conditional probabilities of d MNN for other classification tasks, such as out of distribution detection.
Another is in using the spectral measure for finding groups of variables that are extreme simultaneously, such as in [Engelke and Ivanovs, 2021].
Finally, applications of extremes are important in understanding robustness properties of neural networks [Weng et al., 2018] and the proposed work provides foundation for high dimensional extensions.
Acknowledgements
Material in this paper is based upon work supported by the
Air Force Office of Scientific Research under award number
FA9550-20-1-0397.
AH was supported by NSF Graduate
Research Fellowship.