Uncertainty-Aware Reinforcement Learning for Risk-Sensitive Player Evaluation in Sports Game
Abstract
A major task of sports analytics is player evaluation.
Previous methods commonlymeasured the impact of players’ actions on desirable outcomes (e.g., goals orwinning) without considering the risk induced by stochastic game dynamics.
In thispaper, we design an uncertainty-aware Reinforcement Learning (RL) framework tolearn a risk-sensitive player evaluation metric from stochastic game dynamics.
Toembed the risk of a player’s movements into the distribution of action-values, wemodel their 1) aleatoric uncertainty , which represents the intrinsic stochasticity ina sports game, and 2) epistemic uncertainty , which is due to a model’s insufﬁcientknowledge regarding Out-of-Distribution (OoD) samples.
We demonstrate how adistributional Bellman operator and a feature-space density model can capture theseuncertainties.
Based on such uncertainty estimation, we propose a Risk-sensitive
Game Impact Metric (RiGIM) that measures players’ performance over a seasonby conditioning on a speciﬁc conﬁdence level.
Empirical evaluation, based on over9M play-by-play ice hockey and soccer events, shows that RiGIM correlates highlywith standard success measures and has a consistent risk sensitivity.
Introduction
The advancement of player tracking and object detection systems enables data-driven analytics forprofessional sports players.
A common approach to evaluating the contribution of players is toquantify their action impacts.
Previous performance metrics [1, 2, 3, 4] computed the expectedimpact of an action on scoring or winning a game.
However, actions with signiﬁcantly differentdistributions of impact can have the same expectations.
As a result, the expectation-based metricscannot differentiate the risk-seeking actions from the risk-averse ones.
How to distinguish theseactions and assign proper credits to the players remains a fundamental challenge in sports analytics.
An important step toward a risk-sensitive evaluation metric is to model the distributions of actionvalues.
To achieve this goal, distributional Reinforcement Learning (RL) [5] predicts the supportingquantiles of action-value distributions.
Previous distributional RL methods [6, 7, 8, 9, 10] mainlystudied the virtual environments with deterministic transitions (e.g., Atari [11] or Mujoco [12]),whereas sports games are real environments with stochastic game dynamics and complex contextfeatures.
Moreover, player evaluation, as a data-driven task, requires learning from a ﬁxed datasetwithout exploration.
The model must be able to handle the distribution shift experienced at test time.
To mitigate the impact of distribution shift, Ofﬂine RL algorithms [13] typically strive for conservativepolicies that discourage visits to Out-of-Distribution (OoD) states by penalizing corresponding values.
However, this approach cannot be scaled to player evaluation since 1) our goal is not controlling(i.e.
, improving players’ policy), but using RL as an analytical tool to evaluate observed actions inprofessional games, and 2) penalizing actions in OoD states distorts the evaluation.
In this paper, we design an uncertainty-aware RL framework for risk-sensitive player evaluation.
Figure 1 shows a real-world example introducing our key motivations.
Instead of directly inﬂuencing players’ actions like other RL algorithms, we perform a post-hoc calibration of the learned actionvalues.
The main idea of our framework is to model important types of uncertainty in sports games:1)
Aleatoric uncertainty captures the intrinsic stochasticity of game dynamics caused by stochasticrewards, transition dynamics, and policies.
We show that this stochasticity can be captured by adistributional Bellman operator and propagated between action-value distributions by implementing
Temporal-Difference (TD) learning with distributional RL.2)
Epistemic uncertainty is due to the ﬁnite training samples and OoD state-action pairs duringtesting.
Online RL algorithms can overcome this uncertainty given sufﬁcient exploration in theenvironment [8].
However, when we have only a demonstration dataset with limited samples, theinﬂuence of epistemic uncertainty cannot be ignored.
Striving for simplicity and scalability, we modelthe epistemic uncertainty with a Feature-Space Conditional Normalizing Flow (FS-CNF).
Based on the uncertainty estimations, we develop a Risk-sensitive Game Impact Metric (RiGIM)for player evaluation.
RiGIM ﬁlters the predictions for OoD samples and computes the impact ofplayers’ actions by conditioning on a conﬁdence level.
Empirical evaluation shows that RiGIMis highly correlated with standard measures when compared to other baselines.
We measure theaccuracy of action-value predictions by matching them empirically with game results and evaluatethe risk-sensitivity of RiGIM by its correlations with standard measures at different conﬁdence levels.
Contributions.
1) We design an uncertainty-aware RL framework that enables post-hoc calibrationson action values according to their aleatoric and epistemic uncertainties.
We demonstrate how thedistributional Bellman operator captures the aleatoric uncertainty with action-value distributions fromboth a theoretical and an empirical perspective.
3) Striving for scalability, we design a feature-spacedensity estimator that estimates the epistemic uncertainty with a minimum overhead.
To thebest of our knowledge, RiGIM is the ﬁrst risk-sensitive metric that incorporates the inherent risk inenvironment dynamics into player evaluation.
Although this work mainly focus professional sportsgames, our method is general and can be scaled to other stochastic environments or domains.
Related Works
In this section, we introduce previous works that are most related to our approach.
Uncertainty Estimation for RL.
Uncertainty estimates have been widely used in RL for guidingexploration and stabilizing policies.
To achieve this goal, an effective approach is to measure theuncertainty of future returns : [14] designed an uncertainty Bellman equation that estimates thevariance of the Q-value posterior distributions.
Distributional RL methods [6, 7, 8, 9, 10, 15, 16, 17]directly model the distribution of future returns by computing corresponding quantities.
Bootstrapped
DQN methods [18, 19, 20, 21] learn ensembles of action-value Q functions to capture uncertainty.
Some following works [22, 23] extend the Q-ensemble methods to ofﬂine RL settings by learningfrom a ﬁxed dataset.
Instead of focusing on the returns’ uncertainty, an alternative approach is tomeasure the uncertainty of model dynamics : [24, 25] proposed model-based RL approaches thatpredict the uncertainty of dynamics models and penalize the actions leading to uncertain returns.
Instead of separately capturing the epistemic and aleatoric uncertainties, these methods often quantifythe overall uncertainty with a uniﬁed measure (e.g., variance or entropy).
Another line of approaches that heavily relies on uncertainty estimates is Risk-Sensitive RL(RSRL)
The RSRL agents avoid states with high costs by estimating a risk measure (e.g.,variance, Value at Risk (VaR), or Conditional VaR [27]).
These algorithms [28, 29] often learn acontrolling policy based on a known MDP instead of an ofﬂine dataset (with unknown dynamics).
Player Evaluation.
The most common approach to player evaluation is to quantify the impact oftheir actions on game results [30].
Previous works measured action impacts by predicting 1) whethera goal will be scored within a ﬁxed look-ahead horizon [3], 2) the change of winning chances [31],and 3) the expected number of goals within a possession [32].
Some recent works also trainedaction-value Q-functions by dynamic programming [1], deep Sarsa [2, 33] and Inverse RL [4].
Thesemethods compute an expected action value without modeling their potential risk, and they commonlyassume the training and testing datasets are identically distributed.
Uncertainty-Aware RL framework for Player Evaluation
We represent the dynamics in sports games with a Markov Game model and introduce the motivationof estimating the aleatoric uncertainty and the epistemic uncertainty.
3.1 Finite-Horizon Markov Game Model
Player evaluation metrics commonly evaluate players by how much their actions inﬂuence theopportunity of scoring the next goal [2, 3, 34].
Following this setting, we divide a sports game intogoal-scoring episodes , so that each episode 1) begins immediately after a goal (or at the beginning ofthe game), and 2) terminates when the next goal is scored (or the end of the game is reached).
Froman algorithmic perspective, this setting allows us to bound the support of future-goals distribution(Section 4.1) into [0 , 1] , which leads to faster model convergence and more accurate evaluations.
For a scoring episode of length T H , we model its dynamics with a ﬁnite-horizon Markov gamemodel [35] : G = ( S , A , P T , R , O , T H , γ ) .
At a time step t ∈ [0 , T H ] , an agent k performs an actiona k,t ∈
A k at a game state s t ∈ S after receiving an observation o t ∈
O .
This process generates thenext state s t +1 ∼ P T ( ·| s t , a t ) and a reward r k,t
= R k ( s t , a k,t ) .
γ is a discount factor.
In this paper,we consider two agents k ∈ { Home, Away } representing the home and away teams.
The observeddata D =
[( o 1 , a k, 1 , r 1 ) , ( o 2 , a k, 2 , r 2 ) , . . .
, ( o t , a k,t , r t ) , . . . ]
records the action a k,t performed bythe team k who possesses the puck.
To alleviate the partial observability, a game state includes thegame history: s t :=
( o t , a t − 1 , o t − 1 , . . .
, o 0 ) .
The reward r t is a 1-of-2 indicator vector that speciﬁeswhich team ( Home, Away ) scores.
We assign zeros to r t until a team scores at the end of an episode.
3.2 Uncertainty-Aware RL for Player Evaluation
Learning from Ofﬂine Data.
As a data-driven behavior analytic tool, the player evaluation modelassigns values to players’ actions by learning from an ofﬂine dataset.
Under this setting, previousworks [1, 2, 33, 3] commonly assumed the training and testing datasets are sampled from the sameunderlying distribution.
However, in practice, since the behaviors of players may change when someteam members (especially core players or the coach) are traded or hired during a season, there isno guarantee that the visitation frequency of state-action pairs is consistent in different games.
It isnatural to assume a distributional shift between the games in the training and testing dataset: whilethe value function is trained under one distribution, it will be evaluated on a different distribution.
Calibration with Uncertainty.
To alleviate the inﬂuence of distribution shift, ofﬂine RL algorithms [13] commonly discourage the visit to OoD state-action pairs by lowering their values [36]or penalising their rewards [24] or constraining the updated policy [37, 22].
However, to evaluateplayers’ performance, RL is employed as a policy evaluation tool instead of controlling players.
The trajectories in testing games correspond to observed players’ actions that cannot be changed orinﬂuenced by penalties.
Instead, we perform a post-hoc calibration of the predicted action valuesby modeling their epistemic uncertainty , which is due to a lack of knowledge about OoD samples, and thus the resulting model is uncertain about the returns (Section 4.2).
Since our goal is to developa risk-sensitive player evaluation metric, we estimate distributions of action values to model theiraleatoric uncertainty , which is due to the intrinsic stochasticity in the game dynamics (Section 4.1).
In practice, the quantiﬁcation of aleatoric uncertainty can be inﬂuenced by the epistemic uncertaintyof input samples, so we ﬁlter the OoD samples by utilizing our density estimator (Section 5.1) .
Modelling the Uncertainty of Action Values
The aforementioned uncertainty-aware RL framework requires estimating the aleatoric and epistemicuncertainty for a risk-sensitive player evaluation.
In this section, we introduce our distributional-RLapproach for modelling aleatoric uncertainty and a feature-space density estimator for measuringepistemic uncertainty (Figure 2 shows the model architecture).
team attacks and the other defends. add Spectral Normalization to ResNet outputs.
4.1 Distributional RL for Capturing Aleatoric Uncertainty
Distributional RL learns the distribution of the random variable Z k ( s t , a t ) that corresponds to thenumber of future goals when a player of team k performs action a t in state s t .
In other words, wecan think of Z k ( s t , a t ) as a random variable with outcomes corresponding to the sum of discountedrewards (cid:80) T H ι = t γ ι R k,ι ( S ι , A ι ) , where S ι
= s ι , A ι = a ι , S ι +1 is distributed according to P T ( ·| S ι , A ι )and A ι is distributed according to π ( ·| S ι ) .
Following the Quantile-Regression (QR)-DQN method [6],we represent the distribution of Z by a uniform mixture of N supporting quantiles by ˆ Z k ( s t , a t ) =
[-=1 N (cid:80) Ni =
-][Equation]
( s t ,a t ) , where θ k,i estimates the quantile at the quantile level (or quantile index)ˆ τ i =
τ i −
+ τ i( 1 ≤ i ≤ N , and τ i
=
i/N ) and δ θ k,i denotes a Dirac distribution at θ k,i .
The modeloutputs [ θ k, 1 , . .
, θ k,N ] are monotonically increasing quantile values computed with the spline DQN(SPL-DQN) by following [17].
Distributional Bellman Operator.
When the player of a team k performs an action a t at a state s t ,the agent receives a reward R k ( s t , a t ) and moves to a future state s t +1 ∼ P T ( S t +1 | s t , a t )
where theagent’s
next action a t +1 ∼ π
( A t +1 | S t +1 ) .
This stochastic process can be captured by a distributional
Bellman operator T π [6]:
[Equation]
where X ∆ :=
Y indicates that random variables X
and Y follow the same distribution.
Based onthe distributional Bellman operator, we estimate the supporting quantiles of Z by minimizing thequantile
Huber loss (with threshold η ):
[Equation]
k ( s t +
[Equation]
a t +1 ) − θ i,k
( s t , a t ))
where
[Equation]
Illustration of Temporal Projection .
Figure 3 illustrates the mean ± standard deviation of the actionvalues sampled from the predicted distributions ˆ Z ( s, a ) , where s and a follow the players’ movements in a match between the Flyers (Home team) and the Maple Leafs (Away team) on March 15, 2019.
The ﬁgure plots values of the two output nodes.
We highlight critical events and match contexts toshow the context-sensitivity of our predictions.
We show that the predicted distribution of action values can measure the aleatoric uncertainty.
Proposition 1.
Assume the Bellman consistency holds by ˆ Z ∆ :=
R + γ P π ˆ Z where ˆ Z , R arevector-valued random variables and P π is the transition matrix of the stationary policy π , so
P π ( s,a ) , ( s (cid:48) ,a (cid:48) )
=
P ( s (cid:48) | a, s ) π
( a (cid:48) | s (cid:48) ) , the uncertainty of action-value distributions ˆ Z under an entropymeasure H ( · ) can be given by:
[Equation]
where d π = (1 − γ )
( I − γ P π ) − 1 ∈ [0 , 1] | S ||
A |×| S
||
A | is the induced matrix for distributions overstate-action tuples by following policy π and transition P T .
The proof is in Appendix B.1.
Proposition 1 disentangles the entropy of Z into 1) the entropy ofreward variables that quantiﬁes the uncertainty of current rewards, 2) the uncertainty induced by thediscount factor, which determines how much the current uncertainty estimation should be inﬂuencedby the stochasticity of future rewards or transitions (i.e., a small γ reduces this inﬂuence), and 3) alog-absolute determinant of the induced distribution matrix, which measures the amount of stretch orchange that the transition function P T and the policy π apply to the initial state-action distribution.
Proposition 1 demonstrates that the key components for representing the aleatoric uncertainty can becaptured by Z when the Bellman consistency is reached by learning, which suggests the action-valuedistribution learned by distributional RL is an ideal estimator for aleatoric uncertainty.
However,in practice, the estimation of Z cannot be well generalized to all samples because of insufﬁcientexploration or limited training data, so we need to estimate their epistemic uncertainty.
4.2 Density Estimator for Capturing Epistemic Uncertainty
By deﬁnition, epistemic uncertainty stems from limited training data and is inherent to the modelﬁtting these data.
A common measure of epistemic uncertainty is I ( θ ; y | x, D ) [38, 39, 40]: theamount of information gained when the model θ observes the true label y of an input x .
To estimatethis uncertainty measure, previous works [41, 42, 43] utilized deep ensemble models and treated eachensemble as a sample from the posterior p ( θ |D ) .
However, adding additional deep ensemble layersto a distributional RL model (i.e., as a uniﬁed estimator for the joint distribution p ( Z 1 ,...,K , Θ |D ) )signiﬁcantly increases the model complexity.
Striving for model simplicity and scalability for largedatasets, we build a feature space density estimator [40] to detect OoD samples.
In this work, we design a Feature Space Conditional Normalizing Flow (FS-CNF) to estimate sampledensity in the training distribution with a minimum overhead.
The main components are:
Feature Extractor.
FS-CNF shares the same feature extracting layers with the distributional RLmodels.
Note that a common reason why traditional feature extractors might fail to capture epistemicuncertainty is feature collapse [44], which maps OoD samples to iD regions in feature space.
Toprevent this phenomenon, an ideal feature extractor
f θ must be subjected to a bi-Lipschitz constraint:
[Equation]
[Equation] where 1) the lower Lipschitz bound ensures sensitivity to distances in the input space (i.e., sensitiveto OoD samples) and 2) the upper Lipschitz bound ensures smoothness in the feature space (i.e.,prevents overﬁtting to the input variations).
To ensure this bi-Lipschitz condition in practice, wefollow [44] and use residual networks with spectral normalisation as our feature extractor.
Density Estimator.
Based on the extracted features, FS-CNF utilizes the Masked Auto-regressive
Flow (MAF) [45] design that estimates the density of input variables in the training data distributionwith an auto-regressive constraint.
Additionally, to make FS-CNF more sensitive to the abnormalpredictions in OoD samples, the density model is conditioned on the expected returns of actionvalue distributions, so p ( x | z E ) =
(cid:80) i p ( x i | x 1: i − 1 , z E ) where z E :=
{ E [ Z k ( s, a )] } Kk =1 and x :=(
o, a ) (we use o instead of s since a large input dimension inﬂuences the accuracy of estimation).
We implement p ( x i | x 1: i − 1 , z E ) =
N ( x i | µ i , ( exp ( α i )) 2 ) where µ i =
ψ µ i ( x 1: i − 1 , z E ) and α i =
ψ
α i ( x 1: i − 1 , z E ) .
The neural function ψ is implemented by stacking multiple MADE layers [46].
Player Evaluation
[Figure]
In this section, we introduce our player evaluation metric and risk-sensitive rankings.
[Figure]
5.1 Risk-sensitive Impact Metric
[Figure]
Measuring Risk.
We measure the risk of aplayer’s movement with aleatoric uncertaintysince modeling the intrinsic stochasticity of thegame dynamics is consistent with the goal ofsports analytics.
However, as Figure 4 shows,when we quantify the aleatoric uncertainty withpredictions from distributional RL, the performance is inﬂuenced by the density of input samples (i.e., OoD samples have a lower accuracy).
An effective approach to verify whether ˆ Z k ( s, a )accurately captures the true aleatoric uncertaintyis to check the epistemic uncertainty for eachinput data ( s, a ) [40]: 1) a high input density( p ( ·| z E ) ≥ (cid:15) ) indicates low epistemic uncertainty, (i.e., ( s, a ) is inD) and we can trust thealeatoric uncertainty estimated by distributional
RL. 2) a low input density ( p ( ·| z E ) < (cid:15) ) indicates high epistemic uncertainty (i.e., the input is OoD, and we do not use the model prediction).
Inpractice, (cid:15) is determined based on the validation dataset.
Risk-Sensitive Action Impact.
The impact φ ( s, a ) measures how much an action a changes thefuture return of a player’s team.
In terms of the value function, this is the change in action value dueto a player’s movement.
Previous works [1, 2, 3] computed the impact with the expected next-goalreturn Q ( s t , a t ) =
E [ Z ( s t , a t )] .
Q ( · ) does not take into account the inherent variability of the returnsand thus cannot estimate the risk of an action.
To understand how players respond to risk, we proposea Risk-sensitive Game Impact Metric (RiGIM ) based on ˆ Z k ( s, a ) and p ( ·| z E ) :
[Equation]
where c ∈ [0 , 1] is the conﬁdence level, Z c denotes the (1 − c ) th quantile in Z ( · ) and n ( s, a, l )denotes the number of times that a player l performs action a at a state s in the testing dataset D (cid:48) .
We omit the term r t since r t = 0 except at scoring step T , and s T is the terminal state of a scoringepisode.
φ ( · ) can either be 1) risk-averse (with a large c ), with better sensitivity to bad outcomes or2) risk-seeking (with a small c ), with better sensitivity to positive outcomes.
5.2 Case Study: Player Ranking in Testing Games
We rank players according to their RiGIM scores in the NHL testing games (see the experimentsetting in Section 6) at different conﬁdence levels.
Tables 1 and 2 illustrates how a domain expertcould use our method to gain insight into which types of players exhibit different risk-taking behavior.
Table 1 shows a risk-seeking ranking (conﬁdence c = 0 . 2 ), which favors offensive players (e.g.,
Centres (C)) with strong scoring ability.
Aleksander Barkov, who scores the most points in thesegames, is captured by this ranking.
When we set conﬁdence c to 0.8, Table 2 shows a risk-averseranking which highlights players in defensive positions (e.g., Defensive (D)).
John Klingberg, thedefenseman with the most assists, is listed in the top-10 players.
We believe the difference between
Table 1 and 2 is because that RiGIM is correlated with the action types a player performs.
Intuitively,in ice-hockey, some actions are more risk-seeking (shot) in terms of scoring while other actions arerisk-averse (carry or pass).
In general, players on the backcourt (e.g., defensemen) are more likely toperform risk-averse actions that have smaller variance on the scoring chances.
[Table] [Table]
Empirical Evaluation
Dataset.
Our experiments utilize both a ice-hockey and a soccer dataset from the National Hockey
League (NHL) and major European soccer leagues, which contains 9,213,371 events, covering 195teams, 4,172 games
[Table]
and 6,513 players.
These datasets consist of events around the ball.
Each eventrecords the identity and action of the player possessing the ball, with time stamps and features ofthe game context (see all the game features in Appendix A.1).
To the best of our knowledge, thisis the most extensive study for player evaluation .
Note that we do not utilize virtual environmentslike Atari [11] or Mujoco [12] because 1) the dynamics in these environments are deterministicwithout uncertainty to be modeled.
2) this paper mainly studies sports games that have stochasticdynamics [30], which are valid test-beds for our method.
Experiment Settings.
We divide the dataset into a training set (80%), a validation set (10%), and atesting set (10%) according to game dates, so that games in the testing set happened after the gamesin the training and validation set.
To predict the action values in the testing games, the metric mustremain robust to OoD data points.
We report the results averaged over 5 independent runs.
Comparison Methods.
We employ an ablation design that iteratively removes parts from RiGIM .
GIM removes the uncertainty estimator by directly using a Deep Recurrent Q-Network for estimatingaction values [2].
T0-GIM removes the recurrent model and uses a Deep Q-Network (DQN) forthe value function.
We then replace the RL framework with a supervised learning framework forestimating action values by following VAEP [3].
Instead of using function approximators, Scoring
Impact (SI) [1] implements the tabular-based value iteration algorithm for computing action valuesfrom discretized spatial and temporal features.
Expected-Goal (EG) metric directly uses the expectedaction values instead of impact values for measuring player performance.
The last metric Plus-Minus( + / − ) is based on game statistics and measures the goal-gain with and without the player on court.
We summarize these metrics in Table 3.
[Table]
To study how well SP-CNF boosts model performance, we compare 1) a Gaussian Discriminant
Analysis (GDA)-RiGIM metric that replaces
SP-CNF with GDA [40], and 2) a Naive (Na)-
RiGIM metric that removes the epistemic uncertainty estimator and uses all the predicteddistributions to compute players’ impact (see
Appendix A.2 and A.3 for more details.).
6.1 Player Evaluation Performance: Correlations with Standard Measures
We follow [2, 4] and compute the correlations between player ranking metrics and standard measureson the testing games in a game season, because 1) the player (or agent) evaluation task has noground-truth labels or rewards to maximize, 2) the correlation to all measures (including penaltymeasures) can measure whether the metrics can form a comprehensive evaluation to a player’s overallperformance.
We study 11 success measures for ice hockey and 8 success measures for soccer.
Tomake the results more comprehensive, we also add 5 penalty measures .
The studied measures arepopular measures from the NHL and soccer statistics websites 1 .Following
the popular risk-measureslike Conditional Value at Risk (CVaR), we treat c as a hyper-parameter.
Since the comparisonmethods are expectation-based metrics, we study the options of 1) setting the conﬁdence RiGIM to 0.5 for a fair comparison 2) empirically determining c with the validation set ( c ∗ = 0 . 34and 0 . 49 for the ice hockey and soccer datasets).
The risk-sensitive results are shown in Section 6.2.
Tables 4 and 5 show the average correlations in 5 independent runs on the testing dataset (see Tables
C.1 and C.2 in Appendix for the complete mean ± standard deviation results).
RiGIM achieves thehighest correlations with 14 out of 19 success measures and the smallest correlations with 3 outof 5 penalty measures in ice hockey and soccer games.
This observation shows that RiGIM is acomprehensive metric that can detect both the positive and the negative inﬂuence of a player.
If weremove SP-CNF or replace it with other uncertainty estimators, most correlations become weakerexcept for the correlations with the SHP and SHG measures.
This is because SHP and SHG rarelyhappen in a season (scoring with fewer players on ice is difﬁcult).
SP-CNF detects this phenomenonand assigns small densities to these rarely-happened events.
RiGIM ﬁlters the event with a smalldensity (see equation 5), which might cause the loss of information and make its correlation with
SHP and SHG less signiﬁcant.
SI correlates well with goal measures (Goals and Game Winning
Goals) but has relatively poor correlations with other measures.
This is because assigning an adequatevalue for all actions, including those with only intermediate effects on goal scoring, requires creditpropagation over longer sequences, where neural nets are better at credit propagation than discretizingand using a tabular representation..
For other risk-neutral methods, their performance is generallyless satisfying when compared with risk-aware methods, especially for the + / − metric, which showsthe importance of capturing risk and modeling the context features. PPP), Time On Ice (TOI)
[Table] and Shots (S).
The penalty measure is Penalty Minute (PIM)
[Table]
[Table]
6.2 Sensitivity to Risk: Correlations Conditioning on Different Conﬁdence Levels
We measure whether RiGIM is sensitive to the risk by its correlations with the standard measures,where RiGIM is conditioned on a speciﬁc conﬁdence level c (from 0 to 1), for example, RiGIM ( c ) ,which indicates with probability c that the players’ impact should be at least RiGIM ( c ) .
Figures 5 and 6 show the correlations at different conﬁdence levels for ice-hockey and soccer games.
RiGIM is sensitive to risk, in the sense that it has different correlations with standard measures atthese conﬁdence levels, whereas GIM, as a risk-neutral metric, is unaware of the risk, and thus itscorrelations remain unchanged.
Compared to other baselines, our RiGIM generally maintains highercorrelations with success measures and lower correlations with penalty measures.
The exceptions are the correlations with the SHP, SHG, and OTG.
For the same reason as discussed above, SP-CNFmay ﬁlter them during testing.
We ﬁnd when c becomes smaller, RiGIM ( c ) becomes risk-seeking,and thus achieves a higher correlation with success measures.
However, the correlations drop when capproaches 0.
This is because ˆ Z 0 k denotes the estimates at the largest quantile level (see equation 5),which is corresponding to the most optimistic estimation action value (i.e., label scoring for all theshots).
The overly risk-seeking estimation can induce a mismatch between estimated values andgame facts, and thus cannot reﬂect the real contributions of players.
6.3 The Prediction Accuracy: Match Future Scoring Frequencies With Action-Values
We study how well the predicted action-value distributions match the real next-goal scoring frequencies under discrete game contexts.
These game contexts are constructed by dividing the continuousstate space into discrete bins.
To calculate the empirical scoring frequency associated with each bin,we assign an observed state s to a bin B according to the values of three discrete context features inthe current observation: Manpower Differential, Goal Differential, and Period.
The empirical andestimated scoring probabilities for a bin (with size | B | ) are deﬁned as follows:
a) Empirical Scoring Chances : for each s ∈ B , we set g k ( s ) =
if the observed scoring-episodecontaining state s ends with a goal by team k .
Then G ∗ k ( B ) =
| B | (cid:80) s ∈ B g k ( s ) .b)
Estimated Scoring Chances : for each s ∈ B , given N samples from the calibrated distributionz k ( s ) ∼ ˆ Z k ( s, a ) I p ( ·| z E ) ≥ (cid:15) , the estimated chances are: ˆ G k ( B ) =
1 N | B | (cid:80) s ∈ B (cid:80) Nn =1 z k,n ( s ) .
Table 6 shows the average absolute difference between ˆ G and G ∗ based on 5 independent runs.
Weimplement a Wilcoxon signed rank test [47] to study whether the predictions from baseline methodsare different from that of RiGIM for all samples.
Our baselines are the learning-based action-valuesmetrics for the context-aware evaluation.
RiGIM achieves a minimum distance in 12 out of 17 bins.
This is because 1) FS-CNF outperforms GDA by computing a more accurate uncertainty estimatorfor ﬁltering OoD states and 2) the distribution estimates contain richer information than expectationestimates, allowing ˜ Z ( · ) to better match the scoring frequencies than Q ( · ) .
This observation isconsistent with the ﬁndings in [6]. [Table]
6.4 Limitations
A Study for Sports Games.
This work uses stochastic sports games as the testbeds for uncertaintyestimation, but we argue the same methods can be easily migrated to other applications in stochasticenvironments such as autonomous driving or healthcare.
Our approach of measuring aleatoric andepistemic risks can be adapted to ofﬂine RL for learning conservative policies.
Evaluation Instead of Controlling.
Our method focuses on player evaluation instead of control.
Webelieve both tasks are challenging with respect to different aspects.
Evaluation requires the actionvalues to accurately reﬂects an agent’s real contribution to game-winning.
An ideal evaluation metriccan provide in-game predictions of game outcomes, which is important for the sports industry.
Conclusion and Future Work
In this paper, we designed an RL framework for quantifying the aleatoric uncertainty and the epistemicuncertainty from stochastic sports datasets.
This framework enabled distributional RL and an FS-CNFmodel to estimate both uncertainties, with which we proposed a risk-sensitive evaluation metric
RiGIM .
Empirical results show that RiGIM correlates well with success measures and the correlationis sensitive different risks.
A direction of future work is to extend our model to other domains.
Submission of papers to NeurIPS
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Check Section 6.4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Check ourAppendix D.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Check
Proposition 1 and the proof in Appendix B.
(b) Did you include complete proofs of all theoretical results?
[Yes] Check Proposition
and the proof in Appendix B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Check a detailed
dataset introduction in Appendix A.1 and the instructions in Appendix A.2.(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Check the experiment settings in Section 6 and Appendix A.2.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We use 5 random seeds for all experiments.
Check
the experiment settings in Section 6, our experiment results and Table C.1 and C.2 inAppendix.(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Check Appendix A.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] There is not formal
citation, so we add the link to the data source.
Check our dataset introduction in
Appendix A.1.
(b) Did you mention the license of the assets?
[N/A]
There is no such licence that we can
cite for this dataset.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] We have corporated on many projects.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] Check our dataset introduction in Appendix
A.1.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]