Uncertainty-Aware Reinforcement Learning for Risk-Sensitive Player Evaluation in Sports Game
Abstract
A major task of sports analytics is player evaluation.
Previous methods commonly measured the impact of players’ actions on desirable outcomes (e.g., goals or winning) without considering the risk induced by stochastic game dynamics.
In this paper, we design an uncertainty-aware Reinforcement Learning (RL) framework to learn a risk-sensitive player evaluation metric from stochastic game dynamics.
To embed the risk of a player’s movements into the distribution of action-values, we model their 1) aleatoric uncertainty , which represents the intrinsic stochasticity in a sports game, and 2) epistemic uncertainty , which is due to a model’s insufﬁcient knowledge regarding Out-of-Distribution (OoD) samples.
We demonstrate how a distributional Bellman operator and a feature-space density model can capture these uncertainties.
Based on such uncertainty estimation, we propose a Risk-sensitive
Game Impact Metric (RiGIM) that measures players’ performance over a season by conditioning on a speciﬁc conﬁdence level.
Empirical evaluation, based on over 9M play-by-play ice hockey and soccer events, shows that RiGIM correlates highly with standard success measures and has a consistent risk sensitivity.
Introduction
The advancement of player tracking and object detection systems enables data-driven analytics for professional sports players.
A common approach to evaluating the contribution of players is to quantify their action impacts.
Previous performance metrics [1, 2, 3, 4] computed the expected impact of an action on scoring or winning a game.
However, actions with signiﬁcantly different distributions of impact can have the same expectations.
As a result, the expectation-based metrics cannot differentiate the risk-seeking actions from the risk-averse ones.
How to distinguish these actions and assign proper credits to the players remains a fundamental challenge in sports analytics.
An important step toward a risk-sensitive evaluation metric is to model the distributions of action values.
To achieve this goal, distributional Reinforcement Learning (RL) [5] predicts the supporting quantiles of action-value distributions.
Previous distributional RL methods [6, 7, 8, 9, 10] mainly studied the virtual environments with deterministic transitions (e.g., Atari [11] or Mujoco [12]), whereas sports games are real environments with stochastic game dynamics and complex context features.
Moreover, player evaluation, as a data-driven task, requires learning from a ﬁxed dataset without exploration.
The model must be able to handle the distribution shift experienced at test time.
To mitigate the impact of distribution shift, Ofﬂine RL algorithms [13] typically strive for conservative policies that discourage visits to Out-of-Distribution (OoD) states by penalizing corresponding values.
However, this approach cannot be scaled to player evaluation since 1) our goal is not control (i.e., improving players’ policy), but to use RL as an analytical tool to evaluate observed actions in professional games, and 2) penalizing actions in OoD states distorts the evaluation.
In this paper, we design an uncertainty-aware RL framework for risk-sensitive player evaluation.
Figure 1 shows a real-world example introducing our key motivations.
Instead of directly inﬂuencing players’ actions like other RL algorithms, we perform a post-hoc calibration of the learned action values.
The main idea of our framework is to model important types of uncertainty in sports games: 1) Aleatoric uncertainty captures the intrinsic stochasticity of game dynamics caused by stochastic rewards, transition dynamics, and policies.
We show that this stochasticity can be captured by a distributional Bellman operator and propagated between action-value distributions by implementing
Temporal-Difference (TD) learning with distributional RL.
2) Epistemic uncertainty is due to the ﬁnite training samples and OoD state-action pairs during testing.
Online RL algorithms can overcome this uncertainty given sufﬁcient exploration in the environment [8].
However, when we have only a demonstration dataset with limited samples, the inﬂuence of epistemic uncertainty cannot be ignored.
Striving for simplicity and scalability, we model the epistemic uncertainty with a Feature-Space Conditional Normalizing Flow (FS-CNF).
Based on the uncertainty estimations, we develop a Risk-sensitive Game Impact Metric (RiGIM) for player evaluation.
RiGIM ﬁlters the predictions for OoD samples and computes the impact of players’ actions by conditioning on a conﬁdence level.
Empirical evaluation shows that RiGIM is highly correlated with standard measures when compared to other baselines.
We measure the accuracy of action-value predictions by matching them empirically with game results and evaluate the risk-sensitivity of RiGIM by its correlations with standard measures at different conﬁdence levels.
Contributions.
1) We design an uncertainty-aware RL framework that enables post-hoc calibrations on action values according to their aleatoric and epistemic uncertainties.
We demonstrate how the distributional Bellman operator captures the aleatoric uncertainty with action-value distributions from both a theoretical and an empirical perspective.
3) Striving for scalability, we design a feature-space density estimator that estimates the epistemic uncertainty with a minimum overhead.
To the best of our knowledge, RiGIM is the ﬁrst risk-sensitive metric that incorporates the inherent risk in environment dynamics into player evaluation.
Although this work mainly focus professional sports games, our method is general and can be scaled to other stochastic environments or domains.
Related Works
In this section, we introduce previous works that are most related to our approach.
Uncertainty Estimation for RL.
Uncertainty estimates have been widely used in RL for guiding exploration and stabilizing policies.
To achieve this goal, an effective approach is to measure the uncertainty of future returns : [14] designed an uncertainty Bellman equation that estimates the variance of the Q-value posterior distributions.
Distributional RL methods [6, 7, 8, 9, 10, 15, 16, 17] directly model the distribution of future returns by computing corresponding quantities.
Bootstrapped
DQN methods [18, 19, 20, 21] learn ensembles of action-value Q functions to capture uncertainty.
Some following works [22, 23] extend the Q-ensemble methods to ofﬂine RL settings by learning from a ﬁxed dataset.
Instead of focusing on the returns’ uncertainty, an alternative approach is to measure the uncertainty of model dynamics : [24, 25] proposed model-based RL approaches that predict the uncertainty of dynamics models and penalize the actions leading to uncertain returns.
Instead of separately capturing the epistemic and aleatoric uncertainties, these methods often quantify the overall uncertainty with a uniﬁed measure (e.g., variance or entropy).
Another line of approaches that heavily relies on uncertainty estimates is Risk-Sensitive RL (RSRL)
The RSRL agents avoid states with high costs by estimating a risk measure (e.g., variance, Value at Risk (VaR), or Conditional VaR [27]).
These algorithms [28, 29] often learn a controlling policy based on a known MDP instead of an ofﬂine dataset (with unknown dynamics).
Player Evaluation.
The most common approach to player evaluation is to quantify the impact of their actions on game results [30].
Previous works measured action impacts by predicting 1) whether a goal will be scored within a ﬁxed look-ahead horizon [3], 2) the change of winning chances [31], and 3) the expected number of goals within a possession [32].
Some recent works also trained action-value Q-functions by dynamic programming [1], deep Sarsa [2, 33] and Inverse RL [4].
These methods compute an expected action value without modeling their potential risk, and they commonly assume the training and testing datasets are identically distributed.
Uncertainty-Aware RL framework for Player Evaluation
We represent the dynamics in sports games with a Markov Game model and introduce the motivation of estimating the aleatoric uncertainty and the epistemic uncertainty.
3.1 Finite-Horizon Markov Game Model
Player evaluation metrics commonly evaluate players by how much their actions inﬂuence the opportunity of scoring the next goal [2, 3, 34].
Following this setting, we divide a sports game into goal-scoring episodes , so that each episode 1) begins immediately after a goal (or at the beginning of the game), and 2) terminates when the next goal is scored (or the end of the game is reached).
From an algorithmic perspective, this setting allows us to bound the support of future-goals distribution (Section 4.1) into [0 , 1] , which leads to faster model convergence and more accurate evaluations.
For a scoring episode of length T H , we model its dynamics with a ﬁnite-horizon Markov game model [35] : G = ( S , A , P T , R , O , T H , γ ) .
At a time step t ∈ [0 , T H ] , an agent k performs an action a k,t ∈
A k at a game state s t ∈ S after receiving an observation o t ∈
O .
This process generates the next state s t +1 ∼ P T ( ·| s t , a t ) and a reward r k,t
= R k ( s t , a k,t ) .
γ is a discount factor.
In this paper, we consider two agents k ∈ { Home, Away } representing the home and away teams.
The observed data D =
[( o 1 , a k, 1 , r 1 ) , ( o 2 , a k, 2 , r 2 ) , . . .
, ( o t , a k,t , r t ) , . . . ]
records the action a k,t performed by the team k who possesses the puck.
To alleviate the partial observability, a game state includes the game history: s t :=
( o t , a t − 1 , o t − 1 , . . .
, o 0 ) .
The reward r t is a 1-of-2 indicator vector that speciﬁes which team ( Home, Away ) scores.
We assign zeros to r t until a team scores at the end of an episode.
3.2 Uncertainty-Aware RL for Player Evaluation
Learning from Ofﬂine Data.
As a data-driven behavior analytic tool, the player evaluation model assigns values to players’ actions by learning from an ofﬂine dataset.
Under this setting, previous works [1, 2, 33, 3] commonly assumed the training and testing datasets are sampled from the same underlying distribution.
However, in practice, since the behaviors of players may change when some team members (especially core players or the coach) are traded or hired during a season, there is no guarantee that the visitation frequency of state-action pairs is consistent in different games.
It is natural to assume a distributional shift between the games in the training and testing dataset: while the value function is trained under one distribution, it will be evaluated on a different distribution.
Calibration with Uncertainty.
To alleviate the inﬂuence of distribution shift, ofﬂine RL algorithms [13] commonly discourage the visit to OoD state-action pairs by lowering their values [36] or penalising their rewards [24] or constraining the updated policy [37, 22].
However, to evaluate players’ performance, RL is employed as a policy evaluation tool instead of controlling players.
The trajectories in testing games correspond to observed players’ actions that cannot be changed or inﬂuenced by penalties.
Instead, we perform a post-hoc calibration of the predicted action values by modeling their epistemic uncertainty , which is due to a lack of knowledge about OoD samples, and thus the resulting model is uncertain about the returns (Section 4.2).
Since our goal is to develop a risk-sensitive player evaluation metric, we estimate distributions of action values to model their aleatoric uncertainty , which is due to the intrinsic stochasticity in the game dynamics (Section 4.1).
In practice, the quantiﬁcation of aleatoric uncertainty can be inﬂuenced by the epistemic uncertainty of input samples, so we ﬁlter the OoD samples by utilizing our density estimator (Section 5.1) .
Modelling the Uncertainty of Action Values
The aforementioned uncertainty-aware RL framework requires estimating the aleatoric and epistemic uncertainty for a risk-sensitive player evaluation.
In this section, we introduce our distributional-RL approach for modelling aleatoric uncertainty and a feature-space density estimator for measuring epistemic uncertainty (Figure 2 shows the model architecture).
where one team attacks and the other defends.
We add Spectral Normalization to ResNet outputs.
4.1 Distributional RL for Capturing Aleatoric Uncertainty
Distributional RL learns the distribution of the random variable Z k ( s t , a t ) that corresponds to the number of future goals when a player of team k performs action a t in state s t .
In other words, we can think of Z k ( s t , a t ) as a random variable with outcomes corresponding to the sum of discounted rewards (cid:80) T H ι = t γ ι R k,ι ( S ι , A ι ) , where S ι = s ι , A ι = a ι , S ι +1 is distributed according to P T ( ·| S ι , A ι ) and A ι is distributed according to π ( ·| S ι ) .
Following the Quantile-Regression (QR)-DQN method [6], we represent the distribution of Z by a uniform mixture of N supporting quantiles by ˆ Z k ( s t , a t ) =
N (cid:80) N i =1 δ θ k,i ( s t ,a t ) , where θ k,i estimates the quantile at the quantile level (or quantile index)
ˆ τ i =
τ i −
+ τ i ( 1 ≤ i ≤ N , and τ i
=
i/N ) and δ θ k,i denotes a Dirac distribution at θ k,i .
The model outputs [ θ k, 1 , . . .
, θ k,N ] are monotonically increasing quantile values computed with the spline DQN (SPL-DQN) by following [17].
Distributional Bellman Operator.
When the player of a team k performs an action a t at a state s t , the agent receives a reward R k ( s t , a t ) and moves to a future state s t +1 ∼ P T ( S t +1 | s t , a t ) {+where the agent’s next action a t +
+}
∼ π
( A t +1 | S t +1 ) .
This stochastic process can be captured by a distributional
Bellman operator T π [6]:
[Equation]
where X ∆ :=
Y indicates that random variables X
and Y follow the same distribution.
Based on the distributional Bellman operator, we estimate the supporting quantiles of Z by minimizing the quantile Huber loss (with threshold η ):
[Equation]
[Equation]
Illustration of Temporal Projection .
Figure 3 illustrates the mean ± standard deviation of the actionvalues sampled from the predicted distributions ˆ Z ( s, a ) , where s and a follow the players’ movements in a match between the Flyers (Home team) and the Maple Leafs (Away team) on March 15, 2019.
The ﬁgure plots values of the two output nodes.
We highlight critical events and match contexts to show the context-sensitivity of our predictions.
We show that the predicted distribution of action values can measure the aleatoric uncertainty.
Proposition 1.
Assume the Bellman consistency holds by ˆ Z ∆ :=
R + γ P π ˆ Z where ˆ Z , R are vector-valued random variables and P π is the transition matrix of the stationary policy π , so
P π ( s,a ) , ( s (cid:48) ,a (cid:48) )
=
P ( s (cid:48) | a, s ) π
( a (cid:48) | s (cid:48) ) , the uncertainty of action-value distributions ˆ Z under an entropy measure H ( · ) can be given by:
[Equation]
where d π = (1 − γ )
( I − γ P π ) − 1 ∈ [0 , 1] | S ||
A |×| S
||
A | is the induced matrix for distributions over state-action tuples by following policy π and transition P T .
The proof is in Appendix B.1.
Proposition 1 disentangles the entropy of Z into 1) the entropy of reward variables that quantiﬁes the uncertainty of current rewards, 2) the uncertainty induced by the discount factor, which determines how much the current uncertainty estimation should be inﬂuenced by the stochasticity of future rewards or transitions (i.e., a small γ reduces this inﬂuence), and 3) a log-absolute determinant of the induced distribution matrix, which measures the amount of stretch or change that the transition function P T and the policy π apply to the initial state-action distribution.
Proposition 1 demonstrates that the key components for representing the aleatoric uncertainty can be captured by Z when the Bellman consistency is reached by learning, which suggests the action-value distribution learned by distributional RL is an ideal estimator for aleatoric uncertainty.
However, in practice, the estimation of Z cannot be well generalized to all samples because of insufﬁcient exploration or limited training data, so we need to estimate their epistemic uncertainty.
4.2 Density Estimator for Capturing Epistemic Uncertainty
By deﬁnition, epistemic uncertainty stems from limited training data and is inherent to the model ﬁtting these data.
A common measure of epistemic uncertainty is I ( θ ; y | x, D ) [38, 39, 40]: the amount of information gained when the model θ observes the true label y of an input x .
To estimate this uncertainty measure, previous works [41, 42, 43] utilized deep ensemble models and treated each ensemble as a sample from the posterior p ( θ |D ) .
However, adding additional deep ensemble layers to a distributional RL model (i.e., as a uniﬁed estimator for the joint distribution p ( Z 1 ,...,K , Θ |D ) ) signiﬁcantly increases the model complexity.
Striving for model simplicity and scalability for large datasets, we build a feature space density estimator [40] to detect OoD samples.
In this work, we design a Feature Space Conditional Normalizing Flow (FS-CNF) to estimate sample density in the training distribution with a minimum overhead.
The main components are:
Feature Extractor.
FS-CNF shares the same feature extracting layers with the distributional RL models.
Note that a common reason why traditional feature extractors might fail to capture epistemic uncertainty is feature collapse [44], which maps OoD samples to iD regions in feature space.
To prevent this phenomenon, an ideal feature extractor f θ must be subjected to a bi-Lipschitz constraint:
[Equation]
where 1) the lower Lipschitz bound ensures sensitivity to distances in the input space (i.e., sensitive to OoD samples) and 2) the upper Lipschitz bound ensures smoothness in the feature space (i.e., prevents overﬁtting to the input variations).
To ensure this bi-Lipschitz condition in practice, we follow [44] and use residual networks with spectral normalisation as our feature extractor.
Density Estimator.
Based on the extracted features, FS-CNF utilizes the Masked Auto-regressive
Flow (MAF) [45] design that estimates the density of input variables in the training data distribution with an auto-regressive constraint.
Additionally, to make FS-CNF more sensitive to the abnormal predictions in OoD samples, the density model is conditioned on the expected returns of actionvalue distributions, so p ( x | z E ) =
(cid:80) i p ( x i | x 1: i − 1 , z E ) where z E :=
{ E [ Z k ( s, a )] } Kk =1 and x :=
( o, a ) (we use o instead of s since a large input dimension inﬂuences the accuracy of estimation).
We implement p ( x i | x 1: i − 1 , z E ) =
N ( x i | µ i , ( exp ( α i )) 2 ) where µ i =
ψ µ i ( x 1: i − 1 , z E ) and α i =
ψ
α i ( x 1: i − 1 , z E ) .
The neural function ψ is implemented by stacking multiple MADE layers [46].
Player Evaluation
In this section, we introduce our player evaluation metric and risk-sensitive rankings.
5.1 Risk-sensitive Impact Metric
Measuring Risk.
We measure the risk of a player’s movement with the aleatoric uncertainty since modeling the intrinsic stochasticity of the game dynamics is consistent with the goal of sports analytics.
However, as Figure 4 shows, when we quantify the aleatoric uncertainty with predictions from distributional RL, the performance is inﬂuenced by the density of input samples (i.e., OoD samples have a lower accuracy).
An effective approach to verify whether ˆ Z k ( s, a ) accurately captures the true aleatoric uncertainty is to check the epistemic uncertainty for each input data ( s, a ) [40]: 1) a high input density ( p ( ·| z E ) ≥ (cid:15) ) indicates low epistemic uncertainty, (i.e., ( s, a ) is inD) and we can trust the aleatoric uncertainty estimated by distributional
RL. 2) a low input density ( p ( ·| z E ) < (cid:15) ) indicates high epistemic uncertainty (i.e., the input is OoD, and we do not use the model prediction).
In practice, (cid:15) is determined based on the validation dataset.
Risk-Sensitive Action Impact.
The impact φ ( s, a ) measures how much an action a changes the future return of a player’s team.
In terms of the value function, this is the change in action value due to a player’s movement.
Previous works [1, 2, 3] computed the impact with the expected next-goal return Q ( s t , a t ) =
E [ Z ( s t , a t )] .
Q ( · ) does not take into account the inherent variability of the returns and thus cannot estimate the risk of an action.
To understand how players respond to risk, we propose a Risk-sensitive Game Impact Metric (RiGIM ) based on ˆ Z k ( s, a ) and p ( ·| z E ) :
[Equation]
where c ∈ [0 , 1] is the conﬁdence level, Z c denotes the (1 − c ) th quantile in Z ( · ) and n ( s, a, l ) denotes the number of times that a player l performs action a at a state s in the testing dataset D (cid:48) .
We omit the term r t since r t = 0 except at scoring step T , and s T is the terminal state of a scoring episode.
φ ( · ) can either be 1) risk-averse (with a large c ), with better sensitivity to bad outcomes or 2) risk-seeking (with a small c ), with better sensitivity to positive outcomes.
5.2 Case Study: Player Ranking in Testing Games
We rank players according to their RiGIM scores in the NHL testing games (see the experiment setting in Section 6) at different conﬁdence levels.
Tables 1 and 2 illustrates how a domain expert could use our method to gain insight into which types of players exhibit different risk-taking behavior.
Table 1 shows a risk-seeking ranking (conﬁdence c = 0 . 2 ), which favors offensive players (e.g.,
Centres (C)) with strong scoring ability.
Aleksander Barkov, who scores the most points in these games, is captured by this ranking.
When we set conﬁdence c to 0.8, Table 2 shows a risk-averse ranking which highlights players in defensive positions (e.g., Defensive (D)).
John Klingberg, the defenseman with the most assists, is listed in the top-10 players.
We believe the differences between
Tables 1 and 2 can be explained by the fact that RiGIM is correlated with the types of actions a player performs.
Intuitively, in ice-hockey, some actions are more risk-seeking (shot) in terms of scoring while other actions are risk-averse (carry or pass).
In general, players in the backcourt (e.g., defensemen) are more likely to perform risk-averse actions that have smaller variance on the scoring chances.
[Table] [Table]
Empirical Evaluation
Dataset.
Our experiments utilize both a ice-hockey and a soccer dataset from the National Hockey
League (NHL) and major European soccer leagues, which contain 9,213,371 events, coveringteams, 4,172 games, and 6,513 players.
These datasets consist of events around the ball.
Each event records the identity and action of the player possessing the ball, with time stamps and features of the game context (see all the game features in Appendix A.1).
To the best of our knowledge, this is the most extensive study for player evaluation .
Note that we do not utilize virtual environments like Atari [11] or Mujoco [12] because 1) the dynamics in these environments are deterministic without uncertainty to be modeled and 2) this paper mainly studies sports games that have stochastic dynamics [30], which are valid test-beds for our method.
Experiment Settings.
We divide the dataset into a training set (80%), a validation set (10%), and a testing set (10%) according to game dates, so that games in the testing set happened after the games in the training and validation set.
To predict the action values in the testing games, the metric must remain robust to OoD data points.
We report the results averaged over 5 independent runs.
Comparison Methods.
We employ an ablation design that iteratively removes parts from RiGIM .
GIM removes the uncertainty estimator by directly using a Deep Recurrent Q-Network for estimating action values [2].
T0-GIM removes the recurrent model and uses a Deep Q-Network (DQN) for the value function.
We then replace the RL framework with a supervised learning framework for estimating action values by following VAEP [3].
Instead of using function approximators, Scoring
Impact (SI) [1] implements the tabular-based value iteration algorithm for computing action values from discretized spatial and temporal features.
Expected-Goal (EG) metric directly uses the expected action values instead of impact values for measuring player performance.
The last metric Plus-Minus ( + / − ) is based on game statistics and measures the goal-gain with and without the player on court.
We summarize these metrics in Table 3.
[Table]
To study how well SP-CNF boosts model performance, we compare 1) a Gaussian Discriminant
Analysis (GDA)-RiGIM metric that replaces
SP-CNF with GDA [40], and 2) a Naive (Na)-
RiGIM metric that removes the epistemic uncertainty estimator and uses all the predicted distributions to compute players’ impact (see
Appendix A.2 and A.3 for more details.).
6.1 Player Evaluation Performance: Correlations with Standard Measures
We follow [2, 4] and compute the correlations between player ranking metrics and standard measures on the testing games in a game season, because 1) the player (or agent) evaluation task has no ground-truth labels or rewards to maximize, 2) the correlation to all measures (including penalty measures) can measure whether the metrics can form a comprehensive evaluation to a player’s overall performance.
We study 11 success measures for ice hockey and 8 success measures for soccer.
To make the results more comprehensive, we also add 5 penalty measures .
The studied measures are popular measures from the NHL and soccer statistics websites 1 .
Following the popular risk-measures like Conditional Value at Risk (CVaR), we treat c as a hyper-parameter.
Since the comparison methods are expectation-based metrics, we study the options of 1) setting the conﬁdence level ( c ) of
RiGIM to 0.5 for a fair comparison 2) empirically determining c with the validation set ( c ∗ = 0 .and 0 .
49 for the ice hockey and soccer datasets).
The risk-sensitive results are shown in Section 6.2.
Tables 4 and 5 show the average correlations in 5 independent runs on the testing dataset (see Tables
C.1 and C.2 in Appendix for the complete mean ± standard deviation results).
RiGIM achieves the highest correlations with 14 out of 19 success measures and the smallest correlations with 3 out of 5 penalty measures in ice hockey and soccer games.
This observation shows that RiGIM is a comprehensive metric that can detect both the positive and the negative inﬂuence of a player.
If we remove SP-CNF or replace it with other uncertainty estimators, most correlations become weaker except for the correlations with the SHP and SHG measures.
This is because SHP and SHG rarely happen in a season (scoring with fewer players on ice is difﬁcult).
SP-CNF detects this phenomenon and assigns small densities to these rare events.
RiGIM ﬁlters the event with a small density (see Equation 5), which might cause the loss of information and make its correlation with SHP and SHG less signiﬁcant.
SI correlates well with goal measures (Goals and Game Winning Goals) but has relatively poor correlations with other measures.
This is because assigning an adequate value for all actions, including those with only intermediate effects on goal scoring, requires credit propagation over longer sequences, where neural nets are better at credit propagation than discretizing and using a tabular representation.
For other risk-neutral methods, their performance is generally less satisfying when compared with risk-aware methods, especially for the + / − metric, which shows the importance of capturing risk and modeling the context features.
[Table]
[Table]
6.2 Sensitivity to Risk: Correlations Conditioning on Different Conﬁdence Levels
We measure whether RiGIM is sensitive to the risk by its correlations with the standard measures, where RiGIM is conditioned on a speciﬁc conﬁdence level c (from 0 to 1), for example, RiGIM ( c ) , which indicates with probability c that the players’ impact should be at least RiGIM ( c ) .
Figures 5 and 6 show the correlations at different conﬁdence levels for ice-hockey and soccer games.
RiGIM is sensitive to risk, in the sense that it has different correlations with standard measures at these conﬁdence levels, whereas GIM, as a risk-neutral metric, is unaware of the risk, and thus its correlations remain unchanged.
Compared to other baselines, our RiGIM generally maintains higher correlations with success measures and lower correlations with penalty measures.
The exceptions are the correlations with the SHP, SHG, and OTG.
For the same reason as discussed above, SP-CNF may ﬁlter them during testing.
We ﬁnd when c becomes smaller, RiGIM ( c ) becomes risk-seeking, and thus achieves a higher correlation with success measures.
However, the correlations drop when c approaches 0.
This is because ˆ Z 0 k denotes the estimates at the largest quantile level (see Equation 5), which corresponds to the most optimistic estimation action value (i.e., label scoring for all the shots).
The overly risk-seeking estimation can induce a mismatch between estimated values and game facts, and thus cannot reﬂect the real contributions of players.
6.3 The Prediction Accuracy: Match Future Scoring Frequencies With Action-Values
We study how well the predicted action-value distributions match the real next-goal scoring frequencies under discrete game contexts.
These game contexts are constructed by dividing the continuous state space into discrete bins.
To calculate the empirical scoring frequency associated with each bin, we assign an observed state s to a bin B according to the values of three discrete context features in the current observation: Manpower Differential, Goal Differential, and Period.
The empirical and estimated scoring probabilities for a bin (with size | B | ) are deﬁned as follows:
a) Empirical Scoring Chances : for each s ∈ B , we set g k ( s ) =
if the observed scoring-episode containing state s ends with a goal by team k .
Then G ∗ k ( B ) =
| B | (cid:80) s ∈ B g k ( s ) .
[Equation]
p [Equation] [Equation]
[Table]
Table 6 shows the average absolute difference between ˆ G and G ∗ based on 5 independent runs.
We implement a Wilcoxon signed rank test [47] to study whether the predictions from baseline methods are different from that of RiGIM for all samples.
Our baselines are the learning-based action-values metrics for the context-aware evaluation.
RiGIM achieves a minimum distance in 12 out of 17 bins.
This is because 1) FS-CNF outperforms GDA by computing a more accurate uncertainty estimator for ﬁltering OoD states and 2) the distribution estimates contain richer information than expectation estimates, allowing ˜ Z ( · ) to better match the scoring frequencies than Q ( · ) .
This observation is consistent with the ﬁndings in [6].
6.4 Limitations
A Study for Sports Games.
This work uses stochastic sports games as the testbeds for uncertainty estimation, but we argue the same methods can be easily migrated to other applications in stochastic environments such as autonomous driving or healthcare.
Our approach of measuring aleatoric and epistemic risks can be adapted to ofﬂine RL for learning conservative policies.
Evaluation Instead of Control.
Our method focuses on player evaluation instead of control.
We believe both tasks are challenging with respect to different aspects.
Evaluation requires the action values to accurately reﬂects an agent’s real contribution to game-winning.
An ideal evaluation metric can provide in-game predictions of game outcomes, which is important for the sports industry.
Conclusion and Future Work
In this paper, we designed an RL framework for quantifying the aleatoric uncertainty and the epistemic uncertainty from stochastic sports datasets.
This framework enabled distributional RL and an FS-CNF model to estimate both uncertainties, with which we proposed a risk-sensitive evaluation metric
RiGIM .
Empirical results show that RiGIM correlates well with success measures and the correlation is sensitive to different risks.
A direction of future work is to extend our model to other domains.
Acknowledgments and Disclosure of Funding
We acknowledge the funding from the Canada CIFAR AI Chairs program, and the support of the
Natural Sciences and Engineering Research Council of Canada (NSERC).
Resources used in this work were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute https://vectorinstitute.ai/partners/ .