B EYOND W INNING AND L OSING : M ODELING H UMAN M OTIVATIONS AND B EHAVIORS WITH V ECTORVALUED I NVERSE R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In recent years, reinforcement learning methods have been applied to model gameplay with great success, achieving super-human performance in various environments, such as Atari, Go and Poker.
However, those studies mostly focus on winning the game and have largely ignored the rich and complex human motivations, which are essential for understanding the agents’ diverse behavior.
In this paper, we present a multi-motivation behavior modeling which investigates the multifaceted human motivations and models the underlying value structure of the agents.
Our approach extends inverse RL to the vectored-valued setting which imposes a much weaker assumption than previous studies.
The vectorized rewards incorporate Pareto optimality, which is a powerful tool to explain a wide range of behavior by its optimality.
For practical assessment, our algorithm is tested on the World of Warcraft Avatar History dataset spanning three years of the gameplay.
Our experiments demonstrate the improvement over the scalarization-based methods on real-world problem settings.
I NTRODUCTION
Reinforcement learning methods have been intensively applied to game environments with great success.
Several landmark research works have been conducted on the games such as such as Atari,
Go, and Poker where RL algorithms achieve super-human performance (Mnih et al. (2015); Silver et al.
; Heinrich & Silver (2016)).
However, the majority of the studies focus on winning the game or achieving high scores, thus have largely ignored the rich and complex human motivations.
The understanding of the motivation and the corresponding reward mechanism, which is essential for computing intelligence, has hence long been open.
In fact, extending the single, scalarized reward function to model multifaceted motivations is non-trivial.
The vector-valued reward does not describe the task as clear and succinct as the scalar reward does, and consequently, the optimality of the policy can be ill-deﬁned.
We argue that the extension instead imposes weaker assumption on the optimality of the policies, hence enable the model to analyze the diverse behavior from the perspective of the reward mechanism.
It is shown to be equivalent to allow arbitrary scalarization function to process the multi-dimensional reward signals with the minimal assumption that the function is monotonic, as desired in analyzing the agents’ behavior in the real world.
Previous studies have been discussing the setting where multiple reward signals are available.
For example, features are commonly used in IRL since its inception (Ng et al.
; Abbeel & Ng (2004)), where features are treated as oracles and the reward is a function of those features.
It was assumed that the combination of features is linear and was later extended to the nonlinear cases such as neural networks e.g. Wulfmeier et al.
Finn et al. (2016)) and Gaussian processes e.g. Levine et al.
In general, the scalarization-based methods do not incur violation of model assumptions if there exists a scalarized reward function (whether observed or not).
However, as the environment is getting more complex and the motivations of the agents become diverse, the uniqueness of the scalar reward does not hold.
In this context, we design the algorithm with weaker assumptions as desired.
In this paper, we propose a novel method called multi-motivation behavior modeling (MMBM) which takes the multifaceted, complex, and diverse motivations into consideration.
Instead of ﬁnding
[Table]
the scalarization in the reward space, we regard any policy that is not dominated by an alternative feasible policy as an optimal policy, as shown in Fig.
We build our analysis upon the theorem that the policy is not dominated if and only if it is optimal under a linear scalarization to connect the set of optimal policy with the simplex of the coefﬁcient vector.
In this way, we characterize the range of the value function under optimal policies as a convex hull.
We further estimate the range via value learning on the extreme points and deﬁne the suboptimality of a policy as the distance between its corresponding value function and the range.
As a result, we design an efﬁcient algorithm with a linear program to minimize the distance and ﬁnd the reward mechanism.
A signiﬁcant advantage of MMBM is that it utilizes only off-policy learning: Both the range of the value function and the linear program depend on only the trajectories and does not require an access to the state dynamic transition.
In this way, the algorithm is applicable to historical datasets such as data from online games.
MMBM is tested on the World of Warcraft Avatar History (WoWAH) dataset Lee et al.
(2011), which records the movement of players over a three-year period.
We use the motivation theory of gameplay (Yee (2006), see Table 1) as the oracle of the vector-valued reward, and compare the analysis of the motivations with previous knowledge-based studies on WoW (Ducheneaut et al.
; Nardi & Harris (2006)).
We further show the signiﬁcant improvement on the inverse learning error over existing studies.
P RELIMINARIES 2. V ECTOR - VALUED M ARKOV D ECISION P ROCESS
We adopt the vector-valued Markov decision process (MDP) introduced by Wakuta (1995).
We follow the convention that for two vectors r , r (cid:48) ∈ R d , r > r (cid:48)
if and only if e T i r > e T i r for all i ∈
[ d ] , where e Ti has its i -th element equal to 1 and 0 otherwise.
We deﬁne r < r (cid:48) , r ≥ r (cid:48) , and r ≮ r (cid:48) similarly.
The vector-valued inﬁnite-horizon MDP is characterized by the ﬁnite state space S , the ﬁnite action space A ,
the transit probability P ( s (cid:48) | s , a ) , the reward signal r = r ( s , a ) ∈ R d , and the discount factor γ .
At each step of the process, the agent chooses a feasible action a ∈ A and the system’s state s transits to s (cid:48) with probability P ( s (cid:48) | s , a ) , while the agent receives an immediate reward of r ( s , a , s (cid:48) ) .
To describe the problem, we ﬁrst deﬁne the value function v π ( s ) ∈ R d with respect to a ﬁxed policy π by
[Equation]
where π chooses actions at each state, either deterministically or stochastically, also without ambiguity we write r t =
E s (cid:48) t
[ r ( s t , a t , s (cid:48) t )] and let s 0 be the initial state.
We will omit the distribution terms s (cid:48) ∼ P , s 0 ∼ ρ 0 ( s ) in the rest part of the discussion.
The objective of the agent is to ﬁnd a policy π such that
[Equation]
holds for every feasible policy π (cid:48) .
Note that the objective of scalar-valued MDP is the special case where r ( · ) ∈ R and d = 1.
To characterize the optimal policies, we ﬁrst denote Π the set of policies that satisfy the condition equation 2.
Also deﬁne V ( s )
=
{ v π ( s ) , π ∈ Π } to be the set of value vectors that are only smaller or equal than ( ≤ ) themselves in the range of the value function.
V ( s ) is also denoted as the Pareto frontier of the range of the value function Van Moffaert & Nowé (2014).
We describe the properties of Π ’s elements.
Similar to the recent studies in multi-object MDPs (Vamplew et al.
(2008); Van Moffaert & Nowé (2014); Jaderberg et al.
; Mossalam et al.
(2016)), we deﬁne the scalarization of the reward signal as
[Equation]
where φ ∈ Φ ⊂ R d is the weight vector of the scalarization and Φ = { φ ∈ R d , (cid:107) φ (cid:107) 1 = 1, φ ≥ 0 } is the simplex.
We then deﬁne the set Π φ of the optimal policies under the scalarized reward signal Π φ = argmax π E
[ ∑ ∞ t =
γ t φ T r t | s 0 , π ] .
The following theorems shows the connection between the optimality of policies under the vectored reward and the optimality of policies under the scalarized reward.
Theorem 1.
For any policy π , π ∈ Π
if and only if there exists a vector φ ∈
Φ such that π ∈ Π φ .
Theorem 2.
There exists ﬁnite number of policies π 1 ,..., π m , such that the range of the value function v π ( s ) with respect to π is the convex hull with v π 1 ( s ) ,..., v π m ( s ) being all its extreme points.
Theorem 1 maps the optimal policies to the reward weight space Φ which is a standard simplex.
Theorem 2 then maps the corresponding values to a convex hull based on which the suboptimality measure can be established.
Both theorems are important to us and our analysis is build upon them.
We refer the readers to Wakuta (1995) for both the proofs.
2. I NVERSE R EINFORCEMENT L EARNING
Inverse reinforcement learning (IRL) reverses the input and output pairs of RL algorithms, computing the rewards function according to the policies or trajectories of the agents.
The topic has been intensively studied and developed since its inception (Ng et al.
(2000)), including max-margin based methods (Abbeel & Ng (2004);
Ratliff et al.
(2006); Syed & Schapire (2008)
; Neu & Szepesvári (2012)), max-entropy IRL (Ziebart et al.
; Boularias et al.
(2011); Finn et al.
Bayesian
IRL (Ramachandran & Amir (2007); Lopes et al.
; Levine et al.
; Michini & How (2012)) and etc.
The initial investigation was conducted on the setting where the demonstration policy is available to the learning algorithm.
Under the assumption that both the state space and the action space are discrete, the algorithm solves a minimax optimization to ﬁnd the reward function.
More recent studies tend to focus on the case where only trajectories are available.
The algorithms vary widely but they can be roughly categorized into two classes.
One architecture is to update the policy function and the reward function coordinately.
In each policy update, the policy is optimized for one step to maximize the value of the current reward.
In each reward update, the reward function is optimized for one step to ensure that the expert demonstration is optimal under the reward function.
The other approaches learn the function approximation of the action-value function, and then test, for each state-action pair in the demonstration, if all the alternative feasible actions lead to values that are no greater than the expert action does (Hester et al.
We extend the IRL to the vector-valued case and argue that the extension makes relatively weak assumptions than the existing methods and is hence desired.
First, we note that in both categories the algorithms make the assumption on the optimality of the demonstration policy (Ng et al.
Abbeel & Ng (2004)).
Such optimality can be furthered formulated as the ≮ relationship between the value of the expert policy and the value of the alternative feasible policy.
Hence, when both sides of the value are extended to the vectored case, ≮ imposes weaker assumption than the ≥ of either the scalarized value functions, or the case that d = 1.
In fact, the element-wise ≥ relation of any dimension is sufﬁcient to achieve the ≮ relation.
Intuitively, any non-dominated policies are feasible under the assumption, which allows arbitrary scalarization function as long as they are monotonic.
To highlight the signiﬁcance of the difference in both the assumptions, we note that it is common that existing IRL approaches relax the assumption by assigning a penalty on the violation of the assumption.
Such relaxation will drive the actual algorithm from its theory and motivation, while it can be avoided by using vectored value.
2. V ALUE F UNCTION A PPROXIMATION
We discuss the approximation of both the state-value function and the action-state value function which is deﬁned as
[Equation]
and by deﬁnition we have v π ( s ) =
E π ( a | s )
[ Q π ( s , a )] .
Without further speciﬁcation, the discussion applies to r ∈ R d as is deﬁned previously.
The approximation is based on the Bellman equation, which describes the recursive relation
[Equation]
[Equation] We have then π ∈ Π if and only if that whenever a (cid:48) is feasible under state s t +
[Equation]
The inequation is not tractable in general, but we describe the previous studies that addresses the case of d = 1 below and present our method in the next section.
The approximate action-state value function can be learned by parametrize the Q ( · ) function and minimize the discrepancy between both sides of equation equation 6, known as Q-learning (Watkins & Dayan (1992);
Sutton et al.
Mnih et al.
; Dabney et al.
Speciﬁcally when d = 1 and ≮ degenerates to ≥ , the discrepancy can be quantiﬁed as the Bellman error
[Equation]
Combined with the fact that the action-state value function is parametrized (e.g. approximated by a neural network), the Bellman error can be minimized by running optimization program such as
SGD.
The Q-learning can be conducted off-policy as described.
The optimization in Q-learning is unstable in general and we borrow the techniques and tricks from
Mnih et al.
(2015) in the implementation.
We re-state the techniques for completeness of our presentation.
First, the order of expectation and maximization are swapped, resulting in the estimator 12 ( Q π ( s , a ) − ( r t + γ max a (cid:48)
Q π ( s (cid:48) , a (cid:48) ))) 2 which is biased but easier to compute.
It then replace the max a (cid:48)
Q π ( s (cid:48) , a (cid:48) ) term by max a (cid:48)
Q π ( s (cid:48) , a (cid:48) | θ − ) , where θ − is the parameter of the function approximation of a previous iteration.
The tweak aims to reduce the instability due to the correlation of both the approximation terms in equation 7.
Third, the reward is clipped and normalized to ensure that the value function is bounded.
Note that our algorithm is compatible with improvements over
Q-learning (Hasselt (2010); Schaul et al.
; Dabney et al.
and those methods can be employed to increase the quality of the estimation.
M ETHODS 3. E STIMATING R ANGE OF THE V ALUE F UNCTION
Theorem 3.
{ v π ( s ) | π ∈ Π } ⊆ conv {{ v π ( s ) | π ∈ Π e 1 } , ... , { v π ( s ) | π ∈ Π e d }} , where conv denotes the convex hull and e i is the vector with the i-th element equal to one and zero otherwise.
The equality holds when each of the set { v π ( s ) | π ∈ Π e i } has an unique element.
Proof.
Without loss of generality we assume that the reward r ≥ 0.
If the uniqueness holds, for any element in { v π ( s ) | π ∈ Π } , there is an explicit φ which satisﬁes (cid:107) φ (cid:107) 1 = 1, φ ≥ 0 such that the element is an optimal value under the reward φ T r .
Hence [Equation]
We proof the reverse by contradiction.
Suppose there exists an π ∈ Π such that v π ( s ) (cid:60) conv {{ v π ( s ) | π ∈ Π e 1 } , ... , { v π ( s ) | π ∈ Π e d }} .
By separation theorem there exists a φ such that φ T v π ( s ) >
φ T z for any z ∈ conv {{ v π ( s ) | π ∈ Π e 1 } , ... , { v π ( s ) | π ∈ Π e d }} .
Let φ −
i be φ except that the i -th element is replaced by zero.
If any element of φ is positive (assume it is the i -th without loss of generality), we pick from Π e i the element z with the largest φ T − i z .
The i -th element of v π ( s ) must be greater than that of z , which contradicts with the deﬁnition of Π e i .
If otherwise all elements in φ are negative, v π ( s ) is dominated by its projection on conv {{ v π ( s ) | π ∈ Π e 1 } , ... , { v π ( s ) | π ∈ Π e d }} ⊆ { v π ( s ) | π ∈ Π } , which is the value of a mixed policy.
It then contradicts with the optimality of π .
The theorem follows.
Theorem 3 provides a lower bound of the distance to the range of the value function.
The bound is also shown to be tight when the optimal policy under the scalar reward e Ti r is unique, and the uniqueness commonly happens in practice.
Armed with the theorem, estimating the range of the value function amounts to estimating those extreme points.
As we have discussed in the Bellman feasibility equation 6, the one-hot vector e i degenerates the setting to Q-learning with scalarized rewards and the value can be efﬁciently approximated.
We estimate d action-state value functions in our algorithm and denote them as Q 1 ( · ) , ... , Q d ( · ) for further use.
The exact function approximator and training process largely depends on the environment and we discuss those technical details in the experiments.
3. D ISTANCE M INIMIZATION
We compute the distance between the value of the alternative policies and the range of the value function, which measures the optimality of the demonstration.
We also compute the direction of projection used to measure the distance which describes the combination of the reward vector.
Let
Q =
( Q 1 , ... , Q 5 ) be the vectorized action-state value function and let y be the projection of direction.
Also denote T to be the set of state-action pairs ( s , a ) appeared in the trajectories.
We assume that the actions are conducted in a way that the incurred value is not dominated by an alternative feasible action a (cid:48) .
Such optimality is learned by maximizing the distance between the set of optimal values conv { Π e 1 , ... , Π e d } and the value of the alternative Q ( s , a (cid:48) ) .
The distance is measured by the Euclidean distance in the projected direction, but it could be extended to other measurements in general.
It is
[Equation]
[Equation]
It is worth note that as we do not impose any scalarization on the vectored reward, the model assumption is easier to be satisﬁed.
Consider that the diversity of actions origins from both the diverse reward function and the suboptimality in the actions (such as act randomly), we add a term c [Equation] y T Q [Equation]
s , a )
[Equation] max [Equation] s ) \
[Equation]
y
[Equation]
Q
[Equation] s , [Equation] )) − to make the algorithm more robust.
With c = 0, it degenerates to the algorithm that only considers the distances.
The minimization is reformulated into the following linear program
[Equation]
[Equation]
We describe the complete algorithm in 1, including the range estimation and the distance minimization.
In algorithm 1, line #4-12 are the Q-learning technique that computes the vertices of the superset of the desired value range.
Line #13-16 are the estimating of the range of the value function, which is used as the lower bound of the distance.
Line #17-18 ﬁnally solves the linear program and ﬁnds the projection vector y which describes the distance and optimality of the reward space.
With our two-step approach, MMBM takes the history of state-action pairs as input, which is usually logged during the gameplay.
It solves y , which is a quantitative description of agents’ motivation and the value structure.
Both the part #4-12 and the part #13-18 are generalized and are compatible with other approaches, such as recent advancement on Q-learning or other parametrization and measurement of the distance between Q ( s , a ) and Q ( s , a (cid:48) ) .
E XPERIMENTS
We highlight that MMBM possesses several merits of IRL algorithms which enables the following analysis over the behavior and motivation of WoW players.
First, the algorithm takes trajectories as input and does not query the policy.
It allows us to process recorded data.
Also, MMBM does not query the environment dynamic which reduces the computational cost especially when the environment is complex and massive.
Third, the algorithm is naturally extended to model the collective behavior of a group when T is composed of trajectories from multiple players.
That helps the model to generate many interesting results as shown below.
4. I MPLEMENTATION D ETAILS
We test MMBM on the WoWAH dataset (Lee et al.
; Bell et al.
; Shen et al.
(2014)), which is an interesting dataset recording a signiﬁcant amount of gameplay data with over 70, players’ movements (regarded as actions) from realm TW-Light’s Hope spanning for the 3-year period.
We treat each player as a human agent who conducts an action at each time interval.
All available data such as current level or joining the guild are treated as observations.
The players’ trajectories are composed of a sequence of movements (actions) and observations (states), which partially reﬂect their playing strategies.
The oracle r is constructed based on the Yee’s research and other WoW case studies (Ducheneaut et al.
; Nardi & Harris (2006)).
It is illustrated in Tbl.in the appendix.
As discussed before, the function approximator which parametrizes the action-value functions is largely depending on the environment.
Taking our experiments on the WoWAH dataset as an example, the Q-network architecture is designed according to the available observations and is applied to all reward signals i = 1, ... , d .
In the network, the categorical elements of the input (e.g. race , class , etc.) are ﬁrst processed by an embedding layer, while the numeral elements (e.g. session length, current level etc.) are ﬁrst fed into a fully connected (FC) layer with rectiﬁer non-linearity.
The output of the embedding layer and FC layer are then concatenated and fed into another FC layer with rectiﬁer non-linearity.
A ﬁnal FC layer is applied to compute the Q ( s , a ) value for each action a ∈ ∪ s A ( s ) .
Our implementation can be viewed as a two-step workﬂow: The ﬁrst step, known as Q-learning, estimates the value functions of the players at different states of the gameplay environment; The second step, an extension of the IRL algorithm, that estimates the optimality of the value function learned at the ﬁrst step.
In essence, the two-step methodology decodes the complex interactions between the players and the game environment in the form of the players’ reward function.
An intuitive illustration of the framework is shown in Fig.
3 in the appendix.
Our code will be publicly released along with this paper.
4. S OLUTION ON D IFFERENT T RAJECTORY S ETS
We present our experimental results of calculating the direction y that most signiﬁcantly separates the demonstrated behavior and the alternative actions.
Recall that y is solved from the linear program equation 9 and note that larger element in y infers relative larger importance of the corresponding element (but it is not linear).
We use trajectories that are randomly drawn from speciﬁc subsets to compose the constraints of equation 9.
We compare the results for different player groups, as shown in 2.
Signiﬁcantly value structures difference is observed between the players at a higher level ( ≥ 50) versus the players at a lower level ( ≤ 49), where the players at the lower level are much more motivated to advance as indicated by the larger component on the Advancement motivation.
It also shows interesting difference among players in different classes, Warrior , Hunter , and Priest , where the Warrior players value more on Advancement and the Priest players value more about relationship.
It agrees with the common knowledge in WoW that the spells of Priest focus on beneﬁting (healing, bufﬁng, etc.)
the team rather than those of Hunder and Warrior whose spells are more related about damage, and damage/tank, respectively.
Lastly, the results also show that players in the guild value more about Teamwork and Relationship motivations as compared to the players that are not in a guild .
The difference of y are distributed into advancement and escapism instead.
Interestingly, those quantitative results agree with previous knowledge-based studies Ducheneaut et al.
(2006); Nardi & Harris (2006) on WoW. Note
that the y vector only demonstrates the direction of the projection hence we normalize the vector to (cid:107) y (cid:107) 1 = 1 in the ﬁgure 4.
P REDICTING A CTIONS AND V ALUES
We evaluate the performance of MMBM in terms of the accuracy of action prediction and the relative accuracy of the estimated value compared with the ground truth.
For prediction, the trained model is required to output the next action of the player, given the current state.
To mitigate the noise and the short interval in the dataset, the prediction is considered correct as long as the predicted action is taken within ﬁve timesteps.
The inverse learning error measures the relative error occurred in the value function estimation, deﬁned as y T | ˆ v − v | / | y T v |
where ˆ v denotes the estimation and v the ground truth.
As the observed reward is always vectorized, we treat each model’s scalarization or projection as if it is correct and count only the error due to value estimation.
We compare MMBM with those methods that can be implemented on WoWAH, which require them to work without querying the environment and the policy.
A close examination of the errors that are made during the prediction yields some interesting understandings.
As MMBM assumes that every player tries to maximize their value in the IRL algorithm, i.e., everyone is regarded as a rational and optimal player.
The assumption is relatively weak enough to include a wide range of diverse behavior, though, MMBM will not be able to distinguish whether a particular action that deviates from an average one is caused by the player’s actual intention or the player’s sub-optimality during the gameplay.
For instance, some of the players could spend hundreds of hours on solo quest but fail to level up quickly.
This could be due to their intention of enjoying doing the quest repeatedly or the players not knowing the optimal strategy to level up.
Considering humans’ different ability or skill levels may help address this limitation of our method and we leave it for future works. [Table]
C ONCLUSIONS AND O PEN P ROBLEMS
We present MMBM, a general IRL model that takes multifaceted human motivations into consideration.
The algorithm relies on relatively weak assumptions and does not require any explicit scalarization of the vector-valued reward function.
Instead, it leverages the Pareto frontier of the value to characterize the set of optimal policies and to measure the optimality of the recorded behavior.
While the algorithm is not relying on the access of policy function nor the dynamics of the environment it can be applied to study complex, interactive environments with its historical dataset.
Our experiments on the WoWAH dataset shows the reasonable analysis on the reward mechanism of the players and also the improved prediction accuracy and inverse learning error.
We view our work as one of the ﬁrst that can combine the richness of psychological and game research theories with the rigorousness of RL models.
Our goal is beyond winning and losing: Not to simply create software agents that beat human in various games or competitions [Table] but to propose methods that can help to understand the intricacy and complexity of human motivations and their behaviors.
We hope to inspire more researchers to investigate this topic further.
Among the challenges in IRL, the vectored reward is the one that has been long open.
MMBM addresses this challenge by incorporating the estimated Pareto frontier of the range of the value function, but the algorithm heavily relies on the exactness of the reward function.
When any element of the reward function is inaccurate, our algorithm is plausible to incur larger error than those who scalarize the vectored reward, as it is not possible in MMBM to adjust the weight of the reward to mitigate the effect of the inaccuracy.
The drawback can be effectively solved if the algorithm is robust with to the reward vector instead of treating it as an oracle.
We leave the problem open for further research.
Category & Deﬁnition
r
[Table]
Advancement describes how fast the player levels up in the game.
It’s the speed the user levels up, divided by the averaged speed at the entire WoWAH.
r
[Table]
Competition describes if the player joins Battleground or Arena and competing with human opponents.
It equals the number of visits.
r
[Table]
Relationship is linear to the duration that the player has been in the current guild .
rTeamwork describes the intention of conducting teamwork, which is the number of recent zones with teamwork features visited.
Zones with teamwork features include Battleground , Arena , Dungeon , Raid , or a zone controlled by The Alliance .
rEscapism is the linear combination of the duration of the recent game session and the number of days the player continuously login to the game recently.
A T HE O RACLE r IN THE E XPERIMENTS
We attach the vector-valued reward, as an oracle, used in our experiments in Tbl.
We note that the vectors should be treated as known and bounded and shall be generated by either collecting observations or applying domain knowledge.
B D YNAMICS OF THE H UMAN M OTIVATION
The motivation of gameplay may evolve.
It can also be impacted by the new design or new versions of the game environment.
We investigate how would a design update affect the players’ motivations and behavior and how we can quantify this impact?
To achieve this, we retrain the linear program 9 with linear constraints random drawing from speciﬁc time ranges.
With the time range moving chronologically, we show the evolution of game motivation characterized by the elements in y .
Fig.
4 illustrates the trend of Advancement , Competition , Relationship , Teamwork , and Escapism .
We observed the dramatic increase in Advancement and Competition during the mid-to-late period on the graph.
It happens at around the 150000th time interval, which coincides with the release of the patch Wraith of the Lich King on November 2008.
Analyzing the game update patch, two primary reasons can explain the increased level of motivation on Advancement.
First, the patch increased the maximum player level from 70 to 80.
As a result, the players with level 70, the previous max level, were rushing to complete the remaining ten leveling ups to reach the new max level.
Second, the patch introduced two new classes in the game, namely Death Knight and Shaman , and this gave incentives to many players to open the secondary accounts and to level up them is the ﬁrst thing to do afterward.
Meanwhile, the reason for more Competition is that many players tend to join player- the versus-player (PvP) to compete with other human players to get more familiar with the mechanism of their new avatar.
It is also noticed that the satisfactions are not independent of each other: players spend more time on advancement usually have insufﬁcient time to complete tasks which require teamwork but provides no experience for leveling up.
As shown in Fig.
4, that the component for teamwork decreases each time the component for advancement increases, and vice versa.
We also analyze the overall trend of the game during the three years when WoWAH was collected.
It turns out that the game emphasis more on teamwork and relationship during the period, partially because the dataset was collected only two years after the game release, and the players are getting more and more involved in the game during that time.
Apart from that, the components of different kinds of motivations are under inﬂuences from both game patches and updates, and game user community.
Overall, our algorithm provides insights in Fig.
4 and the analysis that may be inspiring for researchers and game researchers.