IA-MARL: I MPUTATION A SSISTED M ULTI -A GENT R EINFORCEMENT L EARNING FOR M ISSING T RAINING D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, multi-agent reinforcement learning (MARL) adopts the centralized training with decentralized execution (CTDE) framework that trains agents using the data from all agents at a centralized server while each agent takes an action from its observation.
In the real world, however, the training data from some agents can be unavailable at the centralized server due to practical reasons including communication failures and security attacks (e.g., data modiﬁcation), which can slow down training and harm performance.
Therefore, we consider the missing training data problem in MARL, and then propose the imputation assisted multiagent reinforcement learning (IA-MARL).
IA-MARL consists of two steps: 1) the imputation of missing training data , which uses generative adversarial imputation networks (GAIN), and 2) the mask-based update of the networks , which trains each agent using the training data of corresponding agent, not missed over consecutive times.
In the experimental results, we explore the effects of the data missing probability, the number of agents, and the number of pre-training episodes for GAIN on the performance of IA-MARL.
We show IA-MARL outperforms a decentralized approach and even can achieve the performance of MARL without missing training data when sufﬁcient imputation accuracy is supported.
Our ablation study also shows that both the mask-based update and the imputation accuracy play important roles in achieving the high performance in IA-MARL.
I NTRODUCTION
Reinforcement learning (RL) solves many challenging problems including the game playing (Mnih et al., 2015) and the robot control (Levine et al., 2016), which focus on the single-agent RL environment, modeled as the Markov decision process (Sutton and Barto, 2011).
However, there exist many real-world problems that involve interaction among multiple agents such as multi-robot control (Hüttenrauch et al., 2019) and multiplayer games (Silver et al., 2017; Bard et al., 2020).
Hence, the multi-agent reinforcement learning (MARL) that operates in multi-agent domain has been introduced and now becomes one of the most active and challenging RL research areas.
In MARL, the decentralized approach has been used to train each agent based on its trajectory (Tan, 1993).
However, it often shows unstable and low performance due to non-stationary environment and partially observable information (Tan, 1993; Foerster et al., 2017) that inherits from the decentralization.
Speciﬁcally, as the agents evolve their policies independently, the environment becomes non-stationary, which unstabilizes training at each agent.
In addition, the agent may not observe the information of other agents, which causes low performance in the cooperative or competitive environment (Lowe et al., 2017).
Recently, the centralized training with decentralized execution (CTDE) framework has been introduced for MARL (Oliehoek et al., 2008; Foerster et al., 2018).
This can alleviate the non-stationary environment and the partially observable information problems (Lowe et al., 2017) and encourages coordination among agents (Foerster et al., 2018).
In the execution of CTDE, each agent takes an action based on its observation, while the training of the agents is performed at a centralized server after collecting observations, actions, and rewards of all agents.
In existing works, those data from all agents are assumed to be available at the centralized server, which may not be always true in reality.
The data from distributed agents can be unavailable due to practical reasons including the communication failure, hardware limit, and security attacks (Lakshminarayan et al., 1999; Twala, 2009).
For instance, in wireless sensor applications of MARL such as vehicle tracking (Liang et al., 2020) and environmental monitoring (Li et al., 2020), sensors (i.e., agents) transmit their sensed information to a receiver (i.e., centralized server) for training.
The training data, transmitted from sensors, can be missed when the communication is unstable.
In addition, even when the training data successfully arrives at the centralized server, certain data can be removed from the training dataset due to security attacks such as false data injection (Yan et al., 2016) and unauthorized data modiﬁcation (Ferretti et al., 2014).
As one can readily imagine, this missing training data can cause a serious problem in
MARL as the training cannot be performed.
One possible solution on this missing data problem is to use only the training data that contains data from all agents without missing.
However, in this case, the number of training data can dramatically decrease as the number of agents increases or the data missing happens more often, which delays the training.
Another solution can be the use of imputation for replacing the missing training data in the MARL.
However, the training data of this case can be different from the original data, which potentially degrades the performance.
Therefore, as discussed above, the missing data problem should be carefully considered for bringing the MARL to the next level for wider range of applications.
Despite of it, to the best of our knowledge, the missing data problem in MARL has not been taken into account in existing works.
In this paper, we propose an imputation assisted multi-agent reinforcement learning (IA-MARL) with considering the missing training data problem, where the training data of each agent, consists of observation, action, and reward, can be randomly missed with certain probability.
The proposed
IA-MARL consists of two steps: 1) the imputation of missing training data and 2) the mask-based update of the networks.
Speciﬁcally, for the imputation of the missing training data , we use a generative adversarial imputation network (GAIN) to impute the data from all agents, and we form the data for the training of agents.
We then perform the mask-based update , which trains the value function and the policy of each agent by selectively using the training data of the corresponding agent, not missed over the consecutive times.
In the experimental results, we show the IA-MARL outperforms a decentralized approach and also can achieve the performance of MARL with all training data without missing.
We then also show the performance of IA-MARL for different missing probabilities, the number of agents, and the number of pre-training episodes for GAIN.
From the ablation study, we also verify the importance of the mask-based update as well as the imputation accuracy in multi-agent environments when the training data can be missed.
R ELATED W ORK
Independent Q-learning has been proposed as a decentralized approach to train each agent using its own data independently (Tan, 1993).
The independent Q-learning is used for the tabular environment (Littman, 1994), and deep learning-based approaches are presented in Tampuu et al.
; Gupta et al.
Independent Q-learning, however, suffers from the non-stationary environment and partially observable information problems.
CTDE is one of the solutions for those problems.
In CTDE, the data from all agents are used for the centralized training, while the execution at each agent only requires its observation (Oliehoek et al., 2008).
For instance, the centralized server trains the value function of each agent using observations, actions, and rewards of all agents, while each agent takes an action based on its observation (Lowe et al., 2017) .
Recently, MARL algorithms that adopt CTDE framework have been presented.
For instance, Lowe et al.
(2017) proposes the multi-agent deep deterministic policy gradient (MADDPG) that extends deep deterministic policy gradient (DDPG) for the continuous control of multiple agents.
For the credit assignment problem, the counterfactual baseline (Foerster et al., 2018) and the value function factorization that determines the contribution of each agent are used (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019).
To improve the performance of MARL, the soft value function and the multi-head attention are used in Iqbal and Sha (2019), and the communication between agents that provides additional information to each agent is introduced in Foerster et al.
; Mordatch and Abbeel (2018) while the limited communication channel between agents during the execution is considered in Kim et al. (2019) to address the real world communication channel constraints in MARL.
However, none of the prior work considers the missing training data problem.
Imputation is the research area that replaces missing data, which has been used for many applications including medical data and image concealment (Rubin, 2004).
For the imputation, some techniques such as multivariate imputation by chained equations (MICE) (Buuren and Groothuis-Oudshoorn, 2010), matrix completion (Mazumder et al., 2010), and MissForest (Stekhoven and Bühlmann, 2012) are proposed.
However, when the imputation is used for the data that have large data space (e.g., the data obtained from agents in CTDE), the techniques with insufﬁcient expressive power might result in low performance.
For this case, some imputation techniques that have more expressiveness by adopting the deep neural networks can be more suitable such as the multiple imputation using denoising autoencoders (MIDA) (Gondara and Wang, 2018), the bidirectional recurrent imputation for time series (BRITS) (Cao et al., 2018), and the generative adversarial imputation network (GAIN)
(Yoon et al., 2018).
B ACKGROUND
We consider a decentralized partially observable Markov decision process, deﬁned by a tuple ( S, A, P, r, Ω , O, γ, n ) , where S , A , and Ω are set of states, actions, and observations, respectively.
r and γ are the reward and the discount factor, respectively, and n is the number of agents.
We use s ∈ S , a ∈ A , and o ∈ Ω for a state, an action, and an observation, respectively.
We use subscript i for the corresponding agent and t for the time, e.g., o i,t is the observation of agent i at time t .
We use bold symbols to denote observations, actions, and rewards of all agents, e.g., a t =
( a 1 ,t , · · · , a n,t ) .
Here, P ( s t +1 | s t , a t ) and O ( o t | s t ) are the transition probability and the conditional observation probability, respectively.
3. DDPG AND MADDPG
The objective of the agent in the environment is to maximize the cumulative reward R t =
(cid:80)
Tt (cid:48)
=
t γ t (cid:48) − t r t (cid:48) .
For this, we use the actor-critic method.
The expected cumulative reward for given action and state is Q ( s t , a t ) =
E [ R t | s = s t , a = a t ] , which is called as an action-value function or value function.
Using the Bellman equation, the value function can be rewritten as
Q ( s t , a t ) =
E r t ,s t
[Equation] [Equation]
t +1 [ r t +
[Equation]
( s t +1 ,
[Equation]
t +1 )] .
When the parameter θ is used for the value function approximation, the value function Q can be learned by minimizing the loss L ( θ ) , given as
[Equation]
DDPG is the widely-used choices for the policy update.
The policy parameterized by φ takes state s as an input and outputs deterministic action a = µ φ ( s ) in DDPG, where the gradient of φ is given as
[Equation]
MADDPG is an algorithm that uses DDPG in the CTDE framework (Lowe et al., 2017).
The value function in MADDPG takes observations and actions of all agents as an input.
Meanwhile, the policy of each agent takes its observation as an input since the agent can access to its observation only.
In
CTDE, the loss for the value function and the gradient for the policy are given as
[Equation]
Here, each agent has a different value function and a policy, parameterized respectively as θ i and φ i .
3. I MPUTATION
In statistics, imputation is used to replace missing data with substituted values.
In order to denote the missingness of the data, the mask M that has the same dimension as the data X is used: the k -th component of the mask m k = 0 , when the k -th data x k is missed, and m k = 1 , when x k is not missed.
We denote the obtained data as M (cid:12) X , where (cid:12) is elementwise multiplication.
In the imputation, the completed data ˆ X is given as
[Equation]
where ¯ X is the imputed data using the imputation algorithm including GAIN.
For the loss of the neural network that imputes data, the mean squared error is used, given as
[Equation]
Here, ¯ x k is k -th component of the imputed data.
Note the loss can be estimated when m k = 1 since x k can be used only when it is not missed.
M ETHODS
In this section, after introducing the missing training data problem in MARL and describing the imputation method, we propose IA-MARL.
4. M ISSING T RAINING D ATA P ROBLEM AND I MPUTATION M ETHOD
We consider the MARL environment where the training data of each agent is randomly missed, called the missing training data problem.
Speciﬁcally, the training data from agent i at time t , τ i,t =
( o i,t , a i,t , r i,t ) , is missed with the probability p m i,t =
P ( m i,t = 0 ) < 1 , 1 as well.
where m i,t ∈ { 0 , 1 } denotes whether the data of agent i at time t is missed ( m i,t = 0 ) or not ( m i,t = 1 ).
In the presence of missing data, to obtain completed data of all agents at time t for training, we ﬁrst set the data for the imputation and the mask, respectively, as
[Equation]
where τ t = ( τ 1 ,t , · · · , τ n,t ) and m t = ( m 1 ,t J 1 , | τ 1 ,t | , · · · , m n,t J 1 , | τ n,t | ) .
Here, | · | is a cardinality of set · and J 1 , | τ | is an all-ones vector with length | τ | .
Note that for the accurate imputation of τ i,t , the temporal data correlation as well as the data correlation across different agents should be used.
For the imputation, we use GAIN (Yoon et al., 2018) as it has sufﬁcient expressiveness to impute multi-agent data, does not require original data for the training, and also has state-of-art performance.
The generator G in GAIN takes the obtained data, the mask matrix, and the random matrix as inputs and outputs the imputed data.
The imputed data and the completed data can be presented, respectively, as
[Equation]
Require: Dataset D GAIN contains X t and M t 1: Generate random matrix Z t and hint matrix H t 2: Generate completed data ˆ X t using equation3: Discriminate completed data using equation4: Estimate L D using equation 11 and update discriminator 5:
Generate random matrix Z t and hint matrix H t 6: Generate completed data ˆ X t using equation7: Discriminate completed data using equation8:
Estimate L G using equation 12 and update generator 9: Return ˆ X t where Z t is a random matrix that has the same dimension as X t .
In equation 9, the completed data can be represented as ˆ
X t =
(ˆ τ t − 1 , ˆ τ t , ˆ τ t +1 ) .
The discriminator
D in GAIN takes ˆ X t and the hint matrix
H t as inputs and outputs the probability of being masked for the completed data as ˆ
[Equation]
In equation 10, B t = ( b 1 ,t − 1 , · · · , b n,t +1 ) is a random sequence of 0 and 1 , where p h =
P ( b i,t = 1) is the hint probability.
The role of the hint matrix is to make the discriminator focus on the component with b i,t = 0 .
The objective of the discriminator D is to maximize the probability of estimating the mask matrix correctly, while the objective of the generator G is to minimize the both accuracy of the discriminator and the loss of the imputation.
Therefore, the losses of D and G are, respectively, given by
[Equation]
where D GAIN is the dataset that contains obtained data and its mask matrix, and ˆ m i,t ∈ [0 , 1] is the probability of being masked, i.e., the i -th component of ˆ M t in equation 10.
In equation 12,
L M ( ¯ X t , X t ) is the difference between the imputed data and the obtained data as in equation 6.
The parameter α G controls the importance of the imputation loss, which is a hyperparameter.
The training and the imputation of GAIN are given in Algorithm 1.
[Figure]
In this subsection, we propose IA-MARL, which uses the completed data obtained from GAIN and the mask-based update that stabilizes the training in
MARL for the missing training data problem.
For
IA-MARL, we use MADDPG, but other actor-critic RL methods including policy gradient are also applicable for IA-MARL.
Note that with some modiﬁcation on the value function update, IA-MARL can also be used for the value-based MARL algorithms including VDN and QMIX.
In IA-MARL, the value function and the policy of agent i can be updated using the following loss and gradient.
[Equation]
where θ (cid:48)
i and φ (cid:48)
i indicate the target network parameters periodically updated as in (Hasselt et al., 2016).
Even though we use imputation, the completed data may have different value with the original data, which harms the performance.
To mitigate this, we propose the mask-based update that trains the value function and the policy using ˆ D i , which is the set of the completed data for the training of agent i , given by
[Equation]
where T max is an episode length.
Note that ˆ D i in equation 15 contains data collected across episodes, and ˆ τ T max is included in ˆ D i when m i,T max = 1 to update θ i and φ i at time T max .
The mask-based update means, in the training of agent i , the completed data at time t is used only when the data of agent i is not missed over two consecutive times, t and t + 1 .
An example of ˆ D i is presented in Fig.
As shown in Fig.
1, only the completed data at time 2 and 5 are included in ˆ D 1 , and the completed data at time 1 is included in ˆ D 2 .
In IA-MARL, the imputation and the mask-based update are used to make the policy and the value function (i.e., µ φ
i (ˆ a i,t | ˆ o i,t ) and Q θ i (ˆ o t , ˆ a t ) ) similar to the ones without data missing (i.e., µ φ i ( a i,t | o i,t ) and Q θ i ( o t , a t ) ), respectively.
In case of the policy, as m i,t = 1 (i.e., the data of agent i at time t exists) by the mask-based update, we can have µ φ i (ˆ a i,t | ˆ o i,t ) =
µ φ i ( a i,t | o i,t ) .
In case of the value function, since m i,t m i,t +1 = 1 by the mask-based update, the value function can be updated when ( o i,t , a i,t , r i,t , o i,t +1 , a i,t +1 ) exists that mainly affect the update of Q θ i (ˆ o t , ˆ a t ) .
Hence, if the imputation on the missing data of other agents is accurate, Q θ i (ˆ o t , ˆ a t ) can be similar to Q θ i ( o t , a t ) .
We also show the importance of the mask-based update and accurate imputation in Section 5.3.
1: Initialize D GAIN and GAIN G , D 2: Initialize φ
i , φ (cid:48)
i , θ i , θ (cid:48)
i , ˆ D i ∀ i 3: for episode = 1 to N do 4: if episode = N pre then 5:
Initialize D GAIN , ˆ D i , φ i , φ (cid:48)
i , θ i , θ (cid:48)
i ∀ i 6: end if 7:
Initialize environment and receive o 8: for t = 1 to T max do 9:
For each agent i , select action a i 10:
Store o , a , r , m in D GAIN 11: for all agent i do 12:
Train φ i and θ i using equation 13 and equation13:
Update target network φ (cid:48) i and θ (cid:48)
i 14: end for 15: end for 16:
Train GAIN using D GAIN and update ˆ D i , then initialize D GAIN 17: end for
The training algorithm for IA-MARL is provided in Algorithm 2.
In IA-MARL,
GAIN is periodically trained and outputs completed data using Algorithm 1, where the dataset for GAIN, D GAIN , is initialized to prevent the overﬁtting of GAIN to the trajectory of outdated policies.
We pre-train
GAIN through N pre episodes for better imputation accuracy.
After N pre episodes, we initialize the parameters of all agents, i.e., φ i , φ (cid:48)
i , θ i , θ (cid:48)
i , and ˆ D i , ∀ i , to prevent the agent from training with inaccurate data.
The components in Algorithm 2 can be modiﬁed according to the MARL algorithm, where we use MADDPG in this work.
Therefore, the training procedure follows that in Lowe et al.
(2017), e.g., the action is selected as a i = µ φ i ( o i ) + N , where N is noise for exploration.
In IA-MARL, the number of training data is smaller than the number of obtained data due to the mask-based update.
When the missing probability of agent i over time is equal, i.e., p m i,t =
p m i , ∀ t , and the number of obtained data is N D , the number of completed data for the training of agent i is given as
[Equation]
(16) since the condition in equation 15 is satisﬁed when the data of agent i exists during the consecutive times.
Note that E (cid:2) |
ˆ D i | (cid:3) decreases with p m i .
When the imputation is not applied, the centralized server can only use the data which is not missed from all agents over consecutive times, i.e.,
D P = { τ t | m i,t m i,t +1 = 1 , ∀ i, t ∈ { 1 , · · · , T max − 1 }} ∪ { τ T max | m i,T max , ∀ i } .
In this case, the imputation is not required for the training.
Hence, for given N D , the number of the training data is given as
[Equation]
Therefore, the number of training data without imputation, E (cid:2) |D P | (cid:3) , decreases exponentially with the number of agents n while the number of training data for IA-MARL, E (cid:2) |
ˆ D i | (cid:3) , is not affected by n .
We show the performance of MADDPG without imputation in Appendix B.
E XPERIMENTAL R ESULTS
In this section, we provide MARL environment and hyperparameters, and then show the performance of IA-MARL with different missing probabilities, the number of agents, and the number of pretraining episodes for GAIN.
Furthermore, we provide the ablation study which shows the importance of the mask-based update and accurate imputation in IA-MARL. [Figure]
5. E NVIRONMENT AND H YPERPARAMETERS
Among widely-used MARL benchmark environments including the multi-agent particle environments (MPE) (Lowe et al., 2017; Mordatch and Abbeel, 2018) and the starcraft multi-agent challenge (SMAC) (Samvelyan et al., 2019), we use and modify the MPE that contains mixed cooperative and competitive environments.
Note that SMAC is not used in experiments since MADDPG, adopted in
IA-MARL, generally does not perform well in SMAC (Papoudakis et al., 2021).
The speaker-listener environments are composed of several pairs of agents, where each pair has a speaker and a listener, and landmarks have unique colors.
In the environment, the speaker observes the target color and sends a message to the listener.
The listener then moves after observing the locations of landmarks and the message.
Each speaker-listener pair gets shared rewards, which is determined by the distance between the listener and the target landmark.
Note that the speaker should learn which message to send, and the listener, who does not know the target landmark, should learn the message.
We set the number of landmarks as 2 + n/ 2 so that the number of landmarks increases with the number of pairs (i.e., n/ 2 ).
The tag environments are composed of one prey and multiple predators, where the number of predators is set as 3, 4, and 5 (which means the number of agents are n = 4 , 5 , and 6 , respectively).
In the environment, the objective of the prey is to run away from the predators, and the objective of the predator is to catch the prey.
Each time when the prey collides with the predator, predators get a shared reward while the prey is penalized.
We evaluate the performance of the algorithm across 10 runs with different random seeds and the shade in graphs represents the conﬁdent interval.
We use MADDPG for IA-MARL.
As baselines, we use MADDPG and DDPG without missing training data (i.e., p m = 0 ).
We use the same hyperparameters in Lowe et al.
(2017), except for the training frequency of the network parameters in IA-MARL.
Speciﬁcally, while MADDPG updates every 100 samples and collects 1024 samples before making an initial update, IA-MARL updates every 100 / (1 − p m ) 2 samples and collects 1024 / (1 − p m ) 2 samples before making an initial update.
Accordingly, the target network parameters are less frequently updated in IA-MARL.
In the speaker-listener environment, we evaluate the performance of IA-MARL against MADDPG speaker and listener.
In the tag environment, we evaluate IA-MARL against MADDPG predators and DDPG prey.
5. P ERFORMANCE OF IA-MARL
Figure 2 shows the average rewards as a function of the episodes with different number of pretraining episode and training data missing probability in the speaker-listener environment with one speaker-listener pair.
In Figs.
2a-2d, p m = 0 .
1 , 0 . 2 , 0 . 3 , 0 . 4 is used, respectively.
We observe that the performance of IA-MARL outperforms the decentralized approach, i.e., DDPG, and also can achieve the performance of MARL without missing training data, i.e., MADDPG with p m = 0 , when GAIN is pre-trained sufﬁciently.
Speciﬁcally, IA-MARL achieves the performance of MADDPG with any
N pre for p m = 0 .
1 and N pre ≥ 4 × 10 3 for p m = 0 . 2 and 0 . 3 .
When p m = 0 .
4 , IA-MARL could not achieve the performance of MADDPG, but it is expected to achieve the similar performance of MADDPG with larger N pre .
Note that as p m increases, mainly due to the smaller number of [Figure] training data as shown in equation 16, IA-MARL slowly achieves the performance of MADDPG.
Furthermore, as p m increases, more pre-training episodes are required since GAIN is less trained due to the larger value of imputation loss.
When N pre is small, large imputation error in completed data causes IA-MARL to hardly achieve the performance of MADDPG.
Figure 3 shows the average rewards as a function of the episodes with different pre-training episode and missing probability.
In Fig.
3a, the average rewards are shown for two pairs of speaker-listener.
In Figs.
3b-3d, we show the average rewards when the number of predators is 3, 4, and 5, respectively, where the upper graphs show the rewards of the prey and the lower graphs show those of predators.
We observe IA-MARL can achieve the performance of MADDPG, and more pre-training and training episodes are required as p m increases.
In Fig.
3a, when p m = 0 .
1 and 0 .
2 , IA-MARL achieves the performance of MADDPG slowly.
Furthermore, Figs.
3b-3d also show IA-MARL achieves the performance of MADDPG when p m = 0 .
1 , where the rewards of predators steadily increase, and the rewards of the prey decrease.
However, as p m increases, IA-MARL cannot achieve the performance of MADDPG due to large imputation error.
For instance, in Fig.
3a,
when p m = 0 .
3 , the performance of IA-MARL is in between the performance of MADDPG and DDPG.
Similarly, in Figs.
3c and 3d,
when p m = 0 .
2 and 0 .
3 , IA-MARL cannot achieve the performance of MADDPG.
rewards with/without the mask-based update and with GAIN/random imputation are shown.
We use N pre = 10 4 for
GAIN. We also observe as the number of agents increases, more N pre is required to make IAMARL achieve the performance of MADDPG.
In the speaker-listener environment with p m = 0 .
3 , N pre = 4 × 10 3 is required for n = 2 (see
Figure 2a), while N pre > 10 4 is required for n = 4 (see Figure 3a).
Similarly, in the tag environment with p m = 0 . 2 , N pre = 10 4 is required for n = 4 (see Figure 3b), while N pre > 10 4 is required for n = 5 and n = 6 (see Figures 3c and 3d).
The main reasons to require more N pre for the larger number of agents are 1) environment with more agents becomes more complex and 2) the generator and the discriminator in
GAIN require more training data as the input and output spaces increase with the number of agents.
5. A BLATION S TUDY AND L IMITATION
We compare the performance of IA-MARL with and without the mask-based update and with and without GAIN imputation.
For simplicity, we call IA-MARL without the mask-based update as simple-IA-MARL.
In simple-IA-MARL, agent i is trained using the completed data
ˆ D =
{ ˆ τ t | t ∈ { 1 , · · · , T max − 1 }} , which is different from IA-MARL that trains agent i using data ˆ D i .
When
GAIN is not used, we use random imputation replacing missing training data with uniform random variables, i.e., ˆ o i,t ∼ U (min o i,
t ∈ Ω i ( o i,t ) , max o i,t ∈ Ω
i ( o i,t )) , where Ω i is the set of all o i,t .
Note the random imputation can be regarded as a simple imputation method that has low imputation accuracy.
We show the performance of IA-MARL and Simple-IA-MARL with different imputation method and missing probability in Appendix B.
Figure 4 shows the ablation study of IA-MARL when p m = 0 .
1 and N pre = 10 4 .
Firstly, we observe the average rewards of simple-IA-MARL are even lower than that of DDPG for all imputation methods.
The low performance of simple-IA-MARL is due to the effect of imputation error on the gradient estimation, which is alleviated in IA-MARL by the mask-based update.
We also observe that the average reward of IA-MARL with random imputation is similar or lower than that of DDPG.
Since the accuracy of random imputation is low, both the MARL with/without the mask-based update cannot achieve the performance of MARL without missing data.
Similarly, when the number of pre-training episodes for GAIN, N pre , is small, the imputation accuracy is low, so the performance of IA-MARL is degraded as shown in Figs.
Therefore, when the imputation is not sufﬁciently accurate, the performance of IA-MARL can be low.
C ONCLUSION
We propose IA-MARL for the training of agents in the presence of missing training data.
The key idea is to use the imputation for replacing the missing data and the mask-based update that selectively uses the training data for each agent.
In IA-MARL, we use GAIN for the imputation and MADDPG for
MARL algorithm.
In the experimental results, we verify the performance of IA-MARL for different training data missing probabilities, the number of agents, and the number of pre-training episodes for GAIN.
We show that IA-MARL with missing training data achieves comparable performance with MADDPG without missing training data, when GAIN is pre-trained sufﬁciently.
Through the ablation study, we also show the importance of the mask-based update and the imputation accuracy in IA-MARL for achieving high performance
A H YPERPARAMETERS
We evaluate the performance of the algorithm across 10 runs with different random seeds.
We use grid search to ﬁne-tune the learning rate and the number of hidden nodes of MADDPG and DDPG.
For GAIN, we use the generator and the discriminator consisting of 2 hidden layers with 128 nodes, where ReLU activation is used except for the last layers that use sigmoid.
We use Adam optimizer with a learning rate of 0.001 for the training of GAIN.
The hint probability p h = 0 .
5 and α G =are used, and GAIN is trained every 5 episodes.
The replay buffer for GAIN is initialized after each training.
We use MADDPG for IA-MARL.
Each agent has the value function and the policy consisting of 2 hidden layers with 64 nodes.
We use ReLU for activations except for the last layer of value function that does not use activation and the last layer of policy that uses tanh activation.
We use Adam optimizer for all agents with the learning rate of 0.01 and use target networks with τ = 0 . 01 .
The length of the replay buffer is 10 6 .
MADDPG updates every 100 samples and collects 1024 samples before making an initial update.
Since the mask-based update makes the number of training data for IA-MARL smaller, we update IA-MARL
every 100 / (1 − p m ) 2 samples and collects 1024 / (1 − p m ) 2 samples before making an initial update.
Accordingly, the target network parameters are less frequently updated in IA-MARL.
B A DDITIONAL E XPERIMENTAL R ESULTS
In this Appendix, to show the effectiveness of IA-MARL in terms of the number of training data discussed in Section 4.2, we show the training curves of MADDPG with missing training data and compare them with IA-MARL.
Then we show the performance IA-MARL with different imputation methods.
Furthermore, we evaluate the computational time of IA-MARL.
[Figure]
agents. Figure 5 shows the performance of MADDPG with different p m in Speaker-Listener environment with n = 2 and n = 4 .
We update MADDPG every N D = 100 / { (cid:81) ni
=1 (1 − p
m i ) 2 } steps since the number of data for the training decreases due to the data missing.
When the training data is missed, except for Fig.
5a, MADDPG has lower performance than IA-MARL due to the small number of training data E (cid:2) |D P | (cid:3) as in equation 17.
Note that as the number of agents becomes larger, MADDPG is more slowly trained, while the number of training data of IA-MARL is not affected by the number of agents. [Figure]
Figure 6 shows the performance of IA-MARL and Simple-IA-MARL with different imputation methods. In Figs. 6a and 6b, we replace the missing training data with uniform random variables, i.e., ˆ o i,t ∼ U (min o i,t ∈ Ω i ( o i,t ) , max o i,t ∈ Ω i ( o i,t )) , where Ω i is the set of all o i,t .
In Figs.
6c and 6d, we replace the missing training data with the unobserved token − 1 , i.e., ˆ o i,t = − 1 .
Note that for the both IA-MARL and Simple-IA-MARL, the GAIN shows the better performance than the random imputation and the use of unobserved token for the missing training data.
Figure 7 shows the computation time of IA-MARL across 1000 episodes when the numbers of agents are 2, 4, and 5, respectively.
We use Intel i7-9700K CPU and NVIDIA GeForce RTX 2080Ti GPU for the evaluation.
The time for sampling and MARL training takes a major portion compared to the time for the imputation and GAIN training.
Note that as the number of agents increases, the time for sampling and MARL training linearly increases since each agent has the value function and the policy that should be trained.
On the other hand, GAIN has two networks, i.e., the generator and the discriminator.
Therefore, the time for imputation and GAIN training slowly increases with the number of agents compared with the time for sampling and MARL training.