N EURON AS AN A GENT
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose Neuron as an Agent (NaaA) as a novel framework for reinforcement learning (RL), and explain its optimization method.
NaaA incorporates all neural network units as agents and optimizes the reward distribution as a multi-agent RL problem.
First, showing optimization of NaaA, this report describes the negative result that the performance decreases if we naively consider the units as agents.
To resolve that difﬁculty, we introduce a mechanism from game theory.
As a theoretical result we demonstrate that the agent obeys the system to maximize its counterfactual return as the Nash equilibrium of the mechanism.
Subsequently we show that learning counterfactual returns leads the model to learning optimal topology among units.
We propose adaptive dropconnect , a natural extension of dropconnect.
Finally, we conﬁrm that optimization with the framework of NaaA leads to better performance of RL, with numerical experiments.
Speciﬁcally, we use a single-agent environment from Open AI gym, and a multi-agent environment from ViZDoom.
I NTRODUCTION
Deep reinforcement learning (DRL) succeeds in many areas.
Deep Q-Network (DQN) (Mnih et al., 2015; Silver et al., 2016) ﬁnds the optimal action from a screen sequence from Atari, and selects the move closest to win from a face of a board of Go.
Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) realizes the multiple-join control considering conditions such as friction and gravity factors in a physical space.
The applicability of DRL is becoming wider year by year.
Reasonable performance is reported for 3D games such as Doom (Dosovitskiy & Koltun, 2016).
A neural network is workable for DRL because a neural network abstracts the implicit state in an environment and obtains an informative state representation.
From a micro perspective, the abstraction capability of each unit contributes to the return of the entire system.
Therefore, we address the following question.
Will reinforcement learning work even if we consider each unit as an autonomous agent?
The contribution of this paper is that we propose Neuron as an Agent (NaaA) as a novel framework for RL, and explain its optimization method.
NaaA incorporates all neural network units as agents and optimizes the reward distribution as a multi-agent RL problem.
In the of NaaA reward design, a unit distributes its received reward to other input units, passing its activation to the unit as cost.
Consequently, the actual reward is proﬁt, deﬁned as the difference between inﬂow (received reward) and outﬂow (paid cost).
In the setting, the economic metaphor can be introduced: proﬁt is the balance of revenue and cost.
Therefore, a unit should address tradeoffs between optimization of cumulative revenue maximization and cumulative cost minimization.
This paper is organized as presented below.
First, showing the optimization of NaaA, this report describes the negative result that the performance decreases if we naively consider units as agents.
As a solution to this difﬁculty, we introduce a mechanism of auction which applies game theory.
As a theoretical result, we demonstrate that the agent obeys to maximize its counterfactual return as the
Nash equilibrium.
The counterfactual return is that by which we extend counterfactual reward, the criterion proposed for multi-agent reward distribution problem (Agogino & Tumer, 2006), along a long time axis.
Subsequently, we present that learning counterfactual return leads the model to learning optimal topology between the units.
In addition, we propose adaptive dropconnect , a natural extension of dropconnect (Wan et al., 2013).
Adaptive dropconnect combines dropconnect, which pure-randomly masks the topology, with adaptive algorithm, which prunes the connection with less counterfactual return with higher probability.
It uses " -greedy as a policy, and is equivalent to dropconnect in the case of " = 1 .
It is equivalent to counterfactual return maximization, which constructs the topology deterministically in the case of " = 0 .
Finally, we conﬁrm that optimization with the framework of NaaA leads to better performance of
RL, with numerical experiments.
Speciﬁcally, we use a single-agent environment from Open AI gym, and a multi-agent environment from ViZDoom.
Although considering all the units as agents might be simplistic at ﬁrst glance, it has a wider applicable area.
From the perspective of optimization for a single neural network, it can be applied to pruning by optimizing the topology.
Furthermore, introducing the concept of reward distribution divides the single neural network to numerous autonomous parts.
It enables us not only to address sensor placing problem in IoT for partially observed Markov decision process (POMDP): arbitrary incentivized participants can join the framework.
R ELATED W ORK
NaaA belongs to a class of partially observable stochastic game (POSG)
(Hansen et al., 2004) because it processes multiple units as agents.
POSG, a class of reinforcement learning with multiple agents in a POMDP environment, presents several research issues, one of which is communication.
CommNet (Sukhbaatar et al., 2016), which exploits the characteristics of a unit that is agnostic to the topology of other units, employs backpropagation to train multi-agent communication.
Another one is credit assignment.
Instead of reward R ( a t ) of an agent i for actions at t a t , QUICR-learning (Agogino & Tumer, 2006) maximizes counterfactual reward R ( a t ) (cid:0) R ( a t (cid:0) a it ) , the difference in the case of the agent i takes an action a it ( a t ) and not ( a t (cid:0) a it ).
COMA (Foerster et al., 2017) also maximizes counterfactual rewards in an actor–critic setting.
In the setting, all actors have common critics, which improves both actors and critics with time difference (TD)-error of a counterfactual reward.
This paper uniﬁes both issues: communication and credit assignment.
The main proposal is a framework to manage the agents to maximize the counterfactual return , the extended counterfactual reward along the time axis.
Training a neural network with a multi-agent game is an emerging methodology.
Generative adversarial nets (GAN) (Goodfellow et al., 2014) have the goal of obtaining true generative distribution as a Nash equilibrium of a competitive game that includes two agents with contradictory rewards: a generator and a discriminator.
In game theory, the outcome maximizing overall reward is named
Pareto optimality.
Nash equilibrium is not guaranteed to converge to Pareto optimality.
The difference between them is designated as a dilemma.
Because the existence of a dilemma depends on the reward design, methods to resolve dilemmas with good reward design are being investigated: mechanism design (Myerson, 1983) is also known as inverse game theory.
Mechanism design is applied to auctions (Vickrey, 1961) and matching (Gale & Shapley, 1962).
GAN and our proposal, NaaA, are outcomes from mechanism design.
NaaA applies a digital goods auction (Guruswami et al., 2005) to reinforcement learning with a multi-agent neural network, to obtain a maximized return by units as a Nash equilibrium.
Adaptive DropConnect (ADC), which we propose in a later part of this paper, extends DropConnect (Wan et al., 2013), a regularization technique.
The idea of ADC (instead of dropping each connection between units in constant probability, using skew probability correlated to the absolute value of weights) is eventually closer to Adaptive DropOut (Ba & Frey, 2013), although the derivation differs.
The adjective “adaptive” is added with respect to the method.
Optimizing the neural network with
RL was investigated by Andrychowicz et al.
In contrast to their methods, which use recurrent neural network (RNN) and which therefore have difﬁcult implementation, our method is RNN-free and forms as a layer.
For those reasons, its implementation is simple and fast.
Moreover, it has a wide area of applicability.
B ACKGROUND
First, we consider a POMDP environment in which a single agent acts.
The POMDP environment is a seven-tuple ( S H ; A ; T ; R ;
S O ; O ; (cid:13) ) , where S H represents a set of states, A stands for a set of actions, T denotes a transitive probability, S O represents a possible set of observations, O denotes a set of observation probability, and (cid:13) is the discount rate.
An agent partially predicts state h 2 S H through an observation s 2 S O .
Generally, s has higher dimensions than h , and is complex.
For example, although Atari 2600 has a read only memory (RAM) as the true state, which contains 128 bytes, the generated image from that s has more than 10,000 dimensions.
Therefore, DQN and DRQN abstract s , and create original state representation to predict good action efﬁciently.
(Although the original paper of DQN assumes MDP, the paper of DRQN pointed out that the environment is POMDP).
Although DQN does not address the state transition directly because it is model-free method, some interpretations hold that the hidden state representation is learned in the previous layer of the output layer (Zahavy et al., 2016)
Using the method below, we assume that the agent chooses an action through a neural network.
The POSG environment is a natural extension of POMDP to multi-agent environment deﬁned by a tuple ( S H ; A i ; T ; R i ; S i O ; O i ; (cid:13) i ) i 2I , where I is a ﬁnite set of agents indexed 1.
We employ several concepts from game theory.
Although RL and game theory are typically investigated in parallel, several concepts in game theory can be written in the domain of RL.
The (Bayesian)
Nash equilibrium ^ (cid:25) i is a policy by which all agents maximize their expected reward.
That is,
[Equation]
where A (cid:0) i is a set of actions except for i .
Intuitively, the equation took an expected value of reward to integrate out other agents’ unobserved actions.
Because the Nash equilibrium is sufﬁcient to state only the best action in the most cases, we use the notation with action ^ a in the following.
The design of NaaA is inspired by neuroscience.
A neuron in a neurocircuit consumes adenosine triphosphate (ATP) supplied from connected astrocytes.
The astrocyte is a glia cell, which forms the structure of a brain.
It supplies fuel from the vessel.
Because the amount of ATP is constrained, the discarded neuron will become extinct with execution of apoptosis.
Also, because apoptosis of a neuron is restrained by neurotrophins (NTFs) such as nerve growth factor (NGF) and brain-derived neurotrophic factor (BDNF), neurons which can obtain much NTF will live.
The perspective of interpreting a neuron as an independent living object is known as neural Darwinism (Edelman, 1987).
N EURON AS AN A GENT
A typical artiﬁcial neural network is a directed graph G =
( V ; E ) among the units.
V = f v 1 ; : : : ; v N g is a set of the units.
E (cid:26) V 2 is a set of edges representing connections between two units.
If ( v i ; v j ) 2 E , then connection v i !
v j holds, indicating that v j observes activation of v i .
We denote activation of the unit v i at time t as x it 2 R .
Additionally, we designate a set of units which unit i connects to as N out i =
f j j ( v i ; v j ) 2 Eg and a set of units which unit i is connected from as N in i =
f j j ( v j ; v i ) 2 Eg .
We denote N i
= N in i [ N out i .
NaaA interprets v i as an agent.
Therefore, G is a multi-agent system.
An environment for v i comprises an environment that the multi-agent system itself touches and a set of the unit to which v i directly connects:
f v i 2 V j i 2 N i g .
We distinguish both environments by naming the former as an external environment, and by naming the latter as an internal environment.
v i will receive rewards from both environments.
We add the following assumption for characteristics of the v
i
N1: (Selﬁshness) Instead of minimizing the global training error, at each timing t , v i acts to maximize toward maximizing its own return (cumulative discounted reward)
G it =
∑ Tk =0 (cid:13) k R i;t + k , where (cid:13) 2 [0 ; 1] is the discount rate and T is the terminal time.
N2:
(Conversation) The summation of a reward by which V will receive both an internal and external environment R it over all the units are equivalent to reward R ex t , which the entire multi-agent system receives from the external environment.
N3: (Trade)
The v i receives internal reward (cid:26) jit from v j 2 V in exchange of activation signal x i before transferring the signal to the unit.
At the same time, (cid:26) jit is subtracted from the reward of v j . N4: (NOOP) v
i has NOOP (no operation), for which the return is (cid:14) > 0 as an action.
With NOOP, the unit inputs nothing and outputs nothing.
In terms of neuroscience, N1 states that the unit acts as a cell.
N2 and N3 state the distribution of
NTF.
N4 corresponds to apoptosis.
NOOP is selected when the expected returns of the other actions are non-positive.
In the following, we construct the framework of NaaA from the assumptions.
4. C UMULATIVE D ISCOUNTED P ROFIT M AXIMIZATION F RAMEWORK
We denote the external reward by which unit v i receives at time step t as R ex it , where ∑ ni =1 R ex it =
R ex t holds.
From N3, reward R it , which v i receives at t can be written as the following.
[Equation]
The equation is divided into positive terms and a negative term, we name the former as revenue, and the latter as cost, and denote them respectively as r it
=
R ex it +
[Equation]
We name R it as proﬁt.
In this case, v i maximizes the cumulative discounted proﬁt G it represented as
[Equation]
G it is unobserved unless the time is reached at the end of the episodes.
Because prediction based on the current value is needed to select the optimal actions, we approximate G it with value function
V (cid:25) i i
( s it )
=
E (cid:25) i
[ G it
j s it ] where s it 2 S O .
In this case, the following equation holds.
[Equation]
Therefore, we need only consider maximization of revenue, the value function, and cost minimization.
R it > 0 , i.e., r it >
c it indicates that the unit gives the additional value to the obtained data.
The unit acts NOOP
because V (cid:25) i i
( s it ) (cid:20)
< (cid:14) if R it (cid:20) 0 for all t .
O PTIMIZATION
To maximize the cumulative discounted proﬁt in a framework of NaaA, it is important to balance the two contradicting criteria: revenue r it and cost c it .
To achieve that, we use the mechanism design.
We introduce mechanism design because, unlike several existing studies (Sukhbaatar et al., 2016), NaaA assumes that all agents are not cooperative but selﬁsh.
If we naively optimize the optimization problem of NaaA, then we obtain the trivial solution that the internal rewards will converge to 0, and that all the units become NOOP.
Therefore, the multi-agent system should select the action with no information.
It is equivalent to taking an action randomly.
For that reason, the external reward R e xt shrinks markedly.
5. E NVY - FREE A UCTION
To achieve Pareto optimality, we borrow the idea from the digital goods auction.
The auction theory belongs to mechanism design.
It is intended to unveil the true price of goods.
Digital goods auction is one mechanism from auction theory.
It is target to copyable goods without cost, such as digital books and music.
Although several variations of digital goods auctions exist, we use an envy-free auction (Guruswami et al., 2005) because it requires a simple assumption: the same goods have one price simultaneously.
In NaaA, it can be represented as the following assumption:
N5: (Law of one price)
If (cid:26) j 1 ;i;t ; (cid:26) j 2 ;i;t > 0 , then (cid:26) j 1 ;i;t = (cid:26) j 2 ;i;t .
Therefore, v i has an intrinsic price at the same timing t .
We denote the price as q it .
We present the envy-free auction process at the left of Figure 1.
It shows the negotiation process between one unit in sending activation and a group of units that buy the activation.
The negotiation performed per time step in RL.
We name the unit in sending activation as a seller, and units in buying activation as buyers.
First, the buyer bids the unit in bidding price b jit ( 1 ).
Next, the seller decides the optimal price ^ q it , and performs allocation ( 2 ).
After allocation, the buyers perform payment as (cid:26) jit =
g jit ^
q it ( 3 ).
The seller only sends activation x i to the allocated buyers ( 4 ).
A buyer which cannot receive the activation approximates x i with E (cid:25) [ x i ] .
In the following, we discuss revenue, cost, and value functions based on Eq:(4).
Revenue :
The revenue of a unit is given as
[Equation]
where g ( (cid:1) ; (cid:1) ) is allocation.
It is deﬁned using a step function H ( (cid:1) ) as g ( b; q ) =
H ( b (cid:0) q ) .
d it ( q it ) is a count of units for which the bidding price for q it is greater than or equal to q it , designated as demand.
q it maximizing the equation is designated as the optimal price.
It is denoted as ^ q it .
Because the second term in the equation is independent of q t , the optimal price ^ q it is given as
[Equation]
We present the curve of q it on the right side of Figure 1.
Cost : The cost is an internal reward that the unit should pay to other units.
It is represented as shown below.
[Equation]
Although c it itself is minimized when b ijt = 0 , this represents a tradeoff with the following value function.
Value Function : Activation x i depends on input from the units in N i ni .
It affects the bidding price from units in N o ut
i .
If we minimize b ijt and let b ijt = 0 , then the purchase of activation fails, and the reward the unit can obtain from the units to which the unit connects becomes lower in the future.
Then, we designate the allocation as g it =
( g i 1 t ; : : : ; g iNt ) T , and consider effects for value functions in the cases when a unit succeeds in purchasing v j or not.
The value function can be written as the equation using a state-value function Q ( s i;t +1 ; g i;t +1 ) .
[Equation]
We designate o ijt = Q (cid:25) i i
( s it ; e j ) (cid:0) Q (cid:25) i
i ( s it ; 0 ) as the counterfactual return , which is equivalent to the cumulative discount value of counterfactual reward (Agogino & Tumer, 2006).
That is, the cost the unit will pay is ^ q it in success of purchasing data, and o it otherwise.
Therefore, the optimization problem is presented below.
[Equation]
We take the expectation E ^ q t
[ (cid:1) ] because the asked price ^ q t is unknown for v i , except for ^ q it , and g iit = 0 .
Then, what is bidding price b it to maximize return?
The following theorem holds.
Theorem 5.1.
(Truthfulness) the optimal bidding price for maximizing return is ^ b it
=
o it .
See the Appendix for the proof.
That is, the unit should only consider its counterfactual return (!).
Consequently, in the mechanism of
NaaA, the unit obeys as if performing valuation to the other units, and declares the value truthfully.
Then, the following corollary holds:
Corollary 5.1.
The Nash equilibrium of an envy-free auction ( b it ; q it ) is ( o it ; argmax q qd it ( q )) .
The remaining problem is how to predict o t .
Although several method can be applied to this problem, we use Q -learning to predict o t .
As o it is difference of two Q s, we approximate each of Q .
Other
RL such as SARSA and A3C can be employed.
We parametrize the state with a vector s t which contains input and weight.
ϵ -greedy policy with Q -learning typically suppose that discrete actions
So, as an action, we employ allocation g ijt instead of b it and q it .
The overall algorithm is shown in
Algorithm 1.
5. A DAPTIVE D ROP C ONNECT
Actually, NaaA is useful not only for multi-agent RL, but also for training of the network.
Typical training algorithms of a neural network such as those of RMSProp (Tieleman & Hinton, 2012) and
Adam (Kingma & Ba, 2014) are based on stochastic gradient descent (SGD).
The optimization is performed sequentially.
Therefore, the problem can be interpreted as a problem to update the state (i.e., weight) to the goal, which is minimization of the expected likelihood.
The learning can be accelerated by application of NaaA to the optimizer.
We designate the application of NaaA to SGD as Adaptive DropConnect (ADC), which is eventually a combination of
DropConnect (Wan et al., 2013) and Adaptive DropOut (Ba & Frey, 2013).
We introduce ADC herein as one application of NaaA.
ADC uses NaaA for supervised optimization problem with several revisions.
First, an environment has an input state such as an image.
The agent is expected to update its parameters to maximize its reward obtained from the criterion calculator.
The criterion calculator gives batch-likelihood as the reward to the agent.
The agent is a classiﬁer which updates its weights to maximize the reward from the criterion calculator.
The weights are recorded as an internal state.
As a counterfactual return o ijt , we used a heuristic that uses the absolute value of weight j w ijt j , which is the same technique as that used by Adaptive DropOut.
We use the absolute value of weights because it is the update amount for which the magnitude of error of the output of units is proportional to j w ijt j .
1: for t = 1 to T do 2:
Compute a bidding price for every edge: for ( v j ; v i ) 2 E do b ijt
Q (cid:25) i
( s it ; e j ) (cid:0) Q (cid:25) i
( s it ; 0 ) 3:
Compute an asking price for every node: for v
i 2 V do ^ q it argmax q 2 [0 ; 1 ) qd it ( q ) : 4: for ( v i ; v j ) 2 E do 5:
Compute allocation: g jit
H ( b jit (cid:0) ^ q it ) 6:
Compute the price the agent should pay: (cid:26) jit g jit ^ q it 7: end for 8:
Make a payment: for v i 2 V do R it ∑ j 2 N out i (cid:26) jit (cid:0) ∑ j 2 N in i (cid:26) ijt ; 9:
Make a shipment: for v i 2 V do ~ x ijt =
g ijt x ijt
+ (1 (cid:0) g ijt )(cid:22) x ijt 10: for v
i 2 V do 11:
Observe external state s e xit 12: s it ( s e xit ; ~x it ; (cid:18) i ) 13:
Sample action a e xit (cid:24) (cid:25) e xi ( s it ) 14:
Receive external reward R it
R it + R e xit
( a e xit ) 15:
Update Q (cid:25) i under the manner of Q -learning by calculating the time difference (TD)-error 16: end for 17: end for
1: for t = 1 to T do 2:
Compute a bidding price for every edge: for ( v j ; v i ) 2 E do b ijt j w ijt j 3:
Compute an asking price for every node: for v
i 2 V do ^ q it argmax q 2 [0 ; 1 ) qd it ( q ) : 4: for ( v i ; v j ) 2 E do 5:
Compute allocation: g jit
H ( b jit (cid:0) q
) 6: end for 7:
Sample a switching
Sample the random mask
Generate the adaptive mask:
M
U t ◦ M t + (1 (cid:0) U t ) ◦ G
Compute h t for making a shipment: h t ( M ′ t ◦ W t ) x t + b t 11:
Update W t and b t by backpropagation.
12: end for
The algorithm is presented as Algorithm 2.
Because the algorithm is quite simple, its implementation can be performed easily.
For that reason, it can be widely applied for most general deep learning problems such as image recognition, sound recognition, and even for deep reinforcement learning.
E XPERIMENT
To conﬁrm that NaaA works widely with machine learning tasks, we conﬁrm our method of supervised learning tasks as well as reinforcement learning tasks.
As supervised learning tasks, we use typical machine learning tasks such as image classiﬁcation using MNIST, CIFAR-10, and SVHN.
As reinforcement tasks, we conﬁrm single- and multi-agent environment.
The single-agent environment is from OpenAI gym.
We conﬁrm the result using a simple reinforcement task: CartPole.
In multi-agent, we use ViZDoom, a 3D environment for reinforcement learning.
6. C LASSIFICATION 6.1. S ETUP
In this experiment, we used three types of datasets, MNIST, CIFAR-10 and STL-10.
The task given here is to predict the label for each image.
The number of class is 10 in those three datasets.
The ﬁrst dataset, MNIST, is a collection of black and white images of handwritten digits whose size is 28x28.
The training set and test setare composed of 60,000 examples and 10,000 examples respectively.
The images in CIFAR-10 dataset are colored and the size of each image is 32x32.
The task is to predict what is shown in each picture.
This dataset contains 6,000 images per class (5,000 for training and 1,000 for test).
STL-10 is a dataset for image recognition, the number of which is 1,300 for each class (500 for training and 800 for test).
The size of each image is 96x96.
In this experiment, however, images were resized into 48x48, since the resolution is large compared to the datasets shown above and this dataset requires far more time and resource to compute.
6.1. M ODEL
In this experiment, we compared two models, DropConnect and Adaptive DropConnect (proposed model in this paper).
The baseline model is composed of two convolutional layers and two fully connected layers whose outputs are dropped out (we set the possibility as 0.5).
The labels of input data are predicted using log-softmaxed value of last fully connected layer.
In DropConnect model and Adaptive DropConnect model, ﬁrst fully connected layer is replaced by DropConnected layer and Adaptive DropConnected layer respectively.
Note that DropConnect model corresponds to the our method with " = 1.0 and this means agents do not perform their auctions, and randomly mask the weights.
6.1. R ESULTS
For the MNIST datasets, the models are trained for 10 epochs and then evaluated with the test data.
The numbers of epochs for CIFAR-10 and STL-10 are 20 and 40 respectively.
Experiments are repeated 20 times for each condition, and the average and standard deviation of error rate was calculated.
The results is shown in Table 1.
As expected, with the model using Adaptive DropConnect, the classiﬁcation error rate was lower than both the baseline and DropConnect regardless of the datasets given in this experiment. [Table] DropConnect (Wan et al.,
[Table]
6. S INGLE - AGENT RL
Next, we set the single-agent reinforcement learning task.
We used the CartPole task from OpenAI gym with visual input.
In this setting, the agent must balance a pole while moving a cart.
There is much non-useful information related to the image.
For that reason, pruning the pixels is important.
The result in Table 1 demonstrates that our method improves the standard RL.
6. M ULTI - AGENT RL
The additional feature of NaaA is credit assignment for reward distribution, meaning that if the neural network is divided into multiple agents, it works by playing the auction game.
We conﬁrmed that additional agents complement the main player using ViZDoom, an environment for Doom.
A player in Doom environment should seek the enemy in the map, and then defeat the enemy.
Because
ViZDoom provides several maps, we used ViZDoom.
6.3. S ETUP
We used a scenario based on Defend the Center (DtC), provided by ViZDoom platform.
In DtC, players are placed in the center of a ﬁeld of circle.
They attack enemies that come from the wall.
The game has two players: a main player and a cameraman.
Alhough the main player can attack the enemy with bullets, the cameraman has no way to attack, and only scouts for the enemy.
The action space for the main player is the combination of f attack, turn left, turn right g .
Therefore, the total number of actions is 2 3 = 8 .
The cameraman has two possible actions: f turn left, turn right g .
Although the players can only change direction, they cannot move on the ﬁeld.
The enemy will die if have the attack (bullet) from the main player once, then player receives +1.
As a default on an episode, the ammunition amount is 26.
The main player will die if under attack from the enemy to the extent that health becomes 0, then the player receives -1.
The cameraman will not die if attacked by the enemy.
The episode will terminate when the maim player dies, or after 525 steps have elapsed.
6.3. M ODEL
We compared three models: the proposed method and two comparison targets.
Baseline DQN without communication.
The main player learns standard DQN with the perspective that the player is viewing.
Because the cameraman does not learn, the player continues to move randomly.
Comm DQN with communication.
The main player learns DQN with two perspectives: the player’s own and the cameraman’s.
The communication vector is learned with a feed-forward neural network.
The method is inspired by Commnet.
NaaA
The proposed method.
The main player learns DQN with two perspectives: the player’s own and the cameraman’s.
The transmission of reward and communication are performed using the proposed method.
6.3. R ESULTS
Training is performed in 10 million steps.
Figure 2 Left presents that our model NaaA outperforms two methods.
Improvement is achieved by Adaptive DropConnect.
We conﬁrmed that the cameraman sees the enemy through an episode.
This can be interpreted as the cameraman reporting the enemy position.
In addition to seeing the enemy, the cameraman sees the area behind of main player several times.
This action enables the cameraman to observe attacks from the enemy while seizing a better relative position.
For further interpretation of the result, we present visualization of the revenue that the agent earned in Figure
Right as a heatmap.
The background picture is a screen in Doom taken at the moment when the ﬁlter in CNN is mostly activated.
Figure 3 shows an example of learnt sequence of actions by our method.
D ISCUSSION
Regarding the optimization method, although envy-free auction guarantees truthfulness if the buyer prices are sealed, in cases where buyers can mutually communicate and share price information, the buyer can fake the price with lower demand in a process of collusion.
To address the issue, several solutions such as random sample auction Goldberg et al.
(2006) are proposed.
NaaA is applicable to learning distributed environments on a computer network such as a peer-topeer network, and controlling the sub-modules of robots such as multiple cameras.
Speciﬁcally, it is applicable to various methods as described below.
• Hyperparameter tuning.
Several algorithms have been proposed such as neuroevolution using genetic algorithms.
In the case, proﬁt or counterfactual return is useful for a ﬁtness function.
• Pruning.
Computing costs can be reduced by downsizing a neural network.
• Attention control.
Research of attention is using reinforcement learning to control attention.
• Ensemble.
Our method is applicable to mixed multiple models.
These applications illustrate the direction of our research.
C ONCLUSION AND F UTURE W ORKS
This paper proposed NaaA, a reinforcement learning framework that treats each unit on a neural network as an agent.
First, we pointed out there are dilemma problems if we naively optimize NaaA.
We proposed an optimization method with auction.
Consequently, an action by which units evaluate the counterfactual return of other units is obtained as a Nash equilibrium.
Furthermore, we proposed
Q -learning based algorithm, adaptive dropconnect, to optimize the neural network topology dynamically with evaluation of counterfactual return.
For the evaluation, we performed experiments based on single-agent and multi-agent platforms, demonstrating that our experimentally obtained results improve existing methods.
As a direction of future research we use on-policy methods to perform adaptive dropconnect, and consider applications combining genetic algorithms
A PPENDIX A. P ROOF OF T HEOREM 5.
As for a buyer, the asking price q for a seller is unknown, we address q which has support [0 ; 1 ) , and consideration to maximize E q [ G ( b; q )] , In this case, the following equation holds.
[Equation]
Therefore, the condition to maximize E q [ G ( b; q )] is b = v .