Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021
Tags: datascience
Published: 2021-12-08 11:00:37

---
title: Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021
published: true
date: 2021-12-03 13:50:33 UTC
tags: DataScience
canonical_url: https://apiumhub.com/tech-blog-barcelona/cooperative-multi-agent-reinforcement-learning-qmix-neurips-2021/
---

<section data-id="8b6431a" data-element_type="section">
						<div>
							<div>
					<div data-id="8a197b0" data-element_type="column">
			<div>
							<div>
						<div data-id="43af608" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p style="font-size: 16px; color: #222222; font-style: normal;"><span style="font-weight: bold;">Authors: </span>Gema Parreño, David Suarez<span style="font-weight: bold;"> (Apiumhub). </span></p>
<p style="font-size: 16px; color: #222222; font-style: normal;"><span style="font-weight: bold;">Thanks to: </span>Alberto Hernandez<span style="font-weight: bold;"> (BBVA Innovation Labs)</span></p>
<p style="font-size: 16px; color: #222222; font-style: normal;">The following blogspot aims to do an introduction about Cooperative MARL  and goes through innovations by S. Whiterson Lab, with QMIX (2019)  their current contributions for #<span style="font-weight: bold;">Neurips2021</span>. Going through this article might imply to have certain fundamentals about Reinforcement Learning.</p>
<p style="font-size: 16px; color: #222222; font-style: normal;"> A multi-agent system describes multiple distributed entities—so-called agents—which take decisions autonomously and interact within a shared environment (Weiss 1999). <span style="font-weight: bold;">MARL</span> (Multi-Agent Reinforcement Learning) can be understood as a field related to RL in which a <em>system of agents that interact within an environment to achieve a goal. </em>The Goal of each one of these agents or learnable units is to learn a policy in order to maximize the long term reward, in which each agent discovers a strategy alongside other entities in a common environment and adapts its policy in response to the behavioural changes of others.</p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="fcd2975" data-element_type="section">
						<div>
							<div>
					<div data-id="3acd43a" data-element_type="column">
			<div>
							<div>
						<div data-id="236fdfc" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>Taxonomy</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="f40390a" data-element_type="section">
						<div>
							<div>
					<div data-id="b8434bd" data-element_type="column">
			<div>
							<div>
						<div data-id="8940958" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p>Properties of MARL systems that are key to their modeling and depending on these properties we might be branching into specific particularities of areas of research.<br></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="92b6836" data-element_type="section">
						<div>
							<div>
					<div data-id="fabab31" data-element_type="column">
			<div>
							<div>
						<div data-id="c0032f5" data-element_type="widget" data-widget_type="htmega-datatable-addons.default">
				<div>
			        <div>
            <table>
                                    <thead>
                        <tr>
                            <th>TRAINING (Control of training and execution) </th>
<th>INFORMATION AVAILABLE (What the agents have about other agents)</th>
<th>ENVIRONMENT (Conditions of the environment)</th>
<th>REWARD STRUCTURE (System Behaviour)</th> </tr>
                    </thead>
                                <tbody>
                                            <tr>
                                                            <td>
<strong>Centralized</strong> <br> A central unit takes the decision for each agent in each time step. Policies are updated based on exchange of information during training.</td>
                                                            <td>
<strong>Independent Learner</strong><br> Ignore other’s existence. No rewards &amp; actions info from other’s.</td>
                                                            <td>
<strong>Fully observable</strong><br> The agents are able to access the whole information and the sensory information.</td>
                                                            <td>
<strong>Cooperative</strong><br> Same reward to all agents. They cooperate to achieve a common goal and avoid individual failure.</td>
                                                    </tr>
                                            <tr>
                                                            <td>
<strong>Decentralized</strong><br> Agents make a decision for themselves. Each agent performs updates on its own and develops an individual policy without utilizing foreign information.</td>
                                                            <td>
<strong>Joint-Action Learner</strong><br> Observe actions from other’s a-posteriori.</td>
                                                            <td>
<strong>Partially observable</strong><br> The agent is only able to observe its local information.</td>
                                                            <td>
<strong>Competitive</strong><br> Sumatory of reward is equal to zero. The agents compete against each other maximizing their own reward and minimizing others.</td>
                                                    </tr>
                                    </tbody>
            </table>
        </div>
                		</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="291996b" data-element_type="section">
						<div>
							<div>
					<div data-id="50b9fc8" data-element_type="column">
			<div>
							<div>
						<div data-id="4bb4006" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p>Table 1. This taxonomic schema ( Weiss 1999) proposes to let us know more about the MARL exploration we will talk about today. In cooperative MARL, <em>agents cooperate to achieve a common goal.</em></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="59331a5" data-element_type="section">
						<div>
							<div>
					<div data-id="67f916d" data-element_type="column">
			<div>
							<div>
						<div data-id="8d99c76" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>Challenges</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="dc46c3e" data-element_type="section">
						<div>
							<div>
					<div data-id="0645ccf" data-element_type="column">
			<div>
							<div>
						<div data-id="5e67ca0" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><strong style="color: inherit; font-size: inherit; font-style: inherit;">From the environment perspective, we can enunciate several challenges :</strong></p>
<ul>
<li>
<strong>Non-stationarity: </strong>A single agent faces a moving target problem when the transition probability function changes.</li>
<li>
<strong>Credit assignment problem: </strong>Agent can’t know the impact of its own action towards the team’s success.</li>
<li>The reality of <strong>Partial Observable environment</strong>: Partially Observable Markov Decision Process (POMDP). Most real-world cases of uses and applications are based on Partially observable environments.</li>
</ul>
<p>When we branch from MARL into Cooperative MARL , we focus on reformulating the challenge into a system of agents that interact within an environment to achieve a <em>common</em> goal. These challenges might have more importance depending on the type of behaviour and environment. From the conceptual challenges derived from the agent interaction and performance perspective inside cooperation we can think of the following derived from:</p>
<ul>
<li>
<strong>Coordination: </strong>Accomplishing a joint goal in cooperative settings requires agents to agree on a consensus.</li>
<li>
<strong>Communication: </strong>The learning of meaningful communication protocols in cooperative tasks.</li>
<li>
<strong>Commitment: </strong>Constructing cooperative commitments, so as to overcome incentives to neglect  a cooperative arrangement.</li>
<li>
<strong>Scalability: </strong>MARL algorithms are hard to train: the potentially high number of agents and heterogeneous action space entails a linear growth of computational effort.</li>
</ul>
<p>From now on, we will focus on centralized cooperative MARL and QMIX definition , notation and description.</p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="15dd9a2" data-element_type="section">
						<div>
							<div>
					<div data-id="9cbac3a" data-element_type="column">
			<div>
							<div>
						<div data-id="acc85a4" data-element_type="widget" data-widget_type="html.default">
				<div>
			    <div>
    	<img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig1_1.png" alt="Neurips 1.1" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 7">
        <img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig1_2.png" alt="Neurips 1.2" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 8">
    </div> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="a48d5f4" data-element_type="section">
						<div>
							<div>
					<div data-id="1779a5e" data-element_type="column">
			<div>
							<div>
						<div data-id="d8582ad" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p>Fig 1. Visual representation of MARL properties with some challenges regarding the taxonomy. The zoom area includes areas inside <strong>Cooperative AI posted in Open Problems in Cooperative AI </strong>and<strong> Q-MIX papers</strong>.</p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="7526d06" data-element_type="section">
						<div>
							<div>
					<div data-id="c987d2d" data-element_type="column">
			<div>
							<div>
						<div data-id="bee5ec0" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>Centralized Cooperative Multi-Agent</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="82e04a1" data-element_type="section">
						<div>
							<div>
					<div data-id="5b8d9a5" data-element_type="column">
			<div>
							<div>
						<div data-id="23704bf" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h4>Centralized Cooperative Multi-Agent RL Notations and Formulation for the coordination problem
</h4> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="11815ab" data-element_type="section">
						<div>
							<div>
					<div data-id="5e9a337" data-element_type="column">
			<div>
							<div>
						<div data-id="7130167" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<div>Regarding notation, the main differences between the notations for RL are that we introduce in the tuple the parameter <strong style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;"><b style="color: #008000;">N</b></strong><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;"> that stands for the number of agents, and </span><span style="font-family: Europa; font-size: inherit; font-style: inherit;"><b style="color: #006400;">O = { O1,…On }</b></span><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;"> that is the set of observations for all agents (if different agents have different set of observations, all of them might be represented in this set) and the same happens for the set of observations </span><span style="font-family: Europa; font-size: inherit; font-style: inherit;"><b style="color:#4B0082">U = { U<sub>1 </sub>…U<sub>n </sub>}</b></span><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;"> stands for the </span><strong style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">joint action set for all agents</strong><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">, meaning that the action will be taken in a cooperative manner even though this shall mean that each agent takes a different action. Therefore , taking into account the tuple &lt; <strong style="color: inherit; font-size: inherit; font-style: inherit;"><b style="color: #006400;">N</b></strong></span><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">, S, U, R, P, O, ץ &gt; </span>
</div>
<ul>
 	<li><b style="color: #008000;">N = {1…N} denotes the set of N&gt;1 interacting Agents</b></li>
 	<li>S is the State space of all agents</li>
 	<li><b style="color: #4b0082;">U = { U<sub>1 </sub>…U<sub>n </sub>} joint action set for all agents or the collection of individual action spaces from N agents</b></li>
 	<li>R is the Reward.</li>
 	<li>P : U x S → P( U ) is the probability distribution of actions</li>
 	<li><b style="color: #006400;">O = { O<sub>1 </sub>…O<sub>n </sub>} set of observations for all agents.</b></li>
 	<li>ץ discount factor [0,1)</li>
</ul>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="12870f4" data-element_type="section">
						<div>
							<div>
					<div data-id="82f974f" data-element_type="column">
			<div>
							<div>
						<div data-id="f5b8f8c" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: bold;">Notation 1</span>. Colored letters set the Differences with respect to the traditional Reinforcement Learning approach. <span style="font-weight: bold;">Notation for a Fully cooperative setup</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="e29afc7" data-element_type="section">
						<div>
							<div>
					<div data-id="80ea1e8" data-element_type="column">
			<div>
							<div>
						<div data-id="3c9eee2" data-element_type="widget" data-widget_type="html.default">
				<div>
			    <div>
    	<img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig2_2.png" alt="Neurips 2.1" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 9">
        <img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig2_1.png" alt="Neurips 2.2" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 10">
    </div> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="26b123e" data-element_type="section">
						<div>
							<div>
					<div data-id="dd18b82" data-element_type="column">
			<div>
							<div>
						<div data-id="553d055" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>Fig 2. Visual representation of a fully cooperative and partially observable multiagent environment Dec-POMDPs. The example takes <a style="font-size: 12px;" href="https://arxiv.org/pdf/1902.04043.pdf" rel="nofollow noopener" target="_blank"><strong>SMAC</strong></a> environment 2c_vs_64zg: at each time step t, the environment sends observations to the agents (2 Colosi) about enemy positions and actions of both enemies and the other agent , and each agent (Colosi) produces an action based on their Qtot value function. All the agents share the same reward.</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="3b08453" data-element_type="section">
						<div>
							<div>
					<div data-id="da9980c" data-element_type="column">
			<div>
							<div>
						<div data-id="5677d76" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="d7c87d3" data-element_type="section">
						<div>
							<div>
					<div data-id="251fd7d" data-element_type="column">
			<div>
							<div>
						<div data-id="ec58883" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>Regarding a<span style="font-weight: bold;"> fully cooperative behavior with centralized learning with decentralized execution</span>, the joint <span style="font-weight: bold;">action-value</span> function <span style="font-weight: bold;"><b style="color:#006400">Q</b></span><span style="font-weight: bold;"><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot</span></span> can be decomposed into <b style="color:#006400">N Q-functions for N agents</b>, in which each <span style="font-weight: bold;"><b style="color:#008000">Q-function Q</b></span><span style="font-weight: bold;"><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;"><b style="color:#008000">i</b></span></span>  measures how good each action is, given a state, for the agents following a policy</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="7b98f85" data-element_type="section">
						<div>
							<div>
					<div data-id="13d4803" data-element_type="column">
			<div>
							<div>
						<div data-id="f438cd3" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p style="font-size: 30px; color: #222222; font-style: normal;"><span font-size: inherit font-style:><b style="color:#006400">Q</b></span><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em; color: inherit; font-style: inherit;"><b style="color:#006400">tot</b></span><span font-size: inherit font-style:><b style="color:#006400">(τ, u)</b> = Σ <b style="color:#008000">Q</b><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em; font-style: inherit;">i </span><span style="color: inherit; font-size: inherit; font-style: inherit;">( τ</span><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em; color: inherit; font-style: inherit;"><b style="color:#4B0082">i </b></span><span style="color: inherit; font-size: inherit; font-style: inherit;">, u</span><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em; color: inherit; font-style: inherit;">i  </span><span style="color: inherit; font-size: inherit; font-style: inherit;">)</span></span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="bd45155" data-element_type="section">
						<div>
							<div>
					<div data-id="1683aa9" data-element_type="column">
			<div>
							<div>
						<div data-id="e3bf149" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul>
 	<li>Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot  </span>-&gt; global action-value function</li>
 	<li>
<b style="color:#008000">Q</b><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span>  -&gt; Action-value function for each one of the agents</li>
 	<li>
<b style="color:#006400">τ</b> -&gt; joint action-observation history</li>
 	<li>
<b style="color:#4B0082">U</b> -&gt; joint action</li>
</ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="e02d219" data-element_type="section">
						<div>
							<div>
					<div data-id="9de88c7" data-element_type="column">
			<div>
							<div>
						<div data-id="4b77846" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><strong style="color: inherit; font-size: inherit; font-style: inherit;">Notation 2 </strong><span style="color: inherit; font-size: inherit; font-style: inherit;">. Global Action-value function as a sum of individual action-value functions, one for each agent.</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="f67ba7f" data-element_type="section">
						<div>
							<div>
					<div data-id="eae53c3" data-element_type="column">
			<div>
							<div>
						<div data-id="421b0ab" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p style="font-size: 16px; color: #222222; font-style: normal;"><span style="color: inherit; font-size: inherit; font-style: inherit;">Q-Mix paper, published in 2018 by T. Rashid et al explores a hybrid value-based multi-agent reinforcement learning method , adding a </span><span style="font-weight: bold; color: inherit; font-size: inherit; font-style: inherit;">constraint </span><span style="color: inherit; font-size: inherit; font-style: inherit;">and a mixing Network structure in order to make the learning stable, faster and ultimately better in a controlled setup.</span></p>
<p style="font-size: 16px; color: #222222; font-style: normal;">As a conceptual key idea for QMIX is to understand <span style="font-weight: bold;">centralized learning</span> ( <b style="color:#006400">Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot</span></b> ) with <span style="font-weight: bold;">decentralized execution paradigm </span>( <b style="color:#008000">Q<span style="font-weight: bold;"><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span></span></b> ), also known as CTDE : agents are trained in a centralized way with access to the overall action-observation history ( <b style="color:#006400">τ</b>) and global state during training , but during execution have access only to their own local action-observation histories ( <b style="color:#006400">τ</b><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;"><b style="color:#4B0082">i</b></span> )</p>
<p style="font-size: 16px; color: #222222; font-style: normal;">One of the main first ideas is to verify a constraint that enforces the monotonicity of the relationship between the <b style="color:#006400">global action-value function Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot </span></b>and the <b style="color:#008000">action-value function of each one of the agents Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span></b> in every action. This constrained action allows each agent to participate in a decentralised execution by <span style="font-weight: bold;">choosing greedy actions</span> with respect to its action value function</p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="d23e9d4" data-element_type="section">
						<div>
							<div>
					<div data-id="6414c3b" data-element_type="column">
			<div>
							<div>
						<div data-id="d2de4b9" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p style="font-size: 30px; color: #222222; font-style: normal;">მ<b style="color:#006400">Q<sub>tot</sub></b>  / მ<b style="color: #008000;">Q<sub>i</sub></b>  ≥ 0, ∀a</p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="bd3978f" data-element_type="section">
						<div>
							<div>
					<div data-id="cd43525" data-element_type="column">
			<div>
							<div>
						<div data-id="c8b7857" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><strong style="color: inherit; font-size: inherit; font-style: inherit;">Notation 3. </strong><span style="color: inherit; font-size: inherit; font-style: inherit;">The Global argmax Action-Value function divided for the argmax Action-Value function of each agent is 0 or higher, for every action</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="cf87068" data-element_type="section">
						<div>
							<div>
					<div data-id="30f9d46" data-element_type="column">
			<div>
							<div>
						<div data-id="726c051" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p style="font-size: 16px; color: #222222; font-style: normal;"><span style="color: inherit; font-size: inherit; font-style: inherit;">This function allows each agent to participate in a decentralized execution by choosing greedy actions with respect to its value function .</span></p>
<p style="font-size: 16px; color: #222222; font-style: normal;">The overall QMIX architecture shows two main differentiated parts :</p>

<ul>
 	<li>
<span style="font-weight: bold;">Agent Networks</span>: for each agent A<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span> , there is an agent Network that represents its action-value function. It receives the current observation and the last action as input at each time step and returns a <b style="color:#008000">Q action-value function Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span></b> . The NN topology is inside the DRQN family that makes use of GRU, as it facilitates the learning over longer timescales and probably converges faster. This means that if we are dealing with an environment with, for example, two colosi agents, we might have.</li>
</ul>
<ul>
 	<li>
<span style="font-weight: bold;">Mixing Network: </span>A feedforward Network that takes the agents outputs (<b style="color:#008000">Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">i</span> for every one of the agents</b>) and outputs the <b style="color:#006400">total Action-value function Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot</span></b> . Inside this architecture we find the creative and innovative part, in which the weights of the Neural Networks are produced by a separate <span style="font-weight: bold;">hypernetwork</span>, meaning that there is a NN that generates the weights for another network. The output of the hypernetwork is then a vector forced to be positive, <span style="font-weight: bold;">making it possible to condition the weights of the monotonicity</span>.</li>
</ul>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="a2cfb82" data-element_type="section">
						<div>
							<div>
					<div data-id="89410de" data-element_type="column">
			<div>
							<div>
						<div data-id="07ea3a5" data-element_type="widget" data-widget_type="html.default">
				<div>
			<div>
    	<img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig3_1.png" alt="Neurips 3.1" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 11">
        <img src="https://apiumhub.com/wp-content/uploads/2021/12/Neurips2021_Fig3_2.png" alt="Neurips 3.2" title="Cooperative Multi-Agent Reinforcement Learning and QMIX at Neurips 2021 12">
    </div> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="e52c9ed" data-element_type="section">
						<div>
							<div>
					<div data-id="f106c5b" data-element_type="column">
			<div>
							<div>
						<div data-id="2333186" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p>Fig 3. Overall architecture of QMix proposed by QMIX paper with the main components: the <strong>mixing network</strong> with the <strong>hypernetwork</strong>, that forces monotonicity and the agent networks.</p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="b57600d" data-element_type="section">
						<div>
							<div>
					<div data-id="a991c47" data-element_type="column">
			<div>
							<div>
						<div data-id="2e55906" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<div><span style="color: inherit; font-size: inherit; font-style: inherit; font-family: Europa;">Key ideas from Q-MIX algorithm :</span></div>
<ul>
<li>Satisfy a condition for choosing a <strong>greedy Action-Value function for each agent</strong>
</li>
<li>Each agent has an <strong>agent Network</strong> that calculates the Action-Value function.</li>
<li>A <strong>Mixing-Network</strong> calculates the weights forced to be positive, based on the states in order to calculate the joint action-value function Q<sub>tot</sub>.</li>
</ul>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="30e93fc" data-element_type="section">
						<div>
							<div>
					<div data-id="5e7e179" data-element_type="column">
			<div>
							<div>
						<div data-id="9b6825e" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>Regularized Softmax Deep Multi-Agent Q-learning at Neurips 2021</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="d48804c" data-element_type="section">
						<div>
							<div>
					<div data-id="9b4e5e4" data-element_type="column">
			<div>
							<div>
						<div data-id="85bca39" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h4>Neurips 2021: Regularized Softmax Deep Multi-Agent Q-learning
</h4> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="95eb512" data-element_type="section">
						<div>
							<div>
					<div data-id="7c8359c" data-element_type="column">
			<div>
							<div>
						<div data-id="502fcd7" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><div>
<div>
<span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">During #Neurips2021, the lab will present the challenge of </span><span style="font-weight: bold; color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">practical severe overestimation </span><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;">Q-MIX presents, proposing a regularization-based update scheme that penalizes large <b style="color:#008000">Q</b></span><span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em; color: inherit; font-family: Europa; font-style: inherit;">tot</span><span style="color: inherit; font-family: Europa; font-size: inherit; font-style: inherit;"> values that stabilizes learning and a softmax operator that reduces overestimation bias.</span>
</div>
<p style="font-size: 16px; color: #222222; font-style: normal;"><span style="font-weight: bold;">Overestimation</span> is an important challenge because it indeed can be accumulated and <span style="font-weight: bold;">be counterproductive for performance of value-based algorithms</span> . Besides, the fact that there are multiple agents inside a MARL scenario derives into the <span style="font-weight: bold;">joint-action space exponentially increasing </span>with the number of agents and this can be considered an issue. In the case of Q-MIX, the overestimation fenomena can not only come from the calculation of <b style="color:#008000">Qi</b> but also from the <span style="font-weight: bold;">mixing network.</span></p>
<p style="font-size: 16px; color: #222222; font-style: normal;">First the paper presents some key experimental results from some mental model to tackle the challenge that didn´t show the desired outcomes: a gradient Regularization of the mixing network and a baseline with Qtot by adding a regularized term to the loss  λ (Qtot(s,u) − b(s,u))<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; top: -0.5em;">2</span>, where they used the mean squared error loss and λ is the regularization coefficient.</p>
<p style="font-size: 16px; color: #222222; font-style: normal;">As the final proposal that showed better empirical results they used a softmax for the <span style="font-weight: bold;">joint action-value function <b style="color:#006400">(softmax(Q<span style="font-size: 12px; line-height: 0; position: relative; vertical-align: baseline; bottom: -0.25em;">tot</span></b></span><span style="font-weight: bold;">(s,u)) </span>with principles from Deep Q-Learning, using the state and not the action-observation history τ as in QMIX Value Decomposition Networks approach.</p>
<p style="font-size: 16px; color: #222222; font-style: normal;">For knowing more about this contribution, don´t hesitate to read their paper <a href="https://arxiv.org/pdf/2103.11883.pdf" rel="nofollow noopener" target="_blank">here</a>.</p>

</div></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="6172ae3" data-element_type="section">
						<div>
							<div>
					<div data-id="d8f97a8" data-element_type="column">
			<div>
							<div>
						<div data-id="b7c31e1" data-element_type="widget" data-widget_type="heading.default">
				<div>
			<h2>References</h2> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="257fa600" data-element_type="section">
						<div>
							<div>
					<div data-id="5bbcd439" data-element_type="column">
			<div>
							<div>
						<div data-id="61e957d0" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ol>
<li>Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence, Weiss (1999)</li>
<li>Review of Multi-Agent Deep Reinforcement Learning based on the work , A. Oroojlooy and D. Hajinezhad (2020)</li>
<li>Open Problems in Cooperative AI , A.Dafoe et al. (2020)</li>
<li>QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning</li>
<li>SMAC The StarCraft Multi-Agent Challenge,  Mikayel Samvelyan et al. (2019)</li>
<li>Regularized Softmax Deep Multi-Agent Q-Learning (2021)</li>
</ol></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section>