Cooperative AI Workshop at Neurips 2021
Tags: datascience
Published: 2022-01-24 17:11:46

---
title: Cooperative AI Workshop at Neurips 2021
published: true
date: 2021-12-22 15:39:31 UTC
tags: DataScience
canonical_url: https://apiumhub.com/tech-blog-barcelona/cooperative-ai-workshop-at-neurips-2021/
---

<section data-id="49fb5de" data-element_type="section">
						<div>
							<div>
					<div data-id="3044692" data-element_type="column">
			<div>
							<div>
						<div data-id="ce021f6" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">This initiative appeared for the first time last year with the same name at Neurips 2020. The high level perspective paper of </span><b>Open Problems in Cooperative AI presented on the website </b><span style="font-weight: 400;">offered a general overview of the main challenges that this field presented as a rising</span> <span style="font-weight: 400;">field that include the intersection of other areas defined as </span><b>AI research aiming for helping individuals, humans and machines, to find ways to improve their joint welfare</b><span style="font-weight: 400;">.  It connects clusters of research such as multi-agent AI, game theory and strategic interaction.</span></p>
<p><span style="font-weight: 400;">It will evolve in terms of the dimensions of cooperative opportunities, and the paper posted three main dimensions: </span><b>Understanding</b><span style="font-weight: 400;"> of other agents, their beliefs, incentives and capabilities. </span><b>Communication</b><span style="font-weight: 400;"> between agents, including building a shared language and overcoming mistrust and deception and Constructing cooperative </span><b>commitments</b><span style="font-weight: 400;">, so as to overcome incentive to renege on a cooperative arrangement.</span></p>
<p><span style="font-weight: 400;">This year, a wide range of applications appeared, from banking to peer-review, autonomous driving or even fake-news domain </span></p>
<p><span style="font-weight: 400;">Today we will take a deep dive into some of the </span><i><span style="font-weight: 400;">hints coming from the poster session</span></i><span style="font-weight: 400;">; we would like to thank the authors for their work and efforts, and the organizers for putting all these together. You can find all the papers listed </span><a href="https://www.cooperativeai.com/neurips-2021/workshop-papers" rel="nofollow noopener" target="_blank"><span style="font-weight: 400;">here</span></a><span style="font-weight: 400;">.</span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="4bf1c52" data-element_type="section">
						<div>
							<div>
					<div data-id="d879fb0" data-element_type="column">
			<div>
							<div>
						<div data-id="50b9598" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><span id="docs-internal-guid-1fcc7940-7fff-22cd-09ab-dd2e52549fe1"><span style="font-size: 11pt; font-family: Avenir, sans-serif; color: #000000; background-color: transparent; font-weight: bold; font-variant-numeric: normal; font-variant-east-asian: normal; vertical-align: baseline; white-space: pre-wrap;">Towards Incorporating Rich Social Interactions Into MDPs</span></span></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="15188fc" data-element_type="section">
						<div>
							<div>
					<div data-id="eff9793" data-element_type="column">
			<div>
							<div>
						<div data-id="14d47fd" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Authors: Ravi Tejwani et al.</span><span style="font-weight: 400;"> </span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="28e7f8c" data-element_type="section">
						<div>
							<div>
					<div data-id="52074ba" data-element_type="column">
			<div>
							<div>
						<div data-id="95930dd" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">The interesting research idea coming from this paper aims to extend a nested MDP (Markov Decision Processes coming from Reinforcement Learning), proposing a </span><b>SOCIAL MDP</b><span style="font-weight: 400;">,  where agents reason about other’s arbitrary function rewards. They estimate a  linear reward function for each agent  at each time step. It introduced in the well know tuple two main terms : ξ (estimated reward for other agents at each timestep not at the end of each episode) and g (it can be another goal). </span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="44ea6b3" data-element_type="section">
						<div>
							<div>
					<div data-id="12c1832" data-element_type="column">
			<div>
							<div>
						<div data-id="e7ce056" data-element_type="widget" data-widget_type="html.default">
				<div>
			 <div>
    	<img src="https://apiumhub.com/wp-content/uploads/2021/12/workshop_Fig1_1.png" alt="Workshop Neurips 1.1" title="Cooperative AI Workshop at Neurips 2021 8">
        <img src="https://apiumhub.com/wp-content/uploads/2021/12/workshop_Fig1_2.png" alt=" Workshop Neurips 1.2" title="Cooperative AI Workshop at Neurips 2021 9">
    </div> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="8b1c444" data-element_type="section">
						<div>
							<div>
					<div data-id="5b5a8cb" data-element_type="column">
			<div>
							<div>
						<div data-id="8d32acb" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Fig1.  Decomposition of each agent´s reward at each timestep into a reward coming from the goal, reward coming from the social goal ( depending of each agent´s goal and </span><span style="font-weight: 400;">ξ ) </span><span style="font-weight: 400;">minus the cost of action. </span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="2178c0d" data-element_type="section">
						<div>
							<div>
					<div data-id="61cee8a" data-element_type="column">
			<div>
							<div>
						<div data-id="3e98a6f" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Ambiguity Can Compensate for Semantic Differences in Human-AI communication</b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="335752c" data-element_type="section">
						<div>
							<div>
					<div data-id="0af5518" data-element_type="column">
			<div>
							<div>
						<div data-id="de4667c" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Authors: Ozgecan Kocak, Sanghyun Park, Phanish Puraman</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="4e62bc6" data-element_type="section">
						<div>
							<div>
					<div data-id="dfbdc45" data-element_type="column">
			<div>
							<div>
						<div data-id="70afdfc" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">This paper treats ambiguity and semantic differences as communication difficulties between a human and AI . The main idea is to decouple the semantic structure of the word and give importance to the stimuli ( what the agents are seeing ) and measure ambiguity through a delta normalized parameter . It measures the semantic difference having a normalized parameter δ within a matrix structure.</span></p>
<p><span style="font-weight: 400;">A key point of the algorithm is to iterate until agent codes (language) converge, it means they understand each other. An interesting point they describe in the paper is that agents with less ambiguous language need longer periods of unlearning to lower their prior associations before exploring associations to find a match with the other agent.</span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="8e3b739" data-element_type="section">
						<div>
							<div>
					<div data-id="9cfb6ff" data-element_type="column">
			<div>
							<div>
						<div data-id="6c0b3af" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Promoting Resilience in Multi-Agent Reinforcement Learning via Confusion-Based Communication</b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="77ae0cc" data-element_type="section">
						<div>
							<div>
					<div data-id="69380c3" data-element_type="column">
			<div>
							<div>
						<div data-id="728920d" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Authors: Ozgecan Kocak, Sanghyun Park, Phanish Puraman</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="561db8b" data-element_type="section">
						<div>
							<div>
					<div data-id="cee972d" data-element_type="column">
			<div>
							<div>
						<div data-id="39abd74" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">Part of a larger agenda, exploring the key concept of cooperation and adaptation to perturbation (resilience). In their paper, they explore an innovative manner to measure group resilience based on distance between original MDP M and modified MDP M’ and defined on how to collaborate effectively, maintaining performance while adapting to perturbation.  </span></p>
<p> </p>
<p><span style="font-weight: 400;">In order to achieve collaboration between agents they introduce the confusion-based communication which gets a better performance improving the group’s resilience. In these messages they send what they defined as confusion level. This measure they have to minimize which means that they are being resilience.</span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="7cdc5a4" data-element_type="section">
						<div>
							<div>
					<div data-id="a0b3dd7" data-element_type="column">
			<div>
							<div>
						<div data-id="675cb17" data-element_type="widget" data-widget_type="image.default">
				<div>
					<div>
										<img width="580" height="274" src="https://apiumhub.com/wp-content/uploads/2021/12/Formula_2.png" alt="Formula 2" loading="lazy" title="Cooperative AI Workshop at Neurips 2021 10"> </div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="161ca5c" data-element_type="section">
						<div>
							<div>
					<div data-id="0454730" data-element_type="column">
			<div>
							<div>
						<div data-id="270a5a6" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Fig2.  The level of confusion of agent p at s</span><span style="font-weight: 400;">i</span><span style="font-weight: 400;"> after taking action a</span><span style="font-weight: 400;">i</span><span style="font-weight: 400;">, denoted J</span><span style="font-weight: 400;">si</span><span style="font-weight: 400;">,a</span><span style="font-weight: 400;">i</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="ef50fbf" data-element_type="section">
						<div>
							<div>
					<div data-id="2cf70ce" data-element_type="column">
			<div>
							<div>
						<div data-id="63dfcd4" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Locality Matters : A scalable value decomposition approach for multi-agent reinforcement learning. </b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="46f8a7b" data-element_type="section">
						<div>
							<div>
					<div data-id="26aa1d4" data-element_type="column">
			<div>
							<div>
						<div data-id="72e1002" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><span style="font-weight: 400;">Authors: Roy Zohar, Shie Mannor, Guy Tenneholtz</span></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="2e34f8f" data-element_type="section">
						<div>
							<div>
					<div data-id="505fc60" data-element_type="column">
			<div>
							<div>
						<div data-id="2dd076a" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">This paper explores the experiments of Decoupling the idea of global reward for cooperation into local rewards, with a graph-based structure. Besides, they propose the LOMAQ algorithm , heavily based on an iteration of a well known Q-mix. One of the main ideas is that the Q-function can be maximized using a partition of partial maximizers.</span></p>
<p><span style="font-weight: 400;">Another interesting point that they proposed is, as we mentioned before, a method for decomposing the global reward function into local reward functions using a deep neural network.</span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="3ea6880" data-element_type="section">
						<div>
							<div>
					<div data-id="c2bd092" data-element_type="column">
			<div>
							<div>
						<div data-id="06cf83f" data-element_type="widget" data-widget_type="html.default">
				<div>
			<div>
    	<img src="https://apiumhub.com/wp-content/uploads/2021/12/workshop_Fig2_1.png" alt="Workshop Neurips 2.1" title="Cooperative AI Workshop at Neurips 2021 11">
        <img src="https://apiumhub.com/wp-content/uploads/2021/12/workshop_Fig2_2.png" alt=" Workshop Neurips 2.2" title="Cooperative AI Workshop at Neurips 2021 12">
    </div> </div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="2382221" data-element_type="section">
						<div>
							<div>
					<div data-id="8058d8e" data-element_type="column">
			<div>
							<div>
						<div data-id="b9b04e3" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Fig 3. A visualization of training MARL for a graph of  8 agents . The colored regions represent the feedback that the agents exhibit during training</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="d114802" data-element_type="section">
						<div>
							<div>
					<div data-id="9d3e629" data-element_type="column">
			<div>
							<div>
						<div data-id="c45ab51" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Disinformation, Stochastic Harm, and Costly Effort: A Principal-Agent Analysis of regulating social Media Platforms</b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="a3b610d" data-element_type="section">
						<div>
							<div>
					<div data-id="9f62ece" data-element_type="column">
			<div>
							<div>
						<div data-id="e43c05b" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><span style="font-weight: 400;">Authors: Shehroze Khan, James R. Wright</span></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="cb799d2" data-element_type="section">
						<div>
							<div>
					<div data-id="3618669" data-element_type="column">
			<div>
							<div>
						<div data-id="3548899" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">Interesting exploration of social media platforms in case of use for misinformation in which it is defined as a MDP with Social platform as the agent and the presence of the Regulator as an entity with the purpose of maximizing social-welfare . This regulator figure presence reminds of the institution figure posted in the Open Problems paper.  </span></p>
<p><span style="font-weight: 400;">One of the interesting key concepts here is the introduction of the metric </span><i><span style="font-weight: 400;">e</span></i><span style="font-weight: 400;"> as the effort and the reward as c(e) cost of the effort, with the overall goal of minimizing the cost of the given effort by the platform. They also explore other concepts such as harm as a function. </span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="3202c2e" data-element_type="section">
						<div>
							<div>
					<div data-id="34a5d12" data-element_type="column">
			<div>
							<div>
						<div data-id="74d9d2e" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Multi-lingual agents through multi-headed neural networks</b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="4e90c99" data-element_type="section">
						<div>
							<div>
					<div data-id="938aa57" data-element_type="column">
			<div>
							<div>
						<div data-id="cef0b62" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><span style="font-weight: 400;">Authors: J.D. Thomas, R. Santos-Rodriguez, R. Piechocki, M. Anca</span></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="ce85aea" data-element_type="section">
						<div>
							<div>
					<div data-id="ba5fc69" data-element_type="column">
			<div>
							<div>
						<div data-id="cdce037" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">This paper explores the idea of catastrophic forgetting during communication in agents. The poster is very neat and introduces an interesting Communication Carousel game in which, during training and after E episodes, a rotative protocol is executed to reinforce the message of one agent to other partners in order to avoid catastrophic forgetting.</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="fadd461" data-element_type="section">
						<div>
							<div>
					<div data-id="3c7f15f" data-element_type="column">
			<div>
							<div>
						<div data-id="f707306" data-element_type="widget" data-widget_type="image.default">
				<div>
					<div>
										<img width="884" height="558" src="https://apiumhub.com/wp-content/uploads/2021/12/Fig4.png" alt="Fig4" loading="lazy" title="Cooperative AI Workshop at Neurips 2021 13"> </div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="5051f76" data-element_type="section">
						<div>
							<div>
					<div data-id="9128376" data-element_type="column">
			<div>
							<div>
						<div data-id="83c0a48" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Fig 4. Communication Carousel. Illustration of the N-paralel referential games. After E episodes, the carousel rotates, and all agents interact with a different partner. This continues for the desired number of rotations after which all agents are returned to their original partner for assessment of emergent language maintenance.</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="f8daa21" data-element_type="section">
						<div>
							<div>
					<div data-id="70fa465" data-element_type="column">
			<div>
							<div>
						<div data-id="d5b4ecb" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><ul><li><b>Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria</b></li></ul></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="7f64641" data-element_type="section">
						<div>
							<div>
					<div data-id="fec248c" data-element_type="column">
			<div>
							<div>
						<div data-id="ac7bd45" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><span style="font-weight: 400;">Authors: Kavya Kopparapu et al.</span></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="a3c3988" data-element_type="section">
						<div>
							<div>
					<div data-id="c789c55" data-element_type="column">
			<div>
							<div>
						<div data-id="83a86fc" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div>
<p><span style="font-weight: 400;">They present Hidden Agenda, a two-team social deduction game that provides a 2D environment for studying learning agents in scenarios of unknown team alignment strategies. The environment admits a rich set of strategies for both teams.</span></p>
<p><span style="font-weight: 400;">The play is based on two teams, the first one is made up of four crewmates (numerical advantage) and one impostor (information advantage, known all players’s roles) and it has the following actions and rewards:</span></p>
</div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="cb283e1" data-element_type="section">
						<div>
							<div>
					<div data-id="1ca2092" data-element_type="column">
			<div>
							<div>
						<div data-id="fdbd605" data-element_type="widget" data-widget_type="image.default">
				<div>
					<div>
										<img width="1024" height="344" src="https://apiumhub.com/wp-content/uploads/2021/12/Fig_5.png" alt="Fig 5" loading="lazy" title="Cooperative AI Workshop at Neurips 2021 14"> </div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="b9500c7" data-element_type="section">
						<div>
							<div>
					<div data-id="37fb5e3" data-element_type="column">
			<div>
							<div>
						<div data-id="e7cd4ad" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Fig 5. In the hidden agenda environment, both teams share the</span><b> same action space</b><span style="font-weight: 400;"> divided into Situation and Voting : in the situation action space, they are allowed to move in 4 cardinal directions and turn. In the case of the impostor, it can also fire a freeze beam. In the voting phase, all players vote in order to discover who the impostor is. </span><b>The Reward space</b><span style="font-weight: 400;"> goes both for a team-based reward at the end of each episode and an agent reward for completing certain tasks.</span></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section><section data-id="9c4d513" data-element_type="section">
						<div>
							<div>
					<div data-id="66f30f3" data-element_type="column">
			<div>
							<div>
						<div data-id="e494243" data-element_type="widget" data-widget_type="text-editor.default">
				<div>
					<div><p><span style="font-weight: 400;">Even though Cooperative AI has taken part as a Neurips workshop, it takes part of a broader initiative to push de boundaries and progress of Cooperative Agents in their different forms, aiming to build the infrastructure of the field. To know more about their foundations, don´t hesitate to visit their </span><a href="https://www.cooperativeai.com/foundation" rel="nofollow noopener" target="_blank"><span style="font-weight: 400;">website.</span></a></p></div>
				</div>
				</div>
						</div>
					</div>
		</div>
								</div>
					</div>
		</section>