Competitive Multiagent Deep Reinforcement Learning with Counterfactual Thinking Counterfactual thinking describes a psychological phenomenon that people reinfer the possible results with different solutions about things that have already happened. It helps people to gain more experience from mistakes and thus to perform better in similar future tasks. This paper investigates the counterfactual thinking for agents to find optimal decisionmaking strategies in multiagent reinforcement learning environments. In particular, we propose a multiagent deep reinforcement learning model with a structure which mimics the humanpsychological counterfactual thinking process to improve the competitive abilities for agents. To this end, our model generates several possible actions (intent actions) with a parallel policy structure and estimates the rewards and regrets for these intent actions based on its current understanding of the environment. Our model incorporates a scenariobased framework to link the estimated regrets with its inner policies. During the iterations, our model updates the parallel policies and the corresponding scenariobased regrets for agents simultaneously. To verify the effectiveness of our proposed model, we conduct extensive experiments. Experimental results show that counterfactual thinking can actually benefit the agents to obtain more accumulative rewards from the environments with fair information by comparing to their opponents.

The Effect of Different Types of Internal Rewards in Distributed MultiAgent Deep Reinforcement Learning* Distributed multiagent reinforcement learning in the same environment is prohibitively hard, due to the difficulty of assigning credit for the individual actions of the agent, especially when the agent is a member of a team. Meanwhile, the sparse delayed reward about the team from the environment such as winning makes the learning progress more challenging. To solve the credit assignment and sparse delayed reward problems which are common in multiagent reinforcement learning, researchers usually construct or learn an internal reward signal that acts as a proxy for winning and provides denser rewards for individual agent. To improve the learning effect of a typical multiagent learning task, we conducted three types of internal rewards for multiagent team members and evaluated the effect of these rewards. The results show that not all internal reward can improve the learning effect of multiagent reinforcement learning, it seems that when the learning task is not very complex and the time of finishing the task for the team is not very long, the sparse reward such as winning have the best learning effect, and the learning effect of the other two forms of reward is not as good as that of the simple sparse reward. To some extent, our research results can provide a reference for the design of reward function in the application of distributed multiagent reinforcement learning.