Semi-Supervised Policy Initialization for Playing Games with Language Hints (2021)

Abstract: Using natural language as a hint can supply an additional reward for playing sparse-reward games. Achieving a goal should involve several different hints, while the given hints are usually incomplete. Those unmentioned latent hints still rely on the sparse reward signal, and make the learning process difficult. In this paper, we propose semi-supervised initialization (SSI) that allows the agent to learn from various possible hints before training under different tasks. Experiments show that SSI not only helps to learn faster ( 1.2x ) but also has a higher success rate ( 11% relative improvement) of the final policy.

## introduction most reinforcement learning (rl) methods @xcite rely on an agent to explore and maximize the feedback reward. since designing a reward for each step is impractical, a common setting is only to give out the achieved signal. in atari grand challenge @xcite , only if achieving the goal, the environmental reward is 1. however, this sparse-reward setting makes the agent difficult to learn @xcite . extlang @xcite incorporates language hints as an additional reward to overcome the sparse-reward issue. they first build 45 different tasks under montezuma's revenge game, where each task consists of a starting state, an unknown goal position, and a given hint. they also collect demo clips, which are partial playing records (states and actions) that each corresponds to a hint, as shown in fig. 1 . to provide an additional reward, extlang pre-trains a reward module to reflect the relevance between agent actions and the given hint when exploring a task. in this way, they can supply a hint reward instead of only the sparse environmental reward to make the agent easier to explore. though providing an additional reward, the hints are usually incomplete @xcite . considering a task in fig. 1 , to achieve the goal, figure 1 : for a task with hint "jump over the skull", both "climb up the ladder" and "jump to get the key" are useful latent hints and can be learned during ssi. the agent should "jump over the skull," "climb up the ladder," and "jump to get the key." however, the given hint only contains the first one, and learning those latent hints still relies on the sparse environmental reward. to deal with this issue, we propose semi-supervised initialization (ssi) that enables the agent to experience various possible hints in advance. we adopt a hint module to generate possible hints for random states and allow the agent to learn from them. with ssi, agents have a better-initialized policy during training for each task. from another point of view, in this paper, we propose a semi-supervised initialization method and investigate the abilities of nlp on controlling complex actions in game environments @xcite . we perform ssi first and train-evaluate on those tasks built from extlang. experimental results show that with ssi, better-initialized policy not only learns faster but also has a higher success rate. hint module (h) h generates a possible hint l for a state s. h adopts cnn to extract the visual feature v of s and attention-based @xcite gru @xcite as the decoder to produce a series of words w as a hint l: ## conclusion and ethical considerations in this paper, we propose semi-supervised initialization that makes the agent learn from various possible hints in advance before play games with language hint. by semi-supervised initialization, the agent can have a better-initialization policy, which benefits further task-training. the experiments show that semi-supervised initialization not only helps the agent to learn faster but also has a higher success rate of the final policy. our presented ssi can benefit future vision-and-language research for practical applications. in terms of negative impact, since the initialization is learned from those instructions, if there is bias in the original dataset, it may have some potential issues.