MultiHand Direct Manipulation of Complex Constrained Virtual Objects We present a method to manipulate virtual objects which are constrained complexly and reconfigurable as if they would exist in a real world by using multiple hands simultaneously. A complexly constrained, and reconfigurable object, such as a Rubikxe2x80x99s cube, is hard to describe its physical motion constraints, mainly because they are determined by the grasping situation and dynamically changeable. Rather than describing the physical motion constraints in a general form, we more focus on the multiple hand interaction of a complex object. A complex object is divided into multiple subparts which are grasped by each hand, and the constraints between the subparts are optimized for inducing natural and continuous movement. For this, we propose a dynamically adjustable data structure for representing object parts grasped by multiple hands, and an optimizationbased pose estimation of the constrained subparts along with their grasped hands. The experiments show that human subjects can manipulate a complexly constrained object such as a Rubikxe2x80x99s cube without any difficulty as if it exists in the realworld.

Learning to Solve a Rubikxe2x80x99s Cube with a Dexterous Hand We present a learningbased approach to solving a Rubikxe2x80x99s cube with a multifingered dexterous hand. Despite the promising performance of dexterous inhand manipulation, solving complex tasks which involve multiple steps and diverse internal object structure has remained an important, yet challenging task. In this paper, we tackle this challenge with a hierarchical deep reinforcement learning method, which separates planning and manipulation. A modelbased cube solver finds an optimal move sequence for restoring the cube and a modelfree cube operator controls all five fingers to execute each move step by step. To train our models, we build a highfidelity simulator which manipulates a Rubikxe2x80x99s Cube, an object containing highdimensional state space, with a 24DoF robot hand. Extensive experiments on 1400 randomly scrambled Rubikxe2x80x99s cubes demonstrate the effectiveness of our method, achieving an average success rate of 90.3%. To benefit the community, we make our simulator and source code publicly available to advance future research1.