Biography

Eiji Uchibe joined Department of Brain Robot Interface, ATR Computational Neuroscience Laboratories in 2015 as Principal Researcher. He received the Ph.D. degree in mechanical engineering from Osaka University in 1999. He worked as a Post-Doctoral Research Associate at Osaka University (1999-2001), Researcher at ATR International (2001-2004), Researcher (2004-2011), and Group Leader (2011-2015) at Okinawa Institute of Science and Technology. His research interests are in reinforcement learning, robotics, evolutionary computation, computational neuroscience, and their applications.

Abstract

Deep inverse reinforcement learning

This paper proposes model-free deep inverse reinforcement learning to find nonlinear reward function structures. We formulate inverse reinforcement learning as a problem of density ratio estimation, and show that the log of the ratio between an optimal state transition and a baseline one is given by a part of reward and the difference of the value functions under the framework of linearly solvable Markov decision processes. The logarithm of density ratio is efficiently calculated by binomial logistic regression, of which the classifier is constructed by the reward and state value function. The classifier tries to discriminate between samples drawn from the optimal state transition probability and those from the baseline one. Then, the estimated state value function is used to initialize the part of the deep neural networks for forward reinforcement learning. The proposed deep forward and inverse reinforcement learning is applied into two benchmark games: Atari 2600 and Reversi. Simulation results show that our method reaches the best performance substantially faster than the standard combination of forward and inverse reinforcement learning as well as behavior cloning.