  Preference-based learning of reward functions, where the reward function is
learned using comparison data, has been well studied for complex robotic tasks
such as autonomous driving. Existing algorithms have focused on learning reward
functions that are linear in a set of trajectory features. The features are
typically hand-coded, and preference-based learning is used to determine a
particular user's relative weighting for each feature. Designing a
representative set of features to encode reward is challenging and can result
in inaccurate models that fail to model the users' preferences or perform the
task properly. In this paper, we present a method to learn both the relative
weighting among features as well as additional features that help encode a
user's reward function. The additional features are modeled as a neural network
that is trained on the data from pairwise comparison queries. We apply our
methods to a driving scenario used in previous work and compare the predictive
power of our method to that of only hand-coded features. We perform additional
analysis to interpret the learned features and examine the optimal
trajectories. Our results show that adding an additional learned feature to the
reward model enhances both its predictive power and expressiveness, producing
unique results for each user.
