  On-the-job learning consists in continuously learning while being used in
production, in an open environment, meaning that the system has to deal on its
own with situations and elements never seen before. The kind of systems that
seem to be especially adapted to on-the-job learning are dialogue systems,
since they can take advantage of their interactions with users to collect
feedback to adapt and improve their components over time. Some dialogue systems
performing on-the-job learning have been built and evaluated but no general
methodology has yet been defined. Thus in this paper, we propose a first
general methodology for evaluating on-the-job learning dialogue systems. We
also describe a task-oriented dialogue system which improves on-the-job its
natural language component through its user interactions. We finally evaluate
our system with the described methodology.
