  Automatic speech recognition (ASR) technologies today are primarily optimized
for given datasets; thus, any changes in the application environment (e.g.,
acoustic conditions or topic domains) may inevitably degrade the performance.
We can collect new data describing the new environment and fine-tune the
system, but this naturally leads to higher error rates for the earlier
datasets, referred to as catastrophic forgetting. The concept of lifelong
learning (LLL) aiming to enable a machine to sequentially learn new tasks from
new datasets describing the changing real world without forgetting the
previously learned knowledge is thus brought to attention. This paper reports,
to our knowledge, the first effort to extensively consider and analyze the use
of various approaches of LLL in end-to-end (E2E) ASR, including proposing novel
methods in saving data for past domains to mitigate the catastrophic forgetting
problem. An overall relative reduction of 28.7% in WER was achieved compared to
the fine-tuning baseline when sequentially learning on three very different
benchmark corpora. This can be the first step toward the highly desired ASR
technologies capable of synchronizing with the continuously changing real
world.
