  Adversarial training of end-to-end (E2E) ASR systems using generative
adversarial networks (GAN) has recently been explored for low-resource ASR
corpora. GANs help to learn the true data representation through a two-player
min-max game. However, training an E2E ASR model using a large ASR corpus with
a GAN framework has never been explored, because it might take excessively long
time due to high-variance gradient updates and face convergence issues. In this
paper, we introduce a novel framework for fine-tuning a pre-trained ASR model
using the GAN objective where the ASR model acts as a generator and a
discriminator tries to distinguish the ASR output from the real data. Since the
ASR model is pre-trained, we hypothesize that the ASR model output (soft
distribution vectors) helps to get higher scores from the discriminator and
makes the task of the discriminator harder within our GAN framework, which in
turn improves the performance of the ASR model in the fine-tuning stage. Here,
the pre-trained ASR model is fine-tuned adversarially against the discriminator
using an additional adversarial loss. Experiments on full LibriSpeech dataset
show that our proposed approach outperforms baselines and conventional
GAN-based adversarial models.
