  Designing a speech-to-intent (S2I) agent which maps the users' spoken
commands to the agents' desired task actions can be challenging due to the
diverse grammatical and lexical preference of different users. As a remedy, we
discuss a user-taught S2I system in this paper. The user-taught system learns
from scratch from the users' spoken input with action demonstration, which
ensure it is fully matched to the users' way of formulating intents and their
articulation habits. The main issue is the scarce training data due to the user
effort involved. Existing state-of-art approaches in this setting are based on
non-negative matrix factorization (NMF) and capsule networks. In this paper we
combine the encoder of an end-to-end ASR system with the prior NMF/capsule
network-based user-taught decoder, and investigate whether pre-training
methodology can reduce training data requirements for the NMF and capsule
network. Experimental results show the pre-trained ASR-NMF framework
significantly outperforms other models, and also, we discuss limitations of
pre-training with different types of command-and-control(C&C) applications.
