In Automatic Speech Recognition it is still challenging to learn useful intermediate representations when using highlevel (or abstract) target units such as words. For that reason, character or phoneme based systems tend to outperform word-based systems when just few hundreds of hours of training data are being used. In this paper, we first show how hierarchical multi-task training can encourage the formation of useful intermediate representations. We achieve this by performing Connectionist Temporal Classification at different levels of the network with targets of different granularity. Our model thus performs predictions in multiple scales for the same input. On the standard 300h Switchboard training setup, our hierarchical multi-task architecture exhibits improvements over single-task architectures with the same number of parameters. Our model obtains 14.0% Word Error Rate on the Eval2000 Switchboard subset without any decoder or language model, outperforming the current state-of-the-art on acoustic-to-word models. Index Termsmulti-task learning, ASR, CTC Automatic Speech Recognition (ASR) systems, traditionally, were composed of three components: a phonemebased acoustic model (AM), a word-based language model (LM), and a pronunciation lexicon that maps a sequence of phonemes to words [1]. This setup was predominating for traditional systems based on Gaussian Mixture Models (GMMs) and Hidden Markov models (HMMs) [2] . More recently, systems that used a single loss function based directly on the predicted sequence (i.e. end-to-end models) showed similar performance by using characters as target units. This approach is used in Sequence-to-Sequence (S2S) [3] models and Connectionist Temporal Classification (CTC) [4], whereby not requiring pronunciation lexicon anymore (i.e. characters as target units) simplified the ASR pipeline. To simplify even more the ASR pipeline, [5] suggest to use words instead of characters as target units. Even though this forces a closed vocabulary setting, experiments show BiLSTM … CTC Loss BiLSTM n < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f z j k 9 N m o 2 f z 0 g j s i l z l 5 j n m F p X U 2 C V J C p 8 L g z y L F f b j 6 f 0 i 7 7 + g s T L X T z Q r M M r 4 W M t U C k 7 O 6 o y a r a A d L M W 2 I V x D C 9 Y a N b + G S S 7 K D D U J x a 0 d h E F B U c U N S a F w 7 g 9 L i w U X U z 7 G g U P N M 7 R R t R x z z i 6 d k 7 A 0 N + 5 o Y k v 3 9 4 u K Z 9 b O s t j d z D h N 7 G a 2 M P / L B i W l t 1 E l d V E S a r H 6 K C 0 V o 5 w t d m a J N C h I z R x w Y a S b l Y k J N 1 y Q a 8 Z 3 H Y S b G 2 9 D 7 7 o d B u 3 w M Y A 6 n M M F X E E I N 3 A H D 9 C B L g h I 4 B X e v Y n 3 5 n 2 s u q p 5 6 9 L O 4 I + 8 z x 8 4 x I o 4 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N t n m D L J 1 N g R z 1 8 Q x o C q I Y j L d 6 r c = " > A A A B 3 X i c b Z B L S w M x F I X v + K y 1 a n X r J l g E V 2 X G j S 4 F N y 5 b s A 9 o h 5 J J 7 7 S x m c y Q 3 B H K 0 F / g x o U i / i 1 3 / h v T x 0 J b D w Q + z k n I v S f K l L T k + 9 / e 1 v b O 7 t 5 + 6 a B 8 W D k 6 P q m e V t o 2 z Y 3 A l k h V a r o R t 6 i k x h Z J U t j N D P I k U t i J J v f z v P O M x s p U P 9 I 0 w z D h I y 1 j K T g 5 q 6 k H 1 Z p f 9 x d i m x C s o A Y r N Q b V r / 4 w F X m C m o T i 1 v Y C P 6 O w 4 I a k U D g r 9 7 v F w V P r J 0 m k b u Z c B r b 9 W x u / p f 1 c o p v w 0 L q L C f U Y v l R n C t G K Z t v z Y b S o C A 1 d c C F k W 5 W J s b c c E G u m 7 I r I V h f e R P a 1 / X A r w d N H 0 p w D h d w B Q H c w B 0 8 Q A N a I A D h B d 7 g 3 X v y X r 2 P Z V 1 b 3 q q 3 M / g j 7 / M H w t m L n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N t n m D L J 1 N g R z 1 8 Q x o C q I Y j L d 6 r c = " > A A A B 3 X i c b Z B L S w M x F I X v + K y 1 a n X r J l g E V 2 X G j S 4 F N y 5 b s A 9 o h 5 J J 7 7 S x m c y Q 3 B H K 0 F / g x o U i / i 1 3 / h v T x 0 J b D w Q + z k n I v S f K l L T k + 9 / e 1 v b O 7 t 5 + 6 a B 8 W D k 6 P q m e V t o 2 z Y 3 A l k h V a r o R t 6 i k x h Z J U t j N D P I k U t i J J v f z v P O M x s p U P 9 I 0 w z D h I y 1 j K T g 5 q 6 k H 1 Z p f 9 x d i m x C s o A Y r N Q b V r / 4 w F X m C m o T i 1 v Y C P 6 O w 4 I a k U D g r 9 7 v F w V P r J 0 m k b u Z c B r b 9 W x u / p f 1 c o p v w 0 L q L C f U Y v l R n C t G K Z t v z Y b S o C A 1 d c C F k W 5 W J s b c c E G u m 7 I r I V h f e R P a 1 / X A r w d N H 0 p w D h d w B Q H c w B 0 8 Q A N a I A D h B d 7 g 3 X v y X r 2 P Z V 1 b 3 q q 3 M / g j 7 / M H w t m L n w = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " c k o U t 7 c S N q a K M m O z q d g Q v N W X 1 0 n 7 q u 6 5 d a / l 1 h q 3 R R x l O I N z u A Q P r q E B 9 9 A E H x g g P M M r v D m P z o v z 7 n w s W 0 t O M X M K f + B 8 / g D V 0 Y z u < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " I q W 2 N 3 6 Q k t B w n t G Q a v L h E 1 D L r r E = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e A m J v O v y Z A r Z E Z M L a F M c X s r Y W O q K D M 2 m 5 I N w V t 9 e Z 2 0 r 6 q e W / W a 1 5 X 6 b R 5 H E c 7 g H C 7 B g x r U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 a C k 8 + c w h 8 4 n z / X E Y z y < / l a t e x i t >
More recently, in Automatic Speech Recognition, REF proposed also the use of multiple levels of segmentation in a hierarchical multitask learning structure.
Hierarchical Multi Task Learning With CTC
ArXiv
ArXiv
Computer Science