Lip reading has received increasing attention in recent years. This paper focuses on the synergy of multilingual lip reading. There are more than 7,000 languages in the world, which implies that it is impractical to train separate lip reading models by collecting large-scale data per language. Although each language has its own linguistic and pronunciation features , the lip movements of all languages share similar patterns  . Based on this idea, in this paper, we try to explore the synergized learning of multilingual lip reading  , and further propose a synchronous bidirectional learning (SBL) framework for effective synergy of multilingual lip reading. Firstly, we introduce the phonemes as our modeling units for the multilingual setting . Similar phoneme always leads to similar visual patterns . The multilingual setting would increase both the quantity and the diversity of each phoneme shared among different languages. So the learning for the multilingual target should bring improvement to the prediction of phonemes . Then, a  SBL block is proposed to  infer the target unit when given its previous and later context. The rules for each specific languagewhich the model itself judges to be is learned in this fill-in-the-blank manner . To make the learning process more targeted at each particular language, we introduce an extra task of predicting the language identity  in the learning process. Finally, we perform a thorough comparison on LRW (English) and LRW-1000 (Mandarin) . The results outperform the existing state of the art by a large margin, and show the promising benefits from the synergized learning of different languages  .
Lip reading has received increasing attention in recent years. This paper focuses on the synergy of multilingual lip reading. There are about as many as 7000 languages in the world, which implies that it is impractical to train separate lip reading models with large-scale data for each language. Although each language has its own linguistic and pronunciation rules , the lip movements of all languages share similar patterns due to the common structures of URLans . Based on this idea, we try to explore the synergized learning of multilingual lip reading in this paper , and further propose a synchronous bidirectional learning (SBL) framework for effective synergy of multilingual lip reading. We firstly introduce phonemes as our modeling units for the multilingual setting here. Phonemes are more closely related with the lip movements than the alphabet letters. At the same time, similar phonemes always lead to similar visual patterns no matter which type the target language is . Then, a novel SBL block is proposed to learn the rules for each language in a fill-in-the-blank way. Specifically, the model has to learn to infer the target unit given its bidirectional context, which could represent the composition rules of phonemes for each language . To make the learning process more targeted at each particular language, an extra task of predicting the language identity is introduced in the learning process. Finally, a thorough comparison on LRW (English) and LRW-1000 (Mandarin) is performed, which shows the promising benefits from the synergized learning of different languages and also reports a new state-of-the-art result on both datasets .
R
more than 7,000
about as many as 7000
R
by collecting
with
R
per
for each
R
features
rules
A
due to the common structures of URLans
D
in this paper,
A
in this paper
R
Firstly, we introduce the
We firstly introduce
R
. Similar phoneme always leads
here. Phonemes are more closely related with the lip movements than the alphabet letters. At the same time, similar phonemes always lead
R
. The multilingual setting would increase both the quantity and the diversity of each phoneme shared among different languages. So the learning for the multilingual target should bring improvement to the prediction of phonemes
no matter which type the target language is
A
novel
A
learn the rules for each language in a fill-in-the-blank way. Specifically, the model has to learn to
R
when given its previous and later context. The rules for each specific languagewhich the model itself judges to be is learned in this fill-in-the-blank manner
given its bidirectional context, which could represent the composition rules of phonemes for each language
D
we introduce
A
is introduced
D
we perform
R
. The results outperform the existing state of the art by a large margin, and show
is performed, which shows
A
and also reports a new state-of-the-art result on both datasets