Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation (2023)

Abstract: Visual temporal-aligned translation aims to transform the visual sequence into natural words, including important applicable tasks such as lipreading and fingerspelling recognition. However, various performance habits of specific words by different speakers or signers can lead to visual ambiguity, which has become a major obstacle to the development of current methods. Considering the constraints above, the generalization ability of the translation system is supposed to be further explored through the evaluation results on unseen performers. In this paper, we develop a novel generalizable framework named Contrastive Token-Wise Meta-learning (CtoML), which strives to transfer recognition skills to unseen performers. To the best of our knowledge, employing meta-learning methods directly in the image domain poses two main challenges, and we propose corresponding strategies. First, sequence prediction in visual temporal-aligned translation, which aims to generate multiple words autoregressively, is different from the vanilla classification. Thus, we devise the token-wise diversity-aware weights for the meta-train stage, which encourages the model to make efforts on those ambiguously recognized tokens. Second, considering the consistency of word-visual prototypes across different domains, we develop two complementary global and local contrastive losses to maintain inter-class relationships and promote domain-independent. We conduct extensive experiments on the widely-used lipreading dataset GRID and the fingerspelling dataset ChicagoFSWild, and the experimental results show the effectiveness of our proposed CtoML over existing state-of-the-art methods.

## introduction human communication is dominated by speech, which conveys semantic information through the * equal contribution. † corresponding author acoustic signals. however, persons with dysphonia necessitate reliance on visual perception for independent expressions, such as lip movements and hand gestures. hence, automatic visual language translation is helpful in bridging the gap between people who communicate through diverse senses. in visual language translation tasks, whether it is observing lip movement or understanding gesture sequences, the common denominator is that the visual content and the translated natural language words are temporally aligned. in this paper, we collectively refer to tasks with the above properties as visual temporal-aligned translation. specifically, lipreading, a.k.a. visual speech recognition (vsr), aims to recognize spoken sentences based on lip movements. another task is fingerspelling recognition, where recognized text is generated letter-byletter from the fast and coherent indistinguishable handshapes of signers (figure 2 ). accordingly, mainstream research methods utilize autoregressive models to generate multiple words. however, current methods @xcite show weakness when applied to real-life scenarios, due to the fact that different performers have a variety of performance habits on specific words leading to ambiguity, as shown in figure 1 . moreover, the gap between performers can be magnified in data-limited settings, i.e. some environments or low-resource languages where collection and annotation are expensive. ideally, an applicable visual temporal-aligned translation system is supposed to have excellent generalization ability and convincing recognition accuracy for unseen performers. we argue that referring to methods in the domain generalization task to deal with this dilemma is a feasible solution. concretely, performers with different performance styles or presentation habits can be treated as different domains. for example, in lipreading, some speakers have a slight lip movement, while others have a relatively exaggerated display. another instance in fingerspelling recognition is that the handshapes of each signer on the same letter are diverse, which could be due to factors such as personal habits and movement directions. therefore, domainindependent visual temporal-aligned translation can break through the above obstacles, especially on datasets with limited labeled samples. in this paper, we propose an innovative generalizable framework to deal with the challenging domain-independent visual temporal-aligned translation, called contrastive token-wise metalearning (ctoml). video-sentence pairs of specific performers are used for training, and then we test on unseen performers. as far as we know, directly transferring meta-learning methods in the image domain to lipreading and fingerspelling recognition raises the following two challenges: first, sequence prediction in visual temporalaligned translation autoregressively generates multiple words, which is different from vanilla classification. consequently, we design the token-wise diversity-aware weights for the meta-train phase. the variance of the interacted attention map between performers is measured, and then regarded as the learning difficulty coefficient of the token, so as to concentrate on tackling ambiguous words. second, taking into account the consistency of word-visual prototypes across different domains, we develop two complementary contrastive losses. globally, we frame-word-align the decoded interaction matrix between movement features and sequential sentences to maintain a consistent semantic space of the same class across domains, regardless of the domain-specific variations and class-specific vocabulary positions in the sequence. locally, relying on contrastive constraints, we facilitate the model to draw closer decoded outputs of words that are semantically similar regardless of domain, and pull away those words that are disparate. in summary, the token-wise diversity-aware weights we devised encourage the model to focus on tokens with inherent ambiguity in sequence prediction, and the meta-learning process can simulate various domain shift scenarios to assist in finding generalization learning directions. the effectiveness of our ctoml is demonstrated through extensive experiments on the lipreading benchmark dataset grid and fingerspelling dataset chicagof-swild. our main contributions are as follows: ## experiments we evaluate and compare our ctoml on two challenging visual temporal-aligned translation tasks, lipreading and fingerspelling recognition. our experiments are conducted on two datasets: grid @xcite for lipreading, and chicagof-swild @xcite for fingerspelling recognition. in this section, we provide a brief introduction to the datasets and corresponding evaluation metrics. then, we present concrete experimental settings and compare ctoml with baseline methods. subsequently, we analyze the main results and conduct ablation studies. besides, we also provide qualitative examples and analysis on grid and chicagofswild dataset in the appendix d. ## limitations in this section, we develop a clear discussion of the limitations of this paper. our method faces obstacles when attempting to validate it on datasets other than those previously used in this paper. for example, lrs2 @xcite is a widely used dataset in visual language recognition tasks. however, since lrs2 dataset does not provide speaker identification labels, we cannot easily classify speakers into domain-specific and domainindependent sets. despite the enormous amount of work, re-annotating existing datasets with crowdsourcing or annotating a new real-life dataset with speaker labels is a viable solution. besides, existing data augmentation methods cannot match the generalization requirements on visual temporal-aligned translation perfectly, which inspires researchers to develop targeted augmentation paradigms based on the study in this paper to cooperate with our meta-learning strategies. ## ethics statement the datasets used in this study were those produced by previous researchers, and we followed all relevant legal and ethical guidelines for their acquisition and use. furthermore, we recognize the potential moral hazard of visual temporal-aligned translation tasks, such as their use in surveillance or listening. we are committed to conducting our research ethically and ensuring that our research is beneficial.