Progressive Facial Granularity Aggregation with Bilateral Attribute-based Enhancement for Face-to-Speech Synthesis (2025)

Abstract: For individuals who have experienced traumatic events such as strokes, speech may no longer be a viable means of communication. While text-to-speech (TTS) can be used as a communication aid since it generates synthetic speech, it fails to preserve the user’s own voice. As such, face-to-voice (FTV) synthesis, which derives corresponding voices from facial images, provides a promising alternative. However, existing methods rely on pre-trained visual encoders, and finetune them to align with speech embeddings, which strips fine-grained information from facial inputs such as gender or ethnicity, despite their known correlation with vocal traits. Moreover, these pipelines are multi-stage, which requires separate training of multiple components, thus leading to training inefficiency. To address these limitations, we utilize fine-grained facial attribute modeling by decomposing facial images into non-overlapping segments and progressively integrating them into a multi-granular representation. This representation is further refined through multi-task learning of speaker attributes such as gender and ethnicity at both the visual and acoustic domains. Moreover, to improve alignment robustness, we adopt a multi-view training strategy by pairing various visual perspectives of a speaker in terms of different angles and lighting conditions, with identical speech recordings. Extensive subjective and objective evaluations confirm that our approach substantially enhances face-voice congruence and synthesis stability.

## introduction the ability to communicate using one's own voice is an intrinsic and fundamental aspect of human identity, self-expression, and social interaction. however, a range of neurological and physiological conditions can severely impair speech production mechanisms. for instance, dysarthria is a speech disorder that results from various etiological factors, including cerebrovascular incidents (e.g., strokes) and degenerative neuromuscular disorders such as multiple sclerosis and parkinson's disease @xcite . the severity of dysarthria varies, but nevertheless manifests as slurred, unintelligible, and phonetically distorted speech. in more extreme cases such as total glossectomy or orofacial myofunctional disorders, individuals may completely lose the ability to generate speech. these impairments significantly hinder verbal communication, often leading to frustration, social isolation, and a reduced quality of life @xcite . towards this, text-to-speech (tts) systems @xcite @xcite have been utilized as assistive technologies to convert typed text into artificial speech @xcite . however, while they are effective for communication, they fail to preserve the speaker's unique vocal identity. due to this, personalized multi-speaker tts methods @xcite @xcite could be a potential solution @xcite as they are able to imitate a target speaker's vocal characteristics for speech generation. yet, achieving high-fidelity voice cloning typically requires extensive multi-speaker training or few-shot adaptation with a large amount of speech recordings. thus, while zero-shot multi-speaker synthesis can ideally enable speaker adaptation, it remains infeasible for those without any accessible prior recordings, such as individuals with complete vocal muscle paralysis. this limitation highlights the need for alternative biometric modalities to infer speaker identity and enable personalized voice synthesis that is independent of speech recordings. both early and recent psychological research suggests the existence of a strong correlation between facial identity with vocal characteristics. specifically, studies such as @xcite ; von @xcite ; smith et al. @xcite have indicated that humans can infer aspects of a person's voice, such as pitch and timbre, based solely on their facial features. as a result, this insight has driven investigations into face-to-voice (ftv) synthesis, where a facial image serves as a reference for generating a corresponding voice, and then producing speech in that inferred voice. early methods of ftv synthesis relied on statistical techniques that mapped facial images onto facial landmarks, which were then transformed into eigenvoice representations @xcite . these approaches, however, were limited by the need for manual alignment between facial and vocal features. more recent work has adopted multistage training pipelines @xcite @xcite , wherein a face encoder is first trained to produce embeddings that align with outputs from a separately trained audiobased speech encoder. afterwards, during inference, the face encoder substitutes the speech encoder. while this strategy improves performance, it introduces training complexity and inefficiency. moreover, because the facial representations are specifically trained to align with audio features, they fail to directly capture and utilize fine-grained local facial features such as gender or other demographic attributes that are essential in producing identity-consistent speech. although @xcite attempts to explicitly incorporate such facial attributes, their method requires a combination of demographic metadata, 2d facial features, textures (e.g., skin, muscle), and 3d cranial structures. not only does this introduce substantial complexity in terms of data acquisition and input modalities, but also necessitates a three-stage learning process to integrate these sources of information, which ultimately results in a complex and resource-intensive training pipeline. to address the limitations of prior work, specifically the reliance of inference-time-only visual encoders, underutilization of fine-grained facial features, and dependence on multi-stage training, we present a novel end-to-end ftv synthesis framework, which eschews external models and instead focuses on effective facial feature learning. our method starts with a progressive facial encoder that decomposes and hierarchically aggregates local facial regions to form a rich visual identity embedding. additionally, to ensure alignment of highlevel semantic attributes such as gender and ethnicity, we introduce a bilateral attribute enhancement mechanism, which is applied to both the facial rep-resentation and to the synthesized audio output. a multi-view data augmentation strategy is further adopted where each speech sample is paired with a diverse set of facial images from the same speaker, which are captured under varying poses, lighting conditions, and facial movements. through both subjective and objective evaluations, we demonstrate that our method generates speech with higher speaker fidelity and stronger ftv associations. ## experimental settings we leverage the trainval subset of the lrs opensource dataset @xcite for training, and conduct dataset multi-view augmentation and pseudo-labelling foot_2 as detailed in sections 3.2 and 3.3, respectively. out of a total of 159,511 utterances, the training-validation is split using a ratio of 9:1. audios are resampled to 16,000 hz, and we use a filter and window length of 1024, and hop size of 256 for mel-spectrogram processing. for fair comparisons, all experiments adhere to a batch size of 20, and 350,000 training iterations are conducted for approximately two days using a single a6000 gpu. the total number of parameters is approximately 43.6 million. ## conclusion in this paper, we have presented a novel end-toend framework for ftv synthesis that emphasizes effective facial representation learning and crossmodal attribute alignment. in contrast to prior methods that depend on multi-stage pipelines or external pretrained models, our approach progressively aggregates local facial features to construct a robust identity embedding and then enforces semantic consistency through bilateral supervision of demographic attributes such as gender and ethnicity. additionally, by incorporating a multi-view faceaudio alignment strategy, we improve the model's ability to maintain vocal consistency across diverse visual inputs. comprehensive evaluations across five assessment tasks and multiple objective met-rics confirm that our method significantly improves speaker fidelity and identity preservation, thus offering a viable solution for personalized speech synthesis even in the absence of voice recordings.