YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
π§βπ¨ Text-to-Face Generation with BERT, GAN & CelebA
A deep learning project for text-to-face image generation, where natural-language descriptions are converted into realistic facial images using Sentence-BERT embeddings and a conditional Generative Adversarial Network (GAN).
The project contains two text-conditioned face generation models:
- Baseline Text-to-Face Generator β a lightweight conditional generator.
- Attention-based Text-to-Face GAN β an improved architecture using Self-Attention, Spectral Normalization, and a conditional Discriminator.
π Project Overview
The goal of this project is to generate a face image from a textual description.
For example, given:
"The female has high cheekbones. Her hair is black. She has arched eyebrows, a big nose and bushy eyebrows. She is young, smiling and wearing lipstick."
the model attempts to generate a corresponding face.
The overall pipeline is:
Text Description
β
βΌ
Sentence-BERT
β
βΌ
768-D Text Embedding
β
βΌ
Text Conditioning
β
+
Random Noise
β
βΌ
Generator
β
βΌ
Generated Face
The project uses the CelebA dataset with natural-language descriptions associated with facial images.
π― Objectives
The project aims to explore:
- Text-to-image generation
- Conditional GANs
- Text embeddings
- Sentence-BERT
- Face generation
- GAN architecture design
- Self-Attention
- Spectral Normalization
- Conditional Discriminators
- Image-text matching
- Generative model training
ποΈ Dataset
The project uses the CelebA (CelebFaces Attributes Dataset).
CelebA contains large-scale celebrity face images together with facial attribute annotations.
The project additionally uses textual descriptions generated from the facial attributes.
Dataset sources
- Kaggle CelebA Dataset
- CUHK Multimedia Lab β CelebA
The project expects a dataset containing:
Face Images
+
Text Descriptions
+
CelebA Attribute Information
π Text Descriptions
Each image is associated with one or more natural-language descriptions.
Example:
The female has pretty high cheekbones and an oval face.
She has brown hair.
She has arched eyebrows and a pointy nose.
She is smiling, seems attractive and young.
She has rosy cheeks and heavy makeup.
She is wearing earrings and lipstick.
Another example:
He wears a 5 o'clock shadow.
His hair is brown and straight.
He has a slightly open mouth and a pointy nose.
He looks attractive and young and is smiling.
He is wearing a necktie.
These descriptions provide the semantic information used to condition the image generator.
π§ Text Encoder
The project uses:
SentenceTransformer
with:
all-mpnet-base-v2
The model converts each textual description into a 768-dimensional semantic embedding.
SentenceTransformer("all-mpnet-base-v2")
The descriptions are split into sentences and each sentence is encoded individually.
The resulting sentence embeddings are then averaged:
Text
β
βββ Sentence 1 βββΊ Embedding
βββ Sentence 2 βββΊ Embedding
βββ Sentence 3 βββΊ Embedding
βββ Sentence N βββΊ Embedding
β
βΌ
Mean Embedding
β
βΌ
768-D
This produces a fixed-size representation regardless of the number of sentences.
π’ Text Embedding
The initial embedding size is:
768
The embedding is then reduced to:
256
using a projection layer.
Sentence-BERT
β
βΌ
768 dimensions
β
βΌ
Linear Layer
β
βΌ
256 dimensions
ποΈ Model 1 β Baseline Text-to-Face Generator
The first model is a conditional image generator that combines:
- Random noise
- Text embeddings
- Transposed convolution layers
- Batch Normalization
- ReLU / LeakyReLU
- Tanh output
The model generates RGB face images.
Architecture
The input consists of:
Random Noise
100 dimensions
and:
Text Embedding
768 dimensions
The text embedding is projected to:
256 dimensions
The generator then concatenates the noise and text representation.
Text Description
β
βΌ
Sentence-BERT
β
βΌ
768-D
β
βΌ
Projection
β
βΌ
256-D
β
ββββββββββββββ
β β
βΌ βΌ
Random Noise Text
100-D 256-D
β β
βββββββ¬βββββββ
βΌ
Conditional Input
β
βΌ
ConvTranspose2D
β
βΌ
4 Γ 4
β
βΌ
8 Γ 8
β
βΌ
16 Γ 16
β
βΌ
32 Γ 32
β
βΌ
64 Γ 64
β
βΌ
RGB Image
Generator Configuration
model = Generator(
100, # noise size
128, # feature size
3, # RGB channels
768, # embedding size
256 # reduced embedding size
)
Main parameters
| Parameter | Value |
|---|---|
| Noise Dimension | 100 |
| Initial Feature Size | 128 |
| Text Embedding | 768 |
| Reduced Embedding | 256 |
| Output Channels | 3 |
| Optimizer | Adam |
| Learning Rate | 0.0002 |
| Betas | (0.5, 0.5) |
πΌοΈ Generated Image Resolution
The baseline generator progressively upsamples the feature representation:
4 Γ 4
β
8 Γ 8
β
16 Γ 16
β
32 Γ 32
β
64 Γ 64
β
128 Γ 128
The final output is an RGB face image.
πΎ Pretrained Generator
A trained generator checkpoint can be loaded using:
model.load_state_dict(
torch.load(
"generator_50k.pth",
map_location="cpu"
)
)
model.eval()
This allows the model to generate faces directly from new text descriptions without retraining.
π§ͺ Baseline Inference
Example:
test_noise = torch.randn(
size=(1, 100)
)
test_embeddings = sentence_encoder.convert_text_to_embeddings([
"The female has pretty high cheekbones and an oval face. "
"She has brown hair. She has arched eyebrows and a pointy nose. "
"She is smiling, seems attractive, young, has rosy cheeks and heavy makeup. "
"She is wearing earrings and lipstick."
])
test_image = model(
test_noise,
test_embeddings
)
The generated image is then visualized using torchvision.
π§ Model 2 β Attention-Based Text-to-Face GAN
The second model is a more advanced conditional GAN architecture.
It contains:
- Conditional Generator
- Conditional Discriminator
- Sentence-BERT
- Self-Attention
- Spectral Normalization
- Batch Normalization
- Transposed Convolution
- Conditional image-text discrimination
The architecture is designed to improve image quality and capture long-range spatial relationships in facial features.
ποΈ Generator Architecture
The second generator receives:
Text Embedding
+
Random Noise
The text embedding is processed through:
768
β
256
β
100
The resulting representation is combined with the random noise through element-wise multiplication.
concat_input = torch.mul(
noise,
encoded_text
)
Generator Pipeline
Text
β
βΌ
Sentence-BERT
β
βΌ
768-D
β
βΌ
Linear
β
βΌ
256-D
β
βΌ
Linear
β
βΌ
100-D
β
β
Random Noise βββββββββ€
βΌ
Element-wise
Multiplication
β
βΌ
1 Γ 1
β
βΌ
4 Γ 4 Feature
β
βΌ
8 Γ 8 Feature
β
βΌ
16 Γ 16 Feature
β
βΌ
Self-Attention
β
βΌ
32 Γ 32 Feature
β
βΌ
Self-Attention
β
βΌ
64 Γ 64 Feature
β
βΌ
Self-Attention
β
βΌ
128 Γ 128 Image
ποΈ Self-Attention
The generator includes custom Self-Attention modules.
The attention mechanism allows the model to establish relationships between distant spatial locations.
This is useful for face generation because facial features are not completely independent.
For example:
Eyes
β
ββββββββΊ Nose
β
ββββββββΊ Mouth
β
ββββββββΊ Face Shape
Instead of only processing local convolutional features, Self-Attention allows the network to model broader spatial relationships.
π¬ Self-Attention Architecture
The module uses three projections:
Query
Key
Value
implemented using convolution layers:
self.query_conv
self.key_conv
self.value_conv
Attention scores are calculated using matrix multiplication:
Query Γ Key
β
βΌ
Softmax
β
βΌ
Attention Map
β
βΌ
Value
The attention output is then combined with the original feature map through a learnable parameter:
Output = Ξ³ Γ Attention + Input
π‘οΈ Spectral Normalization
The advanced architecture also implements Spectral Normalization for convolutional layers.
Spectral normalization constrains the magnitude of the network weights and helps stabilize GAN training.
It is applied to:
Generator
β
βββ ConvTranspose2D
βββ ConvTranspose2D
Discriminator
β
βββ Conv2D
Conceptually:
Convolution
β
βΌ
Spectral Normalization
β
βΌ
Controlled Weight Magnitude
β
βΌ
More Stable GAN Training
π΅οΈ Discriminator
The discriminator determines whether an image is:
Real
or
Fake
but this model also receives the corresponding text description.
Therefore, it performs conditional discrimination.
Instead of asking only:
Is this image real?
the discriminator effectively evaluates:
Is this image realistic and consistent with the given text?
π Conditional Discriminator
The architecture is:
Image
β
βΌ
CNN Encoder
β
βΌ
Image Features
β
β
Text βββΊ Sentence-BERT
β
βΌ
Text Encoder
β
βΌ
Text Features
β
βββββββ΄ββββββ
β β
βββββββ¬ββββββ
βΌ
Image + Text
Features
β
βΌ
CNN Layers
β
βΌ
Real / Fake Score
𧩠Wrong Image Training
An important part of the discriminator training is the use of wrong image-text pairs.
For each text description:
Correct Pair
Text ββββββββββΊ Correct Face
and:
Incorrect Pair
Text ββββββββββΊ Different Face
The discriminator learns to distinguish between:
Real Image + Correct Text
and:
Wrong Image + Text
Fake Image + Text
This encourages the generated image to be semantically related to the description.
π Dataset Pipeline
The custom dataset returns:
true_image
true_text
wrong_image
The pipeline is:
CelebA
β
ββββββββββΊ Real Image
β
ββββββββββΊ Text Description
β
ββββββββββΊ Random Wrong Image
Images are resized to:
128 Γ 128
and normalized using:
transforms.Normalize(
mean=(0.5),
std=(0.5)
)
βοΈ Training Configuration
The advanced model uses:
| Parameter | Value |
|---|---|
| Epochs | 20 |
| Batch Size | 16 |
| Dataset Subset | 20,000 |
| Noise Size | 100 |
| Feature Size | 64 |
| Image Size | 128 Γ 128 |
| Channels | 3 |
| Text Embedding | 768 |
| Reduced Embedding | 256 |
| Generator LR | 0.0002 |
| Discriminator LR | 0.0002 |
| Attention | Enabled |
| Optimizer | Adam |
βοΈ GAN Training
The training process alternates between the Generator and Discriminator.
Text
β
βΌ
Text Encoder
β
βΌ
Embeddings
β
βΌ
Generator
β²
β
Random Noise
β
βΌ
Fake Image
β
βββββββββ΄βββββββββ
β β
βΌ βΌ
Generator Discriminator
Loss β
β
ββββββββββββββΌβββββββββββββ
β β β
βΌ βΌ βΌ
Real Image Wrong Image Fake Image
π― Generator Objective
The generator attempts to fool the discriminator.
Text + Noise
β
βΌ
Generator
β
βΌ
Fake Face
β
βΌ
Discriminator
β
βΌ
Real?
The generator loss is calculated using:
nn.BCELoss()
and attempts to make the discriminator classify generated images as real.
π‘οΈ Discriminator Objective
The discriminator receives three types of examples:
1. Real Image + Correct Text
Expected:
1
2. Wrong Image + Text
Expected:
0
3. Generated Image + Text
Expected:
0
Therefore:
Discriminator Loss
β
βββ Real Loss
βββ Wrong Pair Loss
βββ Fake Loss
π Experiment Tracking
The project uses:
Weights & Biases (W&B) for experiment tracking.
wandb.init(
project="text-to-face",
name="n-sagan"
)
The following metrics are tracked:
Generator Loss
Discriminator Loss
Generated Images
Generated images are logged after each configured epoch.
πΌοΈ Visualization
The project visualizes generated images using:
Matplotlib
+
Torchvision
Generated images can be arranged into grids:
torchvision.utils.make_grid(
output,
normalize=True
)
This makes it possible to monitor image quality during training.
π¬ Model Comparison
The project contains two approaches:
| Feature | Baseline Generator | Attention GAN |
|---|---|---|
| Text Conditioning | β | β |
| Sentence-BERT | β | β |
| Random Noise | β | β |
| ConvTranspose | β | β |
| Conditional Generation | β | β |
| Discriminator | β | β |
| Self-Attention | β | β |
| Spectral Normalization | β | β |
| Wrong Image Pairs | β | β |
| W&B Tracking | β | β |
| GAN Training | Partial | β |
| Output | Face | Face |
π Overall Architecture
Text Description
β
βΌ
Sentence-BERT
all-mpnet-base-v2
β
βΌ
768-D Embedding
β
βββββββββββββ΄ββββββββββββ
β β
βΌ βΌ
Baseline Model Attention GAN
β β
β Text Projection
β β
β βΌ
β Random Noise
β β
β βΌ
β Generator
β β
β ββββββββββ΄βββββββββ
β β β
β βΌ βΌ
β Self-Attention Spectral Norm
β β
β βΌ
β Generated Face
β β
β βΌ
β Discriminator
β β²
β β
β Real / Wrong
β Images
β
βΌ
Generated Face
π οΈ Technologies
Deep Learning
- PyTorch
- Torchvision
- PyTorch Neural Networks
Natural Language Processing
- Sentence Transformers
all-mpnet-base-v2- Text Embeddings
Computer Vision
- OpenCV
- PIL
- Torchvision
- Matplotlib
Machine Learning
- NumPy
- Pandas
GAN / Generative AI
- Conditional GAN
- Self-Attention
- Spectral Normalization
- Transposed Convolution
Experiment Tracking
- Weights & Biases
Dataset
- CelebA
π¦ Installation
Clone the repository:
git clone https://github.com/kad99kev/FGTD.git
Move into the project directory:
cd FGTD
Install dependencies:
pip install -r requirements.txt
Install the main deep-learning dependencies if required:
pip install torch torchvision
Install Sentence Transformers:
pip install sentence-transformers
Install experiment tracking:
pip install wandb
π Running the Project
1. Load the Text Encoder
from sentence_transformers import SentenceTransformer
sentence_encoder = SentenceTransformer(
"all-mpnet-base-v2"
)
2. Generate an Image from Text
Prepare a text description:
text = [
"The female has high cheekbones and black hair. "
"She is young and smiling."
]
Convert it into an embedding:
Text
β
Sentence-BERT
β
768-D Embedding
Generate a random latent vector:
noise = torch.randn(
1,
100
)
Then pass both into the generator:
image = generator(
noise,
text_embeddings
)
π§ͺ Example Prompts
Female Face
The female has pretty high cheekbones and an oval face.
Her hair is black.
She has arched eyebrows and a pointy nose.
She is smiling and looks young.
She is wearing earrings and lipstick.
Male Face
The man is young and attractive.
He has brown straight hair and a pointy nose.
He is smiling and wearing a necktie.
Different Facial Attributes
The man has a double chin and high cheekbones.
He has black hair and big lips.
He looks young.
π Suggested Repository Structure
Text-to-Face-GAN/
β
βββ README.md
βββ requirements.txt
β
βββ notebooks/
β βββ baseline_text_to_face.ipynb
β βββ attention_text_to_face_gan.ipynb
β
βββ models/
β βββ generator.py
β βββ discriminator.py
β βββ attention.py
β βββ spectral_norm.py
β
βββ text_encoder/
β βββ sentence_encoder.py
β
βββ dataset/
β βββ text_5_descr_celeba.csv
β βββ list_attr_celeba.csv
β
βββ checkpoints/
β βββ generator_50k.pth
β
βββ outputs/
β βββ generated_faces/
β
βββ results/
βββ wandb/
π§ Key Concepts Demonstrated
Natural Language Processing
- Sentence Embeddings
- Sentence-BERT
- Semantic Representation
- Text Conditioning
Computer Vision
- Face Generation
- Image Preprocessing
- Image Normalization
- Image Visualization
Deep Learning
- PyTorch
- CNNs
- Transposed Convolution
- Batch Normalization
- ReLU
- Tanh
Generative AI
- GANs
- Conditional GANs
- Text-to-Image Generation
- Latent Noise
- Generator / Discriminator Training
Advanced GAN Techniques
- Self-Attention
- Spectral Normalization
- Conditional Discrimination
- Wrong Image-Text Pairing
π Important Design Decisions
Why Sentence-BERT?
Instead of treating text as individual words, Sentence-BERT provides a semantic representation of the entire description.
Natural Language
β
Semantic Embedding
β
768-D Vector
β
Generator Conditioning
This allows descriptions containing multiple facial attributes to be represented in a compact vector.
Why Conditional GAN?
A normal GAN learns:
Random Noise β Image
This project instead learns:
Random Noise + Text β Image
Therefore, the generated image can be influenced by the supplied description.
Why Self-Attention?
Convolutional layers are excellent at learning local patterns, while Self-Attention helps the network model relationships between distant regions of an image.
This can be useful when generating coherent facial structures.
Why Spectral Normalization?
GAN training can be unstable.
Spectral Normalization helps constrain the network's weight matrices and can improve training stability, particularly in the discriminator.
π Expected Workflow
CelebA Dataset
β
βΌ
Image + Caption
β
βββββββββββ΄ββββββββββ
β β
βΌ βΌ
Image Text
β β
β βΌ
β Sentence-BERT
β β
β βΌ
β 768-D
β β
β βΌ
β Projection
β β
β βΌ
β Text Code
β β
ββββββββββββ¬βββββββββ
βΌ
GAN
β
ββββββββββββ΄βββββββββββ
βΌ βΌ
Generator Discriminator
β β
βΌ β
Fake Face βββββββββββββββββ€
β
Real Face βββββββββββββββββ€
β
Wrong Face ββββββββββββββββ€
βΌ
Real / Fake
β οΈ Implementation Notes
The project is primarily an experimental research implementation.
Several aspects can be improved for a cleaner production/research implementation:
- Separate training and inference scripts.
- Add explicit validation/testing datasets.
- Save both Generator and Discriminator checkpoints.
- Add automatic checkpoint recovery.
- Track additional GAN metrics.
- Evaluate generated image quality quantitatively.
- Use a dedicated configuration file.
- Avoid hard-coded CUDA calls and use the configured device consistently.
- Add reproducible random seeds.
- Add automated experiment logging.
For example, instead of:
generator.cuda()
a more portable approach is:
generator.to(cfg.device)
This allows the project to run on either GPU or CPU.
π Future Improvements
Possible extensions include:
- Increase image resolution to 256Γ256.
- Experiment with larger text encoders.
- Use CLIP-based text-image alignment.
- Add perceptual loss.
- Add text-image similarity metrics.
- Add FID evaluation.
- Add Inception Score.
- Improve caption diversity.
- Experiment with different GAN architectures.
- Compare Self-Attention vs standard convolution.
- Implement progressive image generation.
- Add mixed-precision training.
- Add distributed training.
- Improve dataset balancing.
- Add automated checkpointing.
- Build a web interface for text-to-face generation.
π Learning Outcomes
This project provides practical experience with:
- Text-to-image generation
- Conditional GANs
- PyTorch
- Sentence Transformers
- BERT-based embeddings
- CNN architectures
- Transposed convolution
- GAN optimization
- Generator/Discriminator training
- Self-Attention
- Spectral Normalization
- CelebA preprocessing
- Image-text conditioning
- Experiment tracking with W&B
π Keywords
Text-to-Image
Text-to-Face
Face Generation
Generative AI
GAN
Conditional GAN
cGAN
PyTorch
Sentence-BERT
Sentence Transformers
all-mpnet-base-v2
CelebA
Computer Vision
Deep Learning
Self-Attention
Spectral Normalization
Image Generation
Natural Language Processing
Multimodal AI
Generative Models
π Disclaimer
This project is intended for educational, research, and experimental purposes.
Generated faces are synthetic outputs produced by a machine-learning model and should not be interpreted as photographs or evidence of real individuals.
π¨βπ» Project Summary
Text-to-Face Generation with BERT & Conditional GANs is a multimodal generative AI project that connects natural-language descriptions with facial image generation.
The project progresses from a baseline text-conditioned generator to an advanced GAN architecture incorporating Sentence-BERT embeddings, Self-Attention, Spectral Normalization, and a conditional Discriminator.
Text
β
Sentence-BERT
β
768-D Embedding
β
Text Conditioning
+
Random Noise
β
Generator
β
Generated Face
β
Conditional Discriminator
β
GAN Training
The project demonstrates how NLP and Computer Vision can be combined into a single generative AI pipeline for text-guided face synthesis.