manavdhamecha77/iSign-t5-pose-to-text

iSign T5 Pose-to-Text Models

This repository contains trained models for Indian Sign Language Translation (SLT) using pose-based representations and T5 encoder-decoder models.

The models translate sequences of sign language pose keypoints into natural language text.

Models

The repository contains five trained checkpoints:

Model	Description
t5-small	T5-Small with spatial pose features
t5-small-motion	T5-Small with spatial and motion features
t5-base	T5-Base with spatial pose features
t5-large	T5-Large with spatial pose features
t5-large-motion	T5-Large with spatial and motion features

The t5-base-motion checkpoint is not included because it was corrupted.

Repository Structure

.
├── t5-small/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-small-motion/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-base/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
├── t5-large/
│   ├── pose_encoder.pt
│   └── t5/
│       ├── config.json
│       ├── generation_config.json
│       └── model.safetensors
│
└── t5-large-motion/
    ├── pose_encoder.pt
    └── t5/
        ├── config.json
        ├── generation_config.json
        └── model.safetensors

Model Architecture

The system uses a T5-based encoder-decoder architecture adapted for pose-to-text translation.

The pipeline consists of:

1. Pose input represented as sequences of sign language keypoints.
2. Pose encoder that projects pose features into the T5 hidden representation space.
3. T5 encoder-decoder that generates the corresponding natural language sentence.

The pose data is processed using the pose-format Python library.

For motion-enhanced models, temporal motion information is incorporated alongside spatial pose features.

Dataset

The models were trained using a subset of the iSign dataset for Indian Sign Language processing.

The experimental subset contains:

• Total samples: 18,867
• Training samples: 16,979
• Held-out test samples: 1,887

The input consists of pose files, while the target consists of corresponding textual sentence annotations.

Installation

Clone the repository:

git clone https://huggingface.co/manavdhamecha77/iSign-t5-pose-to-text
cd iSign-t5-pose-to-text

Install the required dependencies:

pip install torch transformers pose-format numpy

Loading a Model

The T5 component and pose encoder are stored separately.

For example, to access the T5-Small model:

from transformers import T5ForConditionalGeneration

model = T5ForConditionalGeneration.from_pretrained(
    "manavdhamecha77/iSign-t5-pose-to-text",
    subfolder="t5-small/t5"
)

The corresponding pose encoder is located at:

t5-small/pose_encoder.pt

The pose encoder must be loaded using the same architecture and preprocessing pipeline used during training.

Available Models

T5-Small

t5-small/

Spatial pose features only.

T5-Small + Motion

t5-small-motion/

Uses spatial pose features together with motion-based features.

T5-Base

t5-base/

Uses the T5-Base architecture with spatial pose features.

T5-Large

t5-large/

Uses the T5-Large architecture with spatial pose features.

T5-Large + Motion

t5-large-motion/

Uses T5-Large with spatial and motion features.

Inference

Inference requires:

1. Loading the corresponding T5 model.
2. Initializing the pose encoder architecture.
3. Loading pose_encoder.pt.
4. Processing the input .pose file using pose-format.
5. Applying the same normalization and feature extraction used during training.
6. Passing the encoded pose representation to the T5 model.
7. Generating the translated text.

An inference script should use the exact preprocessing and model architecture from the training code to ensure compatibility.

Dependencies

Main dependencies include:

• Python
• PyTorch
• Hugging Face Transformers
• pose-format
• NumPy

Depending on the original training environment, additional dependencies may be required.

License

Please refer to the original iSign dataset license and terms of use when using the dataset or derived models.

Acknowledgements

This work uses:

• The iSign dataset
• Hugging Face Transformers
• PyTorch
• The pose-format Python library
• T5 models