sprab4/Seq_to_Seq_Translator

Sequence-to-Sequence Translation with Encoder-Decoder Architecture

This project implements a translation model using a Sequence-to-Sequence model. This repository contains the trained Seq2Seq models for English to Haitian Creole translation and viceversa.

Project Overview

• Tokenizer: BART-base tokenizer from Hugging Face's transformers library
• Model: Sequence-to-Sequence with Encoder-Decoder architecture
• Task: Translation between English and Haitian Creole and vice versa
• Languages: The project uses Haitian Creole and English for training the model

Repository Structure

This repository contains the below files:

1. seq2seq_model_en_to_ht.pth: The trained Seq2Seq model for English to Haitian translation
2. seq2seq_model_ht_to_en.pth: The trained Seq2Seq model for Haitian to English translation
3. README.md: This file, explaining the model

Model Architecture

The Seq2Seq Translator consists of:

• Embedding Layer: Converts tokens to vectors
• Encoder: LSTM layers processing input sequence
• Decoder: LSTM layers generating output sequence
• Fully Connected Layer: Maps decoder output to vocabulary

Key Parameters:

• Embedding Dimension: 128
• Hidden Size: 256
• Batch Size: 128
• Block Size (Sequence Length): 16
• Dropout: 0.2
• Learning Rate: 1e-3
• Number of Epochs: 10

Translation Metrics

Both BLEU and ChrF scores were tracked during training to measure the model's performance

BLEU Scores

• Seq2Seq (Ht->En): Shows inconsistent performance, peaks around 0.2
• Seq2Seq (En->Ht): Maintains very low performance near 0

ChrF Scores

• Seq2Seq (En->Ht): Shows peaks around 5.0 with high variability
• Seq2Seq (Ht->En): Fluctuates between 1.0-6.0, unstable performance

Training and Validation Losses

The training and validation losses were recorded throughout the training process. The model shows:

• Variable training behavior
• Higher peak performance but less stability
• Inconsistent validation metrics
• Uneven translation quality between directions

Dataset

• Training Set: 16,000 sentence pairs
• Validation Set: 4,000 sentence pairs
• Data Format: JSON with parallel text pairs
• Tokenization: BART-base tokenizer from Hugging Face

Limitations

1. Fixed sequence length
2. No attention mechanism
3. Memory constraints with long sequences
4. Unstable training behavior
5. Basic encoder-decoder architecture compared to modern standards