YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Seq2Seq-Based English-to-Georgian Translation Model
1. Project Overview This project implements a Seq2Seq translation model for converting English text to Georgian. It utilizes an LSTM-based encoder-decoder architecture and evaluates performance using BLEU and ChrF metrics.
2. Requirements The following libraries are required: • HuggingFace's datasets for loading the OPUS100 dataset • transformers for multilingual tokenization • PyTorch for building and training the Seq2Seq model • evaluate for BLEU and ChrF metrics • sacrebleu for additional BLEU metric evaluation
3. Dataset • Source: OPUS100 Dataset (English-Georgian translation pair) from HuggingFace. • Splits: Training, Validation, and Test datasets.
4. Preprocessing • Tokenization: Sentences are tokenized using the bert-base-multilingual-cased tokenizer. • Truncation and Padding: Applied with a maximum token length of 128. • Batch Processing: Data is processed in batches for efficient training and evaluation.
5. Model Architecture The Seq2Seq model is a custom LSTM-based encoder-decoder architecture with the following components: 1. Embedding Layer: Converts tokens into dense vector representations. 2. LSTM Encoder: Encodes input sequences into context vectors. 3. LSTM Decoder: Decodes context vectors into output sequences. 4. Fully Connected Layer: Maps decoder outputs to the target vocabulary. Key Parameters: • Vocabulary Size: English + Georgian tokens. • Hidden Dimension: 256. • LSTM Layers: 2.
6. Training • Optimizer: Adam with a learning rate of 0.001. • Loss Function: CrossEntropyLoss. • Epochs: 10. • Batch Size: 8. Training includes a teacher forcing ratio of 0.5 to guide the decoder with ground truth inputs during the early stages.
7. Evaluation Performance is evaluated using: 1. BLEU Score: Measures word-level translation quality. 2. ChrF Score: Measures character-level translation quality. Scores are computed after each epoch and saved in CSV files (seq2seq_bleu_scores.csv and seq2seq_chrf_scores.csv) for analysis.
8. Inference The trained model is used for inference by loading saved checkpoints (seq2seq_model_state_dict.pth). Translations are generated by passing English input sequences to the model.
9. Visualization • Loss Curves: Training and validation loss curves are plotted to analyze convergence and detect overfitting. • BLEU Scores: Plotted across epochs to visualize translation quality improvement. • ChrF Scores: Plotted across epochs to assess character-level accuracy.
10. Key Observations • Loss: Gradual reduction in training and validation loss indicates learning progress. • BLEU & ChrF Scores: Consistent improvement reflects enhanced translation quality. • Convergence: Observed through decreasing validation loss and increasing evaluation scores.
11. Future Enhancements 1. Introduce attention mechanisms for better long-sequence translations. 2. Explore deeper or bidirectional LSTM architectures. 3. Compare with Transformer-based architectures like BERT or GPT for improved performance.