Classic Seq2Seq (Sutskever et al. 2014) Reimplementation

This is a faithful implementation of the original sequence-to-sequence paper by Sutskever et al. (2014), "Sequence to Sequence Learning with Neural Networks", trained for German-to-English translation on the WMT19 dataset.

Model Description

Architecture: 4-layer deep LSTM encoder-decoder with 1000 hidden units per layer
Parameters: 194 million parameters
Training Data: WMT19 German-English dataset (35 million sentence pairs)
Training Duration: 10 epochs
Key Innovation: Input sequence reversal (crucial Sutskever et al. finding)
Tokenization: SentencePiece (50k German vocab, 40k English vocab)
Decoding: Beam search with length normalization

Model Details

Architecture

This model implements the exact architecture from the original Sutskever et al. paper:

Encoder: 4-layer LSTM with input sequence reversal
Decoder: 4-layer LSTM with teacher forcing during training
No attention mechanism - pure encoder-decoder as originally conceived
Beam search decoding with configurable beam size (default: 12)

Training Hyperparameters I Used

Optimizer: SGD with momentum (lr=0.7, momentum=0.9)
Loss: Cross-entropy with padding token masking
Regularization: Dropout (0.2) and gradient clipping (5.0)
Hardware Used: Trained on NVIDIA A6000 (10 vCPUs, 60GB RAM, 200GB disk)
Training Time: Roughly 60 hours on 500k dataset subset

Usage

Installation

pip install torch sentencepiece requests tqdm

Quick Start

# Download the model files
python -c "
import requests
from pathlib import Path

files = [
    'best_model.pt',
    'german_sp.model', 
    'english_sp.model',
    'german_sp.vocab',
    'english_sp.vocab'
]

base_url = 'https://huggingface.co/idosumit/seq2seq/resolve/main'
for file in files:
    response = requests.get(f'{base_url}/{file}')
    Path(file).write_bytes(response.content)
    print(f'Downloaded {file}')
"

Using with the Full Codebase

For the complete training and inference pipeline, clone the full repository:

git clone https://gitlab.com/sumitdoesml/classic-seq2seq
cd classic-seq2seq
uv venv && source .venv/bin/activate
pip install -r requirements.txt

# Download pre-trained model
python scripts/download_pretrained.py

# Interactive translation
python scripts/inference.py --interactive --device cpu

# Single sentence translation
python scripts/inference.py --sentence "Hallo, wie geht es dir?" --device cpu

Example Translations

German: "Hallo, wie geht es dir?"
English: "Hello, how are you doing?"

German: "Können Sie mir bitte helfen?"
English: "Can you help me, please?"

German: "Das Wetter ist heute schön."
English: "The weather is beautiful today."

Technical Specifications

Framework: PyTorch
Precision: FP32 (with optional mixed precision training)
Memory Requirements: ~6GB VRAM for inference, 16GB+ for training
Input Length: Up to 100 tokens (filtered during training)
Languages: German → English only

Limitations

Era: This is a 2014-era model without modern improvements like attention
Repetition: May exhibit repetition patterns typical of early seq2seq models
Vocabulary: Limited to SentencePiece vocabularies (50k German, 40k English)
Domain: Trained on general domain text, may not perform well on specialized domains
Unidirectional: German to English only, not bidirectional

Training Details

Hyperparameters

Learning Rate: 0.7 with StepLR decay (gamma=0.5, step_size=3)
Batch Size: 16 (memory-constrained)
Gradient Clipping: 5.0
Dropout: 0.2
Sequence Length: 3-100 tokens

Dataset

Source: WMT19 German-English
Size: 35 million sentence pairs
Preprocessing: Length filtering, quality checks, automatic train/val split
Tokenization: SentencePiece trained on the full dataset

Repository

Full source code, training scripts, and documentation: GitLab Repository

Note: This is an educational/research implementation to understand foundational seq2seq concepts. For production German-English translation, consider modern transformer-based models with attention mechanisms.

Downloads last month: -; Downloads are not tracked for this model. How to track