isatish/NLP_midterm_satish_nepal

Bilingual Language Model

Overview

This project implements a bilingual (English-German) language model using LSTM (Long Short-Term Memory) neural networks. The model is trained on a combined corpus of English and German text, and can generate text in both languages. Features

BPE (Byte Pair Encoding) tokenizer LSTM-based language model Training with checkpointing and resuming capability Text generation in both English and German Performance evaluation using perplexity

Requirements

Python 3.7+ PyTorch NumPy Matplotlib tokenizers

we can install the required packages using: pip install torch numpy matplotlib tokenizers

Usage

Data Preparation First we combine English and German text files respectively.

Training the Model

To train the model, run this cell in the notebook:

model = LSTM(input_size, HIDDEN_SIZE, NUM_LAYERS, output_size).to(device) criterion = nn.CrossEntropyLoss() optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE) checkpoint_path = 'model_checkpoint.ckpt'

train_losses, val_losses, perplexities, training_time = train(model, train_loader, val_loader, criterion, optimizer, NUM_EPOCHS, device, checkpoint_path)

This code will:

Load and preprocess the data Initialize the model Train the model, saving checkpoints periodically Plot the training curve and save it as results/training_curve.png Save the training data as results/training_data.csv

Generating Text

To generate text using the trained model: First run this cell to load the trained model:

def load_model(model_path, tokenizer, seed_text, num_chars, device): model = LSTM(tokenizer.vocab_size(), HIDDEN_SIZE, NUM_LAYERS, tokenizer.vocab_size(), DROPOUT).to(device) model.load_state_dict(torch.load(model_path), pickle_module=pickle) model.eval() return generate_text(model, tokenizer, seed_text, num_chars, device)

Finally run the text generation cells at the bottom of the notebook: print("\nGenerating random English:") random_model_e = LSTM(input_size, HIDDEN_SIZE, NUM_LAYERS, output_size).to(device) print(generate_text(random_model_e, dataset.tokenizer, 20, device))

print("\nGenerating trained English:") print(generate_text(model, dataset.tokenizer, 50, device, seed_text="Once upon a time"))

You can modify the seed text in this script.

Model Architecture

The model uses an LSTM architecture with the following hyperparameters:

Batch Size: 64 Sequence Length: 100 Hidden Size: 256 Number of Layers: 2 Learning Rate: 0.001 Number of Epochs: 50 Dropout: 0.2

Results

The model achieves a final perplexity of 1.1113 on the validation set, indicating excellent performance. However, this unusually low perplexity warrants further investigation to ensure it aligns with the specific nature of the task and data.