iko-01/gpt2_oasst1_50pct

GPT-2 Fine-Tuned on OpenAssistant (OASST1 - 50% Subset)

This repository contains a GPT-2 model fine-tuned on a 50% random sample of the OpenAssistant Conversations Dataset (OASST1). The model was trained to generate human-like conversational responses based on instruction-response pairs extracted from the dataset's conversation trees.

📊 Model Details

• Base Model: openai-community/gpt2 (124M parameters)
• Architecture: GPT2LMHeadModel (Causal Language Modeling)
• Language: English
• License: Apache 2.0

🛠️ Training Details

Dataset Preparation

The model was trained on a reconstructed set of linear conversation threads from the OASST1 dataset. Deleted messages were filtered out to ensure higher quality interactions. To optimize training time and resource usage, exactly 50% of the total conversation threads were randomly sampled (seed 42).

Prompt Format: The model expects standard instruction tuning formatting:

### Human:
{user_message}

### Assistant:
{assistant_response}

Hyperparameters

Parameter	Value
Learning Rate	5e-5
Batch Size	4 (with Gradient Accumulation: 4 -> Effective Batch Size: 16)
Epochs	3
Optimizer	AdamW
Weight Decay	0.01
Warmup Steps	100
Precision	fp16 (Mixed Precision)
Max Length	1024 tokens

Hardware

• Trained using Google Colab (GPU)
• Framework: PyTorch, Hugging Face transformers & datasets

🚀 Usage

You can use this model for text generation using the transformers library.

Python Code

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "iko-01/gpt2_oasst1_50pct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Note: The model was trained with the ### Human: / ### Assistant: format
prompt = "### Human:\nWhat is the capital of France?\n\n### Assistant:\n"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    inputs.input_ids, 
    max_length=150, 
    do_sample=True, 
    top_p=0.95, 
    temperature=0.8,
    pad_token_id=tokenizer.eos_token_id
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚠️ Limitations and Bias

• Base Model Constraints: As it is based on GPT-2 (124M), the model may struggle with complex reasoning, factual accuracy, and long-context coherence compared to larger modern LLMs (like Llama 3 or Qwen).
• Dataset Bias: The OASST1 dataset contains crowdsourced conversations, which may include hallucinations, biases, or toxic language present in the original data.
• Prompt Format: The model strictly expects the ### Human: and ### Assistant: formatting to perform optimally.

📚 Citation

If you use this model or the underlying dataset, please cite the original OpenAssistant paper:

@article{kopf2023openassistant,
  title={OpenAssistant Conversations--Democratizing Large Language Model Alignment},
  author={Köpf, Andreas and Kilcher, Yann and von Rütte, Dimitri and Anagnostidis, Sotiris and Tam, Zhi Rui and Stevens, Kaleb and Abdullah, Abdullah and Berre, Armin and Bui, Duc Minh and Hofmann, Fabian and others},
  journal={arXiv preprint arXiv:2304.07327},
  year={2023}
}