rml1/amharic-llama

Amharic Language Model

This repository contains an instruction-tuned Amharic language model based on TinyLlama-1.1B. The model is trained to understand and generate Amharic text based on instructions.

Model Details

• Base Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
• Training Data: Amharic instruction dataset
• Task: Instruction following in Amharic
• Training Type: Instruction fine-tuning

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("rml1/amharic-llama")
tokenizer = AutoTokenizer.from_pretrained("rml1/amharic-llama")

# Example usage
instruction = "የተጠቀሰው አረፍተ ነገር ያለው አንድምታ አዎንታዊ ነው ዕአሉታዊ"
input_text = f"### Instruction:\n{instruction}\n\n### Response:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training Setup

1. Install Dependencies

pip install -r requirements.txt

2. Set Environment Variables

# Set your Hugging Face token
export HF_TOKEN="your_hugging_face_token"

# Set your Weights & Biases token (optional)
export WANDB_API_KEY="your_wandb_token"

3. Check GPU Status

# Verify GPUs are visible
nvidia-smi

# Check PyTorch GPU access
python -c "import torch; print(f'GPUs available: {torch.cuda.device_count()}')"

4. Start Training

python train_hf.py

Training Details

The model is trained with the following optimizations:

• Mixed precision (bfloat16)
• Gradient checkpointing for memory efficiency
• Cosine learning rate scheduler
• Multi-GPU support with automatic device mapping
• Real-time training monitoring via Weights & Biases

Progress Tracking

The training script provides detailed progress indicators:

• 🚀 Initialization steps
• 📦 Model download progress
• 🔤 Tokenizer setup
• 📚 Dataset loading
• 🏃‍♂️ Training progress with:
  • Batch progress
  • Loss values
  • Learning rate
  • Speed (samples/second)
  • GPU memory usage
  • Time remaining

Limitations

• Based on TinyLlama-1.1B, so it has the inherent limitations of a 1.1B parameter model
• Performance may vary depending on instruction complexity

Citation

@misc{amharic-llama,
  author = {Your Name},
  title = {Amharic Language Model},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/rml1/amharic-llama}}
}