fsndzomga/aya-finetuned-mura-8B-lora

Model Card for Aya Fine-Tuned Model

Model Details

Model Description

This model is a fine-tuned version of the CohereForAI/aya-23-8B base model. It has been fine-tuned using a private dataset of prompt-response pairs that has been curated over the past two years. The fine-tuning process aimed to improve the model's ability to generate relevant and accurate responses in various conversational contexts.

• Developed by: Franck Stéphane NDZOMGA
• Funded by [optional]: FS NDZOMGA
• Shared by [optional]: Franck Stéphane NDZOMGA
• Model type: Causal Language Model with LoRA Adapters
• Language(s) (NLP): Arabic, Chinese (simplified & traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese
• License: Apache-2.0
• Finetuned from model: CohereForAI/aya-23-8B

Uses

Direct Use

This model can be used directly for text generation tasks, such as chatbots, automated customer support, or other conversational AI applications. It can generate responses based on provided prompts, mimicking human-like conversational patterns.

Downstream Use [optional]

The model can be further fine-tuned for specific tasks or integrated into larger systems that require contextual understanding and response generation capabilities. Examples include virtual assistants, personalized content generation, and more.

Out-of-Scope Use

This model is not suitable for tasks requiring factual accuracy and should not be used in applications where generating misinformation could cause harm. Misuse in generating offensive, harmful, or biased content is strongly discouraged.

Bias, Risks, and Limitations

As with many language models, this fine-tuned model may exhibit biases present in the training data. The responses might reflect the specific contexts and perspectives inherent to the dataset it was trained on. Users should be aware of these potential biases and use the model with caution, especially in sensitive or critical applications.

Recommendations

Users should monitor the outputs of the model and implement additional filtering or moderation systems as necessary. Awareness of the model's limitations and potential biases is crucial for responsible deployment.

How to Get Started with the Model

To use this model for text generation, you can load it using the Hugging Face transformers library:

# pip install transformers==4.41.1
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "fsndzomga/aya-finetuned-mura-8B-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Format message with the command-r-plus chat template
messages = [{"role": "user", "content": "who is emmanuel macron ?"}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")

gen_tokens = model.generate(
    input_ids, 
    max_new_tokens=100, 
    do_sample=True, 
    temperature=0.3,
    )

gen_text = tokenizer.decode(gen_tokens[0])
print(gen_text)

Training Details

Training Data

The model was fine-tuned using a private dataset of prompt-response pairs curated by Franck Stéphane NDZOMGA over the past two years. The dataset includes a diverse set of conversational examples designed to improve the model's response generation capabilities.

Training Procedure

• Preprocessing [optional]: The dataset was preprocessed to ensure consistent formatting, including tokenization using the base model's tokenizer and normalization of text inputs.

Training Hyperparameters

• Precision: Mixed precision (fp16)
• Number of epochs: 1
• Batch size: 1 (gradient accumulation steps: 16 to handle memory issues)
• Learning rate: 5e-5
• Warmup steps: 100
• Weight decay: 0.01
• Logging: Every 10 steps
• Checkpoint saving: Every 50 steps, keeping only the latest 2 checkpoints
• Evaluation: Every 50 steps
• Max steps: 100
• Remove unused columns: False
• Mixed Precision: Disabled (fp16=False) to avoid conflicts

Additional Information from Training Code

• The training utilized the PEFT (Parameter Efficient Fine-Tuning) library, specifically leveraging the LoRA (Low-Rank Adaptation) method to fine-tune the CohereForAI/aya-23-8B model.
• The model was loaded with 8-bit precision to manage memory more effectively during training.
• GPU memory management strategies were employed, including setting environment variables to manage CUDA memory allocation and clearing the PyTorch cache.
• The training process involved using the Hugging Face ecosystem for model storage and deployment, and all model files were pushed to the Hugging Face Hub for accessibility.