sprab4/bloomz_fine_tuned_model

Fine-Tuned BloomZ Model

Repository Name: sprab4/bloomz_fine_tuned_model

Model Description

This repository contains a fine-tuned version of the BloomZ model for generating text in Haitian Creole. The BloomZ architecture is optimized for multilingual tasks, and this fine-tuning adapts it to generate fluent and contextually accurate text specifically in Haitian Creole.

Training Details

• Base Model: bigscience/bloomz
• Dataset: Haitian Creole dataset from Hugging Face which follows the alpaca format of instructions, input, and output.
• Fine-Tuning Process:
  • Trained for 2 epochs using the Hugging Face Trainer.
  • Validation set was used to evaluate performance periodically during fine-tuning.
  • Instruction-response formats (Alpaca-style) were prioritized in the training dataset.
• Hyperparameters:
  • learning_rate=5e-5
  • per_device_train_batch_size=2
  • gradient_accumulation_steps=8
  • num_train_epochs=2
  • fp16=False
  • save_steps=500
  • save_total_limit=2
  • logging_steps=50
  • optim="adamw_hf"
  • weight_decay=0.1
  • warmup_steps=500

Results

Quantitative Metrics

BERTScore Varies based on the response for each prompt

BERTScore for the below prompt:

Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz

English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences

• BERTScore Precision: 0.7526
• BERTScore Recall: 0.7445
• BERTScore F1: 0.7485

Example Output

Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz

English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences

Prediction: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz Kouchè la se koupyè nan ouviyon an. Li se fè kouyèl nan fòmèl an a. Lè se vèk la a, ak li se pèmèt pou fòm la. Kouche la te kòt pou kòm la, li te pòmè pou vòk li. La te fèt li, ki te lès pou lè li anpè.

English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences The couch is the cup of the oven. It is done in a formal manner. Time is here, and it is allowed for the form. Lying there was the coast for the as there, it was pomer for his voice. There he was born, which was east for his peaceful time.