llama3-8b-coedit / README.md
jrc's picture
Update README.md
71a3e59 verified
metadata
license: apache-2.0
datasets:
  - grammarly/coedit
language:
  - en
metrics:
  - accuracy
tags:
  - torchtune
  - grammar-correction

Llama3 CoEdit

This is a Llama3 8B based model trained using torchtune on the grammarly/coedit dataset.

Training details

The exact training script (lora_finetune_distributed) and config (8B_lora.yaml) are both included in this repository.

Training command: tune run --nproc_per_node 8 lora_finetune_distributed --config 8B_lora.yaml

Yes I used 8 GPUs :)

In order to add the dataset, I added the following lines to the config:

dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: grammarly/coedit
  template: GrammarErrorCorrectionTemplate
  column_map: {"sentence": "src", "output": "tgt"}
  train_on_input: False
  split: train

Loss curve

image/png

Evaluation results