---
license: apache-2.0
datasets:
- grammarly/coedit
language:
- en
metrics:
- accuracy
tags:
- torchtune
- grammar-correction
---


### Llama3 CoEdit

This is a Llama3 8B based model trained using [torchtune](https://pytorch.org/torchtune) on the `grammarly/coedit` dataset.

### Training details

The exact training script ([lora_finetune_distributed](./lora_finetune_distributed.py)) and config ([8B_lora.yaml](./8B_lora.yaml)) are both included in this repository. 

**Training command**: ``tune run --nproc_per_node 8 lora_finetune_distributed --config 8B_lora.yaml`` 

> Yes I used 8 GPUs :)

In order to add the dataset, I added the following lines to the config:

```
dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: grammarly/coedit
  template: GrammarErrorCorrectionTemplate
  column_map: {"sentence": "src", "output": "tgt"}
  train_on_input: False
  split: train
```

**Loss curve**

![image/png](https://cdn-uploads.huggingface.co/production/uploads/615b01ae487af9ad44dad803/Te9DycG2UVGm_JLnrG9De.png)


### Evaluation results