README.md

metadata

title: ark-instruct-line-item
app_file: /home/tosi-n/ark/gorilla/app.py
sdk: gradio
sdk_version: 4.0.2

ARK - Jack's Accounting ALM Training Framework

This is a base pipeline to build out our task specific repos layer on

Note ReadMe currently for gorrila module which is mainly pipeline for long context window training flexibility. For the chimp module which is for default context window training, it has it's ReadMe in folder path

Usage Requirements

To download and use the pre-trained weights you will need:

1. Hugging Face (HF) account with valid email. Note, the email used for HF must alse be used for the license agreement.
2. Accept the Meta license and acceptable use policy

Installation and Quick Guide

To install and run the application:

1. Clone the repository on your local machine, using git clone and pasting the url of this project.
2. Complete pre-requiste installation. Run the following code:

pip install -r requirements.txt
pip install flash-attn --no-build-isolation

3. Training Pre-trained weights through Fine-tuning QLoRa, LoRa or full. Run the following code in bash script and update args before running for QloRa :

sh runner.sh

4. Merge amd get Trainable LoRA Weight. Run the following code in bash script and update args where necessary:

sh process_wt.sh

5. Test your model by terminal chat. Run the following code in bash script andupdate args where necessary:

sh stream.sh

6. Test your model on gradio UI. Run the following code in bash script and update args where necessary:

sh demo.sh

Training args for Full and LoRA

Fine-tuning

torchrun --nproc_per_node=8 fine-tune.py  \
        --model_name_or_path path_to/Llama-2-7b-hf \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --cache_dir path_to_cache \
        --model_max_length 8192 \
        --use_flash_attn True \
        --low_rank_training False \
        --num_train_epochs 1  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 8     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 1000     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True \
        --max_steps 1000

Supervised Fine-tuning

torchrun --nproc_per_node=8 supervised-fine-tune.py  \
        --model_name_or_path path_to_Llama2_chat_models \
        --bf16 True \
        --output_dir path_to_saving_checkpoints       \
        --model_max_length 32768 \
        --use_flash_attn True \
        --data_path LongAlpaca-12k.json \
        --low_rank_training True \
        --num_train_epochs 3  \
        --per_device_train_batch_size 1     \
        --per_device_eval_batch_size 2     \
        --gradient_accumulation_steps 1     \
        --evaluation_strategy "no"     \
        --save_strategy "steps"     \
        --save_steps 1000     \
        --save_total_limit 2     \
        --learning_rate 2e-5     \
        --weight_decay 0.0     \
        --warmup_steps 20     \
        --lr_scheduler_type "constant_with_warmup"     \
        --logging_steps 1     \
        --deepspeed "ds_configs/stage2.json" \
        --tf32 True

Evaluation

Perplexity Validation