results
This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on the dreamproit/bill_summary_us dataset. It achieves the following results on the evaluation set:
- Loss: 0.7163
Model description
Model has been fine-tuned from llama 2 7B chat model for legal summarization tasks.
Intended uses & limitations
The model has been fine-tuned with legal summarization text for summarization tasks. Can produce repeating text when creating longer outputs. Tested only with english and the bill_summary_us dataset.
Training procedure
SFTTrainer from Hugging Face's TRL library used for fine-tuning process.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: constant
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.8263 | 0.24 | 70 | 0.7693 |
| 0.6035 | 0.48 | 140 | 0.7467 |
| 0.845 | 0.72 | 210 | 0.7347 |
| 0.5782 | 0.96 | 280 | 0.7163 |
Framework versions
- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0
license
Llama 2 is licensed under the LLAMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no library tag.
Model tree for W3bsurf/Llawma-sum-2-7b-chat
Base model
meta-llama/Llama-2-7b-chat-hf