metadata
language:
- en
tags:
- summarization
library_name: peft
datasets:
- scientific_papers
metrics:
- rouge
model-index:
- name: flan-t5-base-finetuned-arxiv
results:
- task:
type: summarization
name: Summarization
dataset:
name: scientific_papers
type: scientific_papers
args: arxiv
metrics:
- name: Rouge1
type: rouge
value: 12.032
- name: Rouge2
type: rouge
value: 4.3841
- name: Rougel
type: rouge
value: 9.8426
- name: Rougelsum
type: rouge
value: 11.1396
flan-t5-base-finetuned-arxiv
This model is a fine-tuned version of google/flan-t5-base on the scientific_papers dataset. It achieves the following results on the evaluation set:
- Loss: 2.485082
- Rouge1: 12.032000
- Rouge2: 4.38100
- Rougel: 9.842600
- Rougelsum: 11.139600
- Gen Len: 19.000000
Training procedure
The following bitsandbytes quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
The following bitsandbytes quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-4
- weight_decay: 0.01
- train_batch_size: 32
- optimizer: paged_adamw_8bit (8-bit adam optimization)
- num_epochs: 4.47
- fp16: False
Framework versions
PEFT 0.5.0
PEFT 0.5.0
Transformers 4.35.0
Pytorch 1.10.1+cu111
Datasets 2.14.7
Tokenizers 0.14.1
bitsandbytes 0.41.2.post2
accelerate 0.24.0
evaluate 0.4.1
rouge-score 0.1.2