flan-t5-base-finetuned-arxiv
This model is a fine-tuned version of google/flan-t5-base on the scientific_papers dataset. It achieves the following results on the evaluation set:
- Loss: 2.485082
- Rouge1: 12.032000
- Rouge2: 4.38100
- Rougel: 9.842600
- Rougelsum: 11.139600
- Gen Len: 19.000000
Training procedure
The following bitsandbytes
quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
The following bitsandbytes
quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-4
- weight_decay: 0.01
- train_batch_size: 32
- optimizer: paged_adamw_8bit (8-bit adam optimization)
- num_epochs: 4.47
- fp16: False
Framework versions
PEFT 0.5.0
PEFT 0.5.0
Transformers 4.35.0
Pytorch 1.10.1+cu111
Datasets 2.14.7
Tokenizers 0.14.1
bitsandbytes 0.41.2.post2
accelerate 0.24.0
evaluate 0.4.1
rouge-score 0.1.2
- Downloads last month
- 1
Dataset used to train knachinen/flan-t5-base-finetuned-arxiv
Evaluation results
- Rouge1 on scientific_papersself-reported12.032
- Rouge2 on scientific_papersself-reported4.384
- Rougel on scientific_papersself-reported9.843
- Rougelsum on scientific_papersself-reported11.140