onurkeles/llama-2-7b-econ-title-generator

LLaMA-2-Econ: Title Generation Model

Model Description

A fine-tuned version of the LLaMA-2-7B model for generating titles for economic research papers. Utilizing techniques like Quantized Low Rank Adaptation (QLoRA) and Parameter Efficient Fine Tuning (PEFT), this model aims to enhance the creativity and relevance of generated titles based on abstracts from economic papers.

Intended Uses & Limitations

This model is designed to assist researchers by generating insightful and relevant titles for their economic research papers. Limitations include potential biases present in the training data and the need for human review to ensure title appropriateness and accuracy.

Training and Evaluation Data

The model was fine-tuned on a collection of economics paper abstracts and their corresponding titles, obtained through the arXiv API, covering a wide range of economic subfields.

Training Hyperparameters:

• QLoRA Settings:
  • lora_rank (lora_r): 64
  • lora_dropout: 0.1
• Precision & Quantization:
  • Precision: 4-bit
  • Computation dtype: float16
  • Quantization type: "nf4", with nested quantization
• Training Schedule:
  • Epochs: 8, with early stopping patience of 2 epochs for efficiency
  • bf16 training enabled
• Optimizer & Learning Rate:
  • Optimizer: paged AdamW with 32-bit precision
  • Learning rate: 2e-4, using a cosine learning rate scheduler
  • Warmup ratio: 0.03
• Additional Settings:
  • Gradient checkpointing and a maximum gradient norm of 0.3
  • Sequences grouped by length for training efficiency
  • PEFT adapters merged into the baseline models for enhanced performance

Evaluation Results

• BLEU: 0.16
• ROUGE-1: 0.45
• ROUGE-2: 0.24
• ROUGE-L: 0.41

Citation

• Keleş, O. & Bayraklı, Ö. T. (Fortcoming 2024, May). LLaMA-2-Econ: Enhancing Title Generation, Classification, and Academic Q&A in Economic Research. To be presented in LREC-COLING 2024, 4th Workshop on ECONLP: Turin, Italy.