Hugging Face's logo

kshitizkhanal7
/
flan-t5-climate-qlora

Transformers
Safetensors
climate-change
flan-t5
qlora
instruction-tuning
Inference Endpoints
Model card Files Community
flan-t5-climate-qlora / README.md
kshitizkhanal7's picture
kshitizkhanal7
Update README.md
74652c7 verified
preview code
|
raw
history blame
3.91 kB
metadata
library_name: transformers
tags:
  - climate-change
  - flan-t5
  - qlora
  - instruction-tuning

Model Card for FLAN-T5 Climate Action QLoRA

This is a QLoRA-finetuned version of FLAN-T5 specifically trained for climate action content analysis and generation. The model is optimized for processing and analyzing text related to climate change, sustainability, and environmental policies.

Model Details

Model Description

  • Developed by: Kshitiz Khanal
  • Shared by: kshitizkhanal7
  • Model type: Instruction-tuned Language Model with QLoRA fine-tuning
  • Language(s): English
  • License: Apache 2.0
  • Finetuned from model: google/flan-t5-base

Model Sources

Uses

Direct Use

The model is designed for:

  • Analyzing climate policies and initiatives
  • Summarizing climate action documents
  • Answering questions about climate change and environmental policies
  • Evaluating sustainability measures
  • Processing climate-related research and reports

Downstream Use

The model can be integrated into:

  • Climate policy analysis tools
  • Environmental reporting systems
  • Sustainability assessment frameworks
  • Climate research applications
  • Educational tools about climate change

Out-of-Scope Use

The model should not be used for:

  • Critical policy decisions without human oversight
  • Generation of climate misinformation
  • Technical climate science research without expert validation
  • Commercial deployment without proper testing
  • Medical or legal advice

Bias, Risks, and Limitations

  • Limited to climate-related content analysis
  • May not perform well on general domain tasks
  • Potential biases from web-based training data
  • Should not be the sole source for critical decisions
  • Performance varies on technical climate science topics

Recommendations

  • Always verify model outputs with authoritative sources
  • Use human expert oversight for critical applications
  • Consider the model as a supplementary tool, not a replacement for expert knowledge
  • Regular evaluation of outputs for potential biases
  • Use in conjunction with other data sources for comprehensive analysis

Training Details

Training Data

  • Source: FineWeb dataset filtered for climate content
  • Selection criteria: Climate-related keywords and quality metrics
  • Processing: Instruction-style formatting with climate focus

Training Procedure

Preprocessing

  • Text cleaning and normalization
  • Instruction templates for climate context
  • Maximum input length: 512 tokens
  • Maximum output length: 128 tokens

Training Hyperparameters

  • Training regime: QLoRA 4-bit fine-tuning
  • Epochs: 3
  • Learning rate: 2e-4
  • Batch size: 4
  • Gradient accumulation steps: 4
  • LoRA rank: 16
  • LoRA alpha: 32
  • Target modules: Query and Value matrices
  • LoRA dropout: 0.05

Environmental Impact

  • Hardware Type: Single GPU
  • Hours used: ~4 hours
  • Cloud Provider: Local
  • Carbon Emitted: Minimal due to QLoRA efficiency

Technical Specifications

Model Architecture and Objective

  • Base architecture: FLAN-T5
  • Objective: Climate-specific text analysis
  • QLoRA adaptation for efficient fine-tuning
  • 4-bit quantization for reduced memory usage

Compute Infrastructure

  • Python 3.8+
  • PyTorch
  • Transformers library
  • bitsandbytes for quantization
  • PEFT for LoRA implementation

Hardware

Minimum requirements:

  • 16GB GPU memory for inference
  • 24GB GPU memory recommended for training
  • CPU inference possible but slower

Citation

If you use this model, please cite:

@misc{khanal2024climate,
  title={FLAN-T5 Climate Action QLoRA},
  author={Khanal, Kshitiz},
  year={2024},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora}}
}