rulins/blip2-t5-llava

Base Model: BLIP2-t5 pretrained version

Finetune data:

• LLAVA 150k (sample one pair of instruction-answer if multi-round conversations)
• MiniGPT4 3500 pairs

Hyper-parameters:

• BLIP2-flant5-xl + LLAVA (initial commits)

  • v0:

  • lr = 2e-5 --> 0.0 with cosine lr scheduler

  • gbs = 32

  • image size = 480

  • weight decay = 0.05

  • v1 (same as LLAVA):

  • lr = 2e-5

  • gbs = 32

  • image size = 224

  • weight decay = 0.0

• Others

  • lr = 2e-5
  • gbs = 32
  • image size = 224
  • weight decay = 0.0