Vintern-1B-v2 ❄️ (Viet-InternVL2-1B-v2) - The LLaVA πŸŒ‹ Challenger

We are excited to introduce Vintern-1B-v2 the Vietnamese πŸ‡»πŸ‡³ multimodal model that combines the advanced Vietnamese language model Qwen2-0.5B-Instruct[1] with the latest visual model, InternViT-300M-448px[2], CVPR 2024. This model excels in tasks such as OCR-VQA, Doc-VQA, and Chart-VQA,... With only 1 billion parameters, it is 4096 context length finetuned from the Viet-InternVL2-1B model on over 3 million specialized image-question-answer pairs for optical character recognition πŸ”, text recognition πŸ”€, document extraction πŸ“‘, and general VQA. The model can be integrated into various on-device applications πŸ“±, demonstrating its versatility and robust capabilities.

[πŸ€— HF Demo]

The special thing is that our model can be easily finetuned with a T4 GPU on Google Colab by following the instructions provided at the end of this section.

Model Details

Model Name Vision Part Language Part
Vintern-1B-v2 InternViT-300M-448px Qwen2-0.5B-Instruct

Vintern-1B-v2 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. Vintern-1B-v2 consists of InternViT-300M-448px, an MLP projector, and Qwen2-0.5B-Instruct.

Training details πŸ“š

The fine-tuning dataset was meticulously sampled in part from the following datasets:
Viet-OCR-VQA πŸ“š, Viet-Doc-VQA πŸ“„, Viet-Doc-VQA-II πŸ“‘, Vista πŸ–ΌοΈ, Viet-Receipt-VQA 🧾, Viet-Sketches-VQA ✏️, Viet-Geometry-VQA πŸ“, Viet-Wiki-Handwriting ✍️, Viet-ComputerScience-VQA πŸ’», Viet-Handwriting-gemini-VQA πŸ–‹οΈ, Viet-Menu-gemini-VQA 🍽️, Viet-Vintext-gemini-VQA πŸ“œ, Viet-OpenViVQA-gemini-VQA 🧠, Viet-Resume-VQA πŸ“ƒ, Viet-ViTextVQA-gemini-VQA πŸ“‘

Benchmarks πŸ“ˆ

Since there are still many different metrics that need to be tested, we chose a quick and simple metric first to guide the development of our model. Our metric is inspired by Lavy[4]. For the time being, we are using GPT-4 to evaluate the quality of answers on two datasets: OpenViVQA and ViTextVQA. Detailed results can be found at the provided here. The inputs are images, questions, labels, and predicted answers. The model will return a score from 0 to 10 for the corresponding answer quality. The results table is shown below.

Model gpt4o-score
OpenViVQA-dev ViTextVQA-dev
Vintern-1B 7.1/10 7.6/10
Vintern-1B-v2 7.7/10 7.7/10

The benchmark result in MTVQA

Models Open-Source Vietnamese Score
Qwen2-VL 72B (Top 1) βœ— 41.6
GPT-4o (Top 2) βœ— 34.2
Vintern-1B-V2 (Top 3) βœ“ 31.7
Qwen2-VL 7B βœ“ 30.0
Claude3 Opus βœ— 29.1
GPT-4o mini βœ— 29.1
GPT-4V βœ— 28.9
Gemini Ultra βœ— 28.6
InternVL2 76B βœ“ 26.9
QwenVL Max βœ— 23.5
Claude3 Sonnet βœ— 20.8
QwenVL Plus βœ— 18.1
MiniCPM-V2.5 βœ“ 15.3
InternVL-V1.5 βœ— 12.4

We are still working on more detailed benchmarks.

Examples

User: <image>
Mô tả hình ảnh một cÑch chi tiết.
Assistant: Bα»©c αΊ£nh lΓ  mα»™t trang giαΊ₯y kαΊ» Γ΄ vuΓ΄ng vα»›i dΓ²ng chα»― viαΊΏt tay bαΊ±ng mα»±c Δ‘en. Bα»‘ cα»₯c Δ‘Ζ‘n giαΊ£n, chỉ cΓ³ nα»™i dung vΔƒn bαΊ£n được viαΊΏt theo chiều dọc tα»« trΓͺn xuα»‘ng dΖ°α»›i. CΓ‘c dΓ²ng chα»― được sαΊ―p xαΊΏp theo thα»© tα»± tα»« trΓͺn xuα»‘ng dΖ°α»›i nhΖ° sau:

- Nghe thαΊ§y đọc thΖ‘
- Em nghe thαΊ§y đọc bao ngΓ y
- TiαΊΏng thΖ‘ đỏ nαΊ―ng xanh cΓ’y quanh nhΓ 
- MÑi chèo nghe vọng sông xa
- Em Γͺm nghe tiαΊΏng của bΓ  nΔƒm xΖ°a
- Nghe trΔƒng thở Δ‘α»™ng tΓ u dα»«a
- RΓ o rΓ o nghe chuyển con mΖ°a giα»―a trời
- ThΓͺm yΓͺu tiαΊΏng hΓ‘t nα»₯ cười
- Nghe thΖ‘ em thαΊ₯y Δ‘αΊ₯t trời Δ‘αΊΉp ra.
- TrαΊ§n Đăng Khoa
User: <image>
Tam giΓ‘c ABC lΓ  gΓ¬ của đường trΓ²n tΓ’m O ?
Assistant: Tam giΓ‘c ABC lΓ  tam giΓ‘c nα»™i tiαΊΏp của đường trΓ²n tΓ’m O.
User: <image>
Mô tả hình ảnh một cÑch chi tiết.
Assistant: Bα»©c αΊ£nh chα»₯p mα»™t con sΓ΄ng ở Việt Nam vα»›i nhα»―ng ngΓ΄i nhΓ  cα»• kΓ­nh được sΖ‘n mΓ u vΓ ng nhαΊ‘t. PhΓ­a trΖ°α»›c lΓ  mα»™t con thuyền nhỏ mΓ u đỏ vΓ  mα»™t chiαΊΏc thuyền lα»›n mΓ u xanh dΖ°Ζ‘ng.
TrΓͺn thuyền cΓ³ mα»™t người Δ‘Γ n Γ΄ng Δ‘ang chΓ¨o thuyền. Hai bΓͺn bờ sΓ΄ng lΓ  nhα»―ng hΓ ng cΓ’y xanh mΓ‘t. BαΊ§u trời trong xanh vΓ  cΓ³ mα»™t vΓ i Δ‘Γ‘m mΓ’y trαΊ―ng.
Ở gΓ³c dΖ°α»›i bΓͺn trΓ‘i của bα»©c αΊ£nh cΓ³ dΓ²ng chα»― "VNEXPRESS".
User: <image>
Mô tả hình ảnh một cÑch chi tiết.
Assistant: Bα»©c αΊ£nh lΓ  mα»™t thΓ΄ng bΓ‘o tuyển dα»₯ng của cΓ΄ng ty Mobifone. Bα»‘ cα»₯c được chia thΓ nh 3 phαΊ§n chΓ­nh:

- PhαΊ§n Δ‘αΊ§u tiΓͺn lΓ  tiΓͺu đề "TUYα»‚N DỀNG" được in chα»― mΓ u đỏ Δ‘αΊ­m, cα»‘ chα»― lα»›n, nαΊ±m ở vα»‹ trΓ­ trung tΓ’m phΓ­a trΓͺn cΓΉng của bα»©c αΊ£nh.

- PhαΊ§n thα»© hai lΓ  phαΊ§n giα»›i thiệu về vα»‹ trΓ­ vΓ  yΓͺu cαΊ§u của α»©ng viΓͺn. PhαΊ§n nΓ y được trΓ¬nh bΓ y theo dαΊ‘ng liệt kΓͺ cΓ‘c mα»₯c:
    - Vα»‹ trΓ­: NhΓ’n viΓͺn kinh doanh
    - Địa Δ‘iểm: ThΓ nh phα»‘ vΓ  cΓ‘c huyện YΓͺn SΖ‘n, SΖ‘n DΖ°Ζ‘ng, HΓ m YΓͺn, ChiΓͺm HΓ³a
    - YΓͺu cαΊ§u: TrΓ¬nh Δ‘α»™: Tα»‘t nghiệp Trung cαΊ₯p trở lΓͺn, Độ tuα»•i: KhΓ΄ng quΓ‘ 35 tuα»•i

- PhαΊ§n thα»© ba lΓ  phαΊ§n giα»›i thiệu về chαΊΏ Δ‘α»™ Δ‘Γ£i ngα»™. PhαΊ§n nΓ y được trΓ¬nh bΓ y theo dαΊ‘ng liệt kΓͺ cΓ‘c mα»₯c:
    - ChαΊΏ Δ‘α»™ Δ‘Γ£i ngα»™:
        - Mα»©c lΖ°Ζ‘ng: 7.000.000Δ‘ - 17.000.000Δ‘
        - Thưởng thi Δ‘ua hΓ ng thΓ‘ng, quΓ½: Thưởng theo doanh sα»‘
        - ChαΊΏ Δ‘α»™ bαΊ£o hiểm theo quy Δ‘α»‹nh
        - Được trau dα»“i cΓ‘c kα»Ή nΔƒng: Giao tiαΊΏp, thuyαΊΏt phα»₯c, chΔƒm sΓ³c khΓ‘ch hΓ ng...

PhαΊ§n cuα»‘i cΓΉng lΓ  phαΊ§n "NỘP Hα»’ SΖ " vα»›i thΓ΄ng tin liΓͺn lαΊ‘c của người nα»™p hα»“ sΖ‘. PhαΊ§n nΓ y được trΓ¬nh bΓ y theo dαΊ‘ng liệt kΓͺ cΓ‘c mα»₯c:
    - Người nhαΊ­n: TrαΊ§n Thα»‹ ThΓΊy
    - Địa chỉ: 547 đường Quang Trung, Tα»• 25, Phường Phan ThiαΊΏt, TP TuyΓͺn Quang
    - Điện thoαΊ‘i liΓͺn hệ: 0901.452.888
    - Email nhαΊ­n hα»“ sΖ‘: thuy.tranthi.kd@mobifone.vn
    - Thời hαΊ‘n nα»™p hα»“ sΖ‘: Δ‘αΊΏn hαΊΏt ngΓ y 28/5/2021
User: <image>
Con nΓ o sαΊ½ thαΊ―ng ?
Assistant: Con thỏ sẽ thắng.
User: <image>
Mô tả chi tiết hình ảnh.
Assistant: Bα»©c αΊ£nh hiển thα»‹ mα»™t Δ‘oαΊ‘n mΓ£ nguα»“n được viαΊΏt bαΊ±ng ngΓ΄n ngα»― lαΊ­p trΓ¬nh Python. ĐoαΊ‘n mΓ£ nΓ y bao gα»“m hai phαΊ§n:
PhαΊ§n Δ‘αΊ§u: NΓͺu tiΓͺu đề "Example of avoiding complex expressions" (MΓ΄ tαΊ£ vΓ­ dα»₯ về việc trΓ‘nh cΓ‘c hΓ m phα»©c tαΊ‘p).
PhαΊ§n chΓ­nh: Bao gα»“m hai hΓ m square_numbers vΓ  main().
HΓ m square_numbers nhαΊ­n mα»™t tham sα»‘ lΓ  danh sΓ‘ch sα»‘ tα»« 1 Δ‘αΊΏn 4. HΓ m nΓ y sα»­ dα»₯ng vΓ²ng lαΊ·p for để duyệt qua mα»—i sα»‘ trong danh sΓ‘ch vΓ  thΓͺm nΓ³ vΓ o danh sΓ‘ch squares. Sau Δ‘Γ³, hΓ m trαΊ£ về danh sΓ‘ch squares.
HΓ m main() tαΊ‘o mα»™t biαΊΏn numbers vα»›i giΓ‘ trα»‹ [1, 2, 3, 4], sau Δ‘Γ³ gọi hΓ m square_numbers vΓ  ghi kαΊΏt quαΊ£ vΓ o biαΊΏn result. Cuα»‘i cΓΉng, hΓ m print(result) sαΊ½ in ra kαΊΏt quαΊ£ của hΓ m main().

Quickstart

Here provides a code snippet to show you how to load the tokenizer and model and how to generate contents. To run inference using the model, follow the steps outlined in our Colab inference notebook Open In Colab

import numpy as np
import torch
import torchvision.transforms as T
# from decord import VideoReader, cpu
from PIL import Image
from torchvision.transforms.functional import InterpolationMode
from transformers import AutoModel, AutoTokenizer

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)

def build_transform(input_size):
    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
    transform = T.Compose([
        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=MEAN, std=STD)
    ])
    return transform

def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
    best_ratio_diff = float('inf')
    best_ratio = (1, 1)
    area = width * height
    for ratio in target_ratios:
        target_aspect_ratio = ratio[0] / ratio[1]
        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
        if ratio_diff < best_ratio_diff:
            best_ratio_diff = ratio_diff
            best_ratio = ratio
        elif ratio_diff == best_ratio_diff:
            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
                best_ratio = ratio
    return best_ratio

def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
    orig_width, orig_height = image.size
    aspect_ratio = orig_width / orig_height

    # calculate the existing image aspect ratio
    target_ratios = set(
        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
        i * j <= max_num and i * j >= min_num)
    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])

    # find the closest aspect ratio to the target
    target_aspect_ratio = find_closest_aspect_ratio(
        aspect_ratio, target_ratios, orig_width, orig_height, image_size)

    # calculate the target width and height
    target_width = image_size * target_aspect_ratio[0]
    target_height = image_size * target_aspect_ratio[1]
    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]

    # resize the image
    resized_img = image.resize((target_width, target_height))
    processed_images = []
    for i in range(blocks):
        box = (
            (i % (target_width // image_size)) * image_size,
            (i // (target_width // image_size)) * image_size,
            ((i % (target_width // image_size)) + 1) * image_size,
            ((i // (target_width // image_size)) + 1) * image_size
        )
        # split the image
        split_img = resized_img.crop(box)
        processed_images.append(split_img)
    assert len(processed_images) == blocks
    if use_thumbnail and len(processed_images) != 1:
        thumbnail_img = image.resize((image_size, image_size))
        processed_images.append(thumbnail_img)
    return processed_images

def load_image(image_file, input_size=448, max_num=12):
    image = Image.open(image_file).convert('RGB')
    transform = build_transform(input_size=input_size)
    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
    pixel_values = [transform(image) for image in images]
    pixel_values = torch.stack(pixel_values)
    return pixel_values

model = AutoModel.from_pretrained(
    "5CD-AI/Vintern-1B-v2",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True,
).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained("5CD-AI/Vintern-1B-v2", trust_remote_code=True, use_fast=False)

test_image = 'test-image.jpg'

pixel_values = load_image(test_image, max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens= 1024, do_sample=False, num_beams = 3, repetition_penalty=2.5)

question = '<image>\nMô tả hình ảnh một cÑch chi tiết.'

response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')

#question = "CÒu hỏi khÑc ......"
#response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
#print(f'User: {question}\nAssistant: {response}')

Finetune on your Data

Open In Colab

Citation

@misc{doan2024vintern1befficientmultimodallarge,
      title={Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese}, 
      author={Khang T. Doan and Bao G. Huynh and Dung T. Hoang and Thuc D. Pham and Nhat H. Pham and Quan T. M. Nguyen and Bang Q. Vo and Suong N. Hoang},
      year={2024},
      eprint={2408.12480},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2408.12480}, 
}

References

[1] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).

[2] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Chen, Zhe, et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites." arXiv preprint arXiv:2404.16821 (2024).

[4] Tran, Chi, and Huong Le Thanh. "LaVy: Vietnamese Multimodal Large Language Model." arXiv preprint arXiv:2404.07922 (2024).

Downloads last month
14
Safetensors
Model size
938M params
Tensor type
BF16
Β·
Inference Examples
Inference API (serverless) does not yet support model repos that contain custom code.

Model tree for tt1225/Vintern-1B-v2-Custom

Finetuned
(2)
this model

Datasets used to train tt1225/Vintern-1B-v2-Custom