README.md

---
library_name: peft
license: gemma
base_model: vidore/colpali-v1.3-hf
tags:
- colpali
- turkish
- TR
datasets:
- selimc/tr-textbook-ColPali
- muhammetfatihaktug/bilim_teknik_mini_colpali
language:
- tr
pipeline_tag: visual-document-retrieval
model-index:
- name: turkish-colpali
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# turkish-colpali

This model is a fine-tuned version of [vidore/colpali-v1.3-hf](https://huggingface.co/vidore/colpali-v1.3-hf) on these datasets:
- [selimc/tr-textbook-ColPali](https://huggingface.co/datasets/selimc/tr-textbook-ColPali)
- [muhammetfatihaktug/bilim_teknik_mini_base_colpali](https://huggingface.co/datasets/muhammetfatihaktug/bilim_teknik_mini_colpali)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/65281302cad797fc4abeffd7/bs8zGLYCYPrjCs8JdsmjA.png)

## Model description

> ColPali is a model based on a novel model architecture and training strategy based on Vision Language Models (VLMs) to efficiently index documents from their visual features. It is a PaliGemma-3B extension that generates ColBERT- style multi-vector representations of text and images. It was introduced in the paper [ColPali: Efficient Document Retrieval with Vision Language Models](https://huggingface.co/papers/2407.01449). 

## Intended uses & limitations

This model is primarily designed for efficient indexing and retrieval of Turkish documents by leveraging both textual and visual features. While traditional RAG systems are limited to text-only retrieval, this model extends RAG capabilities by enabling both textual and visual retrieval, making it particularly effective for applications where visual context is as important as textual content. The model performs best with well-structured Turkish PDF like documents.

## Training and evaluation data

The training data was created via the following steps:
- Downloading PDF files of Turkish textbooks and science magazines that are publicly available on the internet.
- Using the [pdf-to-page-images-dataset](https://huggingface.co/spaces/Dataset-Creation-Tools/pdf-to-page-images-dataset) Space to convert the PDF documents into a single page image dataset
- Use `gemini-2.0-flash-exp` to generate synthetic queries for these documents using the approach outlined [here](https://danielvanstrien.xyz/posts/post-with-code/colpali/2024-09-23-generate_colpali_dataset.html) with additional modifications. This results in [selimc/tr-textbook-ColPali](https://huggingface.co/datasets/selimc/tr-textbook-ColPali) and [muhammetfatihaktug/bilim_teknik_mini_base_colpali](https://huggingface.co/datasets/muhammetfatihaktug/bilim_teknik_mini_colpali).
- Train the model using the fine tuning [notebook](https://github.com/merveenoyan/smol-vision/blob/main/Finetune_ColPali.ipynb?s=35) from [Merve Noyan](https://huggingface.co/merve). Data processing step was modified to include all 3 types of queries. This approach not only adds variety to the training data but also effectively triples the dataset size, helping the model learn to handle diverse query types.

## Usage

The complete code for fine-tuning, testing, and creating similarity maps can be found in the [turkish-colpali GitHub repository](https://github.com/selimcavas/turkish-colpali). All notebooks in the repository are in Turkish to better serve the Turkish NLP community.

```python
from io import BytesIO
from typing import List
import requests
import torch
from IPython.display import display
from PIL import Image
from transformers import ColPaliForRetrieval, ColPaliProcessor

model_name = "selimc/turkish-colpali"
model = ColPaliForRetrieval.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="cuda:0",  # or "mps" if on Apple Silicon
).eval()

processor = ColPaliProcessor.from_pretrained(model_name)

def load_image_from_url(url: str) -> Image.Image:
    """
    Load a PIL image from a valid URL.
    """
    response = requests.get(url)
    return Image.open(BytesIO(response.content))

# Your inputs
images: List[Image.Image] = [
    load_image_from_url(
        "https://ogmmateryal.eba.gov.tr/panel/upload/etki/6305/193.jpg"
    ),
    load_image_from_url(
        "https://ogmmateryal.eba.gov.tr/panel/upload/etki/4726/126.jpg"
    ),
    load_image_from_url(
        "https://ogmmateryal.eba.gov.tr/panel/upload/etki/5105/281.jpg"
    ),
    load_image_from_url(
        "https://ogmmateryal.eba.gov.tr/panel/upload/etki/6336/111.jpg"
    ),
]

queries: List[str] = [
    "ekonomiyi düzeltme çabaları demir yolları gelir gider grafik",
    "bitkilerin yapısı bitkisel dokular meristem doku",
    "besin grupları tablosu karbonhidratlar",
    "Türk milli mücadelesi emperyalizm Atatürk görseli"
]

# Preprocess inputs
batch_images = processor(images=images).to(model.device)
batch_queries = processor(text=queries).to(model.device)

# Forward pass
with torch.no_grad():
    image_embeddings = model(**batch_images).embeddings
    query_embeddings = model(**batch_queries).embeddings

scores = processor.score_retrieval(query_embeddings, image_embeddings)  # (n_queries, n_images)

scores

# tensor([[19.0000, 14.5625, 15.3125, 16.5000],
#        [15.5625, 20.2500, 17.8750, 15.7500],
#        [12.4375, 14.0625, 18.7500, 11.9375],
#        [15.0625, 13.4375, 12.8125, 20.8750]], dtype=torch.bfloat16)
```


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- num_epochs: 1

### Framework versions

- PEFT 0.11.1
- Transformers 4.48.0.dev0
- Pytorch 2.4.1+cu124
- Datasets 2.21.0
- Tokenizers 0.21.0

## Contact Us

- [Selim Çavaş](https://www.linkedin.com/in/selimcavas/)
- [Muhammet Fatih Aktuğ](https://www.linkedin.com/in/muhammetfatihaktug/)

## Citation

```bibtex
@misc{turkish-colpali,
  author = {Selim Çavaş & Muhammet Fatih Aktuğ},
  title = {turkish-colpali: A Fine-tuned ColPali Model for Turkish Document Retrieval},
  year = {2024},
  url = {https://huggingface.co/selimc/turkish-colpali}
}