sophy/finetuned-qwen-referrals

Uploaded finetuned model

• Developed by: sophy
• License: apache-2.0
• Finetuned from model : unsloth/Qwen3-VL-8B-Instruct-unsloth-bnb-4bit

This qwen3_vl model was trained 2x faster with Unsloth and Huggingface's TRL library.

How to use

This model is a vision-language model fine-tuned from Qwen3-VL using Unsloth to extract structured JSON from referral forms.

Install

pip install "unsloth>=2024.10.0" transformers accelerate bitsandbytes pillow huggingface_hub

### Usage (Unsloth – recommended)

The simplest way to use this model is via Unsloth’s FastVisionModel:

```bash
        from unsloth import FastVisionModel
        from PIL import Image
        import torch
        
        MODEL_REPO = "sophy/finetuned-qwen-referrals"
        
        # Load model in 4-bit for lower memory use
        model, tokenizer = FastVisionModel.from_pretrained(
            MODEL_REPO,
            load_in_4bit = True,
            trust_remote_code = True,
        )
        
        # Puts the model in inference mode (disables grad, etc.)
        FastVisionModel.for_inference(model)
        
        image_path = "test.png"   # referral form image
        image = Image.open(image_path).convert("RGB")
        
        messages = [
            {
                "role": "user",
                "content": [
                    {"type": "image", "image": image},
                    {
                        "type": "text",
                        "text": (
                            "Extract all fields and return JSON."
                        ),
                    },
                ],
            }
        ]
        
        # Build a chat-style prompt using the model's chat template
        prompt = tokenizer.apply_chat_template(
            messages,
            add_generation_prompt = True,
            tokenize              = False,
        )
        
        # Tokenise image + prompt together
        inputs = tokenizer(
            image,
            prompt,
            return_tensors = "pt",
        )
        
        # Move to the model's device
        inputs = {k: v.to(model.device) for k, v in inputs.items()}
        
        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens = 2000,
                temperature    = 0.1,
                top_p          = 0.9,
            )
        
        result = tokenizer.decode(outputs[0], skip_special_tokens=True)
        print(result)
    ```

### Usage (pure 🤗 Transformers)

If you prefer to use standard Transformers without Unsloth, you can load the model as a Qwen3-VL vision–language model. A typical pattern is:

``` bash

    from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
    from PIL import Image
    import torch
    
    MODEL_REPO = "sophy/finetuned-qwen-referrals"
    
    processor = AutoProcessor.from_pretrained(
        MODEL_REPO,
        trust_remote_code = True,
    )
    
    model = Qwen3VLForConditionalGeneration.from_pretrained(
        MODEL_REPO,
        torch_dtype       = torch.bfloat16,  # or "auto"
        device_map        = "auto",
        trust_remote_code = True,
    )
    
    image = Image.open("test.png").convert("RGB")
    
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": image},
                {
                    "type": "text",
                    "text": (
                        "Extract all fields and return JSON."
                    ),
                },
            ],
        }
    ]
    
    prompt = processor.tokenizer.apply_chat_template(
        messages,
        add_generation_prompt = True,
        tokenize              = False,
    )
    
    inputs = processor(
        text   = prompt,
        images = image,
        return_tensors = "pt"
    ).to(model.device)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens = 2000,
            temperature    = 0.1,
            top_p          = 0.9,
        )
    
    result = processor.tokenizer.decode(outputs[0], skip_special_tokens=True)
    print(result)