Model Card for Qwen2.5-0.5B-PEFT-On-Quantum-Computer

This model is a hybrid classical-quantum text classifier built by fine-tuning Qwen2.5-0.5B with a custom Quantum LoRA (Q-LoRA) Head. The parameter-efficient fine-tuning (PEFT) is performed using QMeZO (Quantum Memory-Efficient Zeroth-Order Optimization), a gradient-free optimization technique tailored for hybrid quantum-classical systems.

The quantum component is integrated using the PennyLane framework, featuring a 4-qubit variational quantum circuit (VQC) that processes low-rank text representations before classification.

Model Details

• Developed by: mingqiancha233
• Model Type: Hybrid Classical-Quantum Causal Language Model with Classification Head
• Base Model: Qwen/Qwen2.5-0.5B
• Quantum Framework: PennyLane
• Optimization Method: QMeZO (Quantum Zeroth-Order Optimization)
• Language(s): English
• License: Apache-2.0
• Finetuned from model: Qwen/Qwen2.5-0.5B
• Task: Multi-class Text Classification (AG News: World, Sports, Business, Sci/Tech)

---

Model Description

This model explores the intersection of Quantum Machine Learning (QML) and Large Language Models (LLMs). Instead of traditional classical fine-tuning, we freeze the backbone of Qwen2.5-0.5B and introduce a Quantum LoRA Head that maps classical hidden states into a quantum Hilbert space for high-dimensional feature interaction.

Quantum Architecture

1. Dimensionality Reduction (Down-projection): The pooled hidden states from Qwen ($d_{model} = 896$) are projected down to a low-rank subspace ($r = 16$).
2. Quantum State Preparation (Encoding): The 16-dimensional classical vector is mapped to a 4-qubit quantum state using $RY$ rotation gates.
3. Variational Quantum Circuit (VQC):
   • A 2-layer parameterized circuit consisting of arbitrary single-qubit rotations ($Rot$ gates with 3 parameters per qubit).
   • Entanglement is introduced using circular $CNOT$ gates to capture complex inter-token dependencies.
4. Measurement: Expectation values of the $PauliZ$ operator are measured on all 4 qubits.
5. Up-projection & Classification: The quantum measurements are projected back to the hidden dimension, added as a residual connection to the classical features, and passed to the final classification layer.

---

Intended Uses & Limitations

Intended Uses

• Research in Hybrid Quantum-Classical NLP (QNLP).
• Text classification tasks (specifically 4-class news classification).
• Benchmarking zeroth-order optimization (QMeZO) on quantum-enhanced neural networks.

Limitations

• The quantum head is designed for sequence-level classification and is not suited for generative tasks in its current head-configuration.
• Requires pennylane and torch to run the hybrid forward pass.

---

Training Details

Training Dataset

• Dataset: ag_news (fancyzhx/ag_news backup)
• Classes: 4 (World, Sports, Business, Sci/Tech)
• Sequence Length: 64 tokens (with mean pooling over active tokens)

Training Hyperparameters

• Optimizer: QMeZO (Zeroth-Order Gradient-Free)
• Learning Rate ($lr$): 5e-4
• ZO Perturbation ($\epsilon$): 1e-3
• Batch Size: 1
• Max Steps: 400
• LoRA Rank ($r$): 16
• Number of Qubits: 4
• Quantum Layers: 2
• Precision: Float32

---

How to Use

To run inference or utilize this hybrid model, you need to define the quantum-classical network architecture and load the trained weights.

Prerequisites

pip install torch transformers pennylane

Inference Code

import torch
import pennylane as qml
import torch.nn as nn
from transformers import AutoTokenizer, AutoModelForCausalLM

# 1. Define the Quantum Layer and Head (matching the training configuration)
class QuantumLayer(nn.Module):
    def __init__(self, input_dim=16, n_qubits=4, q_layers=2):
        super().__init__()
        self.dev = qml.device("default.qubit", wires=n_qubits)
        self.q_params = nn.Parameter(torch.zeros(q_layers, n_qubits, 3))
        self.input_proj = nn.Linear(input_dim, n_qubits)
        self.output_proj = nn.Linear(n_qubits, input_dim)

        @qml.qnode(self.dev, interface="torch", diff_method=None)
        def circuit(x, weights):
            for i in range(n_qubits):
                qml.RY(x[i], wires=i)
            for l in range(q_layers):
                for i in range(n_qubits):
                    qml.Rot(weights[l, i, 0], weights[l, i, 1], weights[l, i, 2], wires=i)
                for i in range(n_qubits - 1):
                    qml.CNOT(wires=[i, i + 1])
                if n_qubits > 1:
                    qml.CNOT(wires=[n_qubits - 1, 0])
            return [qml.expval(qml.PauliZ(i)) for i in range(n_qubits)]
        
        self.circuit = circuit

    def forward(self, x):
        x_proj = torch.tanh(self.input_proj(x)) * 3.14159265
        outputs = []
        for i in range(x_proj.size(0)):
            q_out = torch.stack(self.circuit(x_proj[i], self.q_params))
            outputs.append(q_out)
        return self.output_proj(torch.stack(outputs, dim=0))

class QuantumLoRAHead(nn.Module):
    def __init__(self, hidden_size, num_labels=4, rank=16):
        super().__init__()
        self.norm = nn.LayerNorm(hidden_size)
        self.down = nn.Linear(hidden_size, rank)
        self.act1 = nn.Tanh()
        self.quantum = QuantumLayer(input_dim=rank)
        self.act2 = nn.Tanh()
        self.up = nn.Linear(rank, hidden_size)
        self.classifier = nn.Linear(hidden_size, num_labels)

    def forward(self, pooled_hidden):
        h = self.norm(pooled_hidden)
        delta = self.up(self.act2(self.quantum(self.act1(self.down(h)))))
        return self.classifier(h + delta)

# 2. Load Base Model and Tokenizer
model_name = "Qwen/Qwen2.5-0.5B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model = AutoModelForCausalLM.from_pretrained(model_name)

# 3. Initialize Head and Load Weights
head = QuantumLoRAHead(hidden_size=base_model.config.hidden_size, num_labels=4)
# Download your 'quantum_lora_head_step_XXX.pt' and load it:
# head.load_state_dict(torch.load("quantum_lora_head_step_400.pt")) 
head.eval()

# 4. Run Inference
text = "The championship game ended with a thrilling overtime victory."
inputs = tokenizer(text, return_tensors="pt", max_length=64, padding="max_length", truncation=True)

with torch.no_grad():
    outputs = base_model.model(input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"])
    # Mean pooling
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    pooled = (outputs.last_hidden_state * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-6)
    
    logits = head(pooled)
    predicted_class = torch.argmax(logits, dim=-1).item()

classes = ["World", "Sports", "Business", "Sci/Tech"]
print(f"Predicted Category: {classes[predicted_class]}")

---

Citation & Acknowledgements

If you use this model in your research, please cite the base Qwen2.5 model and acknowledge the use of PennyLane for quantum circuit integration.

• Qwen2.5: Alibaba Cloud State-of-the-Art LLMs
• PennyLane: Xanadu Quantum Technologies