lbux/cga-deberta-onnx-int8

DeBERTa Toxicity Classifier (ONNX INT8)

Usage

Install dependencies

pip install torch transformers optimum onnxruntime

---

Load model

import torch
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model_path = "models/cga_deberta_onnx_int8"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = ORTModelForSequenceClassification.from_pretrained(
    model_path,
    file_name="model_quantized.onnx"
)

---

Run inference

def predict(text: str) -> float:
    inputs = tokenizer(
        text,
        return_tensors="pt",
        padding=True,
        truncation=True
    )

    with torch.no_grad():
        outputs = model(**inputs)

    probs = torch.softmax(outputs.logits, dim=-1)
    toxicity_score = probs[0][1].item()

    return toxicity_score


# Example
print(predict("This is an example sentence."))

---

Output

• Returns a toxicity score between 0 and 1
• Higher = more likely to be toxic

---