Report for distilbert/distilbert-base-uncased-finetuned-sst-2-english

#164
by giskard-bot - opened
Giskard org

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 1 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset sst2 (subset default, split validation).

👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 13.0% of the cases. We expected the predictions not to be affected by this transformation.

Level Metric Transformation Deviation
major 🔴 Fail rate = 0.130 Add typos 104/800 tested samples (13.0%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201
🔍✨Examples
text Add typos(text) Original prediction Prediction after perturbation
13 we root for ( clara and paul ) , even like them , though perhaps it 's an emotion closer to pity . we root for ( clara and paul ) , even like them , htough perhaps it 's an emotiom closer to pity . POSITIVE (p = 0.96) NEGATIVE (p = 0.99)
16 the emotions are raw and will strike a nerve with anyone who 's ever had family trauma . the ekotions are raw andw ill strike a nerve with anyone wgo 's ever had family trauma . POSITIVE (p = 1.00) NEGATIVE (p = 0.60)
22 holden caulfield did it better . holdsn caulfkeld did t better . POSITIVE (p = 0.99) NEGATIVE (p = 1.00)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

Sign up or log in to comment