Report for distilbert/distilbert-base-uncased-finetuned-sst-2-english
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 1 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset sst2 (subset default, split validation).
You can find a full version of scan report here.
👉Robustness issues (1)
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 13.0% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| major 🔴 | Fail rate = 0.130 | Add typos | 104/800 tested samples (13.0%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 13 | we root for ( clara and paul ) , even like them , though perhaps it 's an emotion closer to pity . | we root for ( clara and paul ) , even like them , htough perhaps it 's an emotiom closer to pity . | POSITIVE (p = 0.96) | NEGATIVE (p = 0.99) |
| 16 | the emotions are raw and will strike a nerve with anyone who 's ever had family trauma . | the ekotions are raw andw ill strike a nerve with anyone wgo 's ever had family trauma . | POSITIVE (p = 1.00) | NEGATIVE (p = 0.60) |
| 22 | holden caulfield did it better . | holdsn caulfkeld did t better . | POSITIVE (p = 0.99) | NEGATIVE (p = 1.00) |
Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.