giskardai/giskard-evaluator · Report for distilbert/distilbert-base-uncased-finetuned-sst-2-english

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 1 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset sst2 (subset default, split validation).

👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 13.0% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
major 🔴	Fail rate = 0.130	Add typos	104/800 tested samples (13.0%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
13	we root for ( clara and paul ) , even like them , though perhaps it 's an emotion closer to pity .	we root for ( clara and paul ) , even like them , htough perhaps it 's an emotiom closer to pity .	POSITIVE (p = 0.96)	NEGATIVE (p = 0.99)
16	the emotions are raw and will strike a nerve with anyone who 's ever had family trauma .	the ekotions are raw andw ill strike a nerve with anyone wgo 's ever had family trauma .	POSITIVE (p = 1.00)	NEGATIVE (p = 0.60)
22	holden caulfield did it better .	holdsn caulfkeld did t better .	POSITIVE (p = 0.99)	NEGATIVE (p = 1.00)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.