giskardai/giskard-evaluator · Report for textattack/bert-base-uncased-SST-2

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 1 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset sst2 (subset default, split train).

👉Robustness issues (1)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.143	Add typos	143/1000 tested samples (14.3%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 14.3% of the cases. We expected the predictions not to be affected by this transformation.

	text	Add typos(text)	Original prediction	Prediction after perturbation
22288	overflowing septic tank	overflowing eptic watnk	LABEL_0 (p = 0.99)	LABEL_1 (p = 0.74)
62547	's never a dull moment in the giant spider invasion comic chiller	's ever a dull noment in the giant spider invasiom comic chiller	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.99)
40620	the marvelous verdu ,	the msrvelous verdu ,	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.99)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

Checkout the Giskard Space and improve your model.
The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!