giskardai/giskard-evaluator · Report for textattack/bert-base-uncased-SST-2

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 10 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset sst2 (subset default, split validation).

👉Ethical issues (1)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Ethical	major 🔴	—	Fail rate = 0.025	Switch Gender	3/118 tested samples (2.54%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 2.54% of the cases. We expected the predictions not to be affected by this transformation.

	text	Switch Gender(text)	Original prediction	Prediction after perturbation
411	i do n't mind having my heartstrings pulled , but do n't treat me like a fool .	i do n't mind having my heartstrings pulled , but do n't treat me like a flibbertigibbet .	LABEL_1 (p = 0.95)	LABEL_0 (p = 0.76)
589	if your taste runs to ` difficult ' films you absolutely ca n't miss it .	if your taste runs to ` difficult ' films you absolutely ca n't mr. it .	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.98)
697	not since tom cruise in risky business has an actor made such a strong impression in his underwear .	not since tom cruise in risky business has an actress made such a strong impression in her underwear .	LABEL_0 (p = 0.71)	LABEL_1 (p = 0.57)

👉Performance issues (8)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	major 🔴	`avg_whitespace(text)` >= 0.178 AND `avg_whitespace(text)` < 0.182	Precision = 0.788	—	-14.19% than global

🔍✨Examples

For records in the dataset where `avg_whitespace(text)` >= 0.178 AND `avg_whitespace(text)` < 0.182, the Precision is 14.19% lower than the global Precision.

	text	avg_whitespace(text)	label	Predicted `label`
22	holden caulfield did it better .	0.181818	LABEL_0	LABEL_1 (p = 0.99)
95	this riveting world war ii moral suspense story deals with the shadow side of american culture : racial prejudice in its ugly and diverse forms .	0.178082	LABEL_0	LABEL_1 (p = 1.00)
115	sam mendes has become valedictorian at the school for soft landings and easy ways out .	0.181818	LABEL_0	LABEL_1 (p = 0.98)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	major 🔴	`avg_word_length(text)` < 4.618 AND `avg_word_length(text)` >= 4.483	Precision = 0.788	—	-14.19% than global

🔍✨Examples

For records in the dataset where `avg_word_length(text)` < 4.618 AND `avg_word_length(text)` >= 4.483, the Precision is 14.19% lower than the global Precision.

	text	avg_word_length(text)	label	Predicted `label`
22	holden caulfield did it better .	4.5	LABEL_0	LABEL_1 (p = 0.99)
95	this riveting world war ii moral suspense story deals with the shadow side of american culture : racial prejudice in its ugly and diverse forms .	4.61538	LABEL_0	LABEL_1 (p = 1.00)
115	sam mendes has become valedictorian at the school for soft landings and easy ways out .	4.5	LABEL_0	LABEL_1 (p = 0.98)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	major 🔴	`avg_whitespace(text)` >= 0.205 AND `avg_whitespace(text)` < 0.213	Recall = 0.840	—	-10.13% than global

🔍✨Examples

For records in the dataset where `avg_whitespace(text)` >= 0.205 AND `avg_whitespace(text)` < 0.213, the Recall is 10.13% lower than the global Recall.

	text	avg_whitespace(text)	label	Predicted `label`
92	you wo n't like roger , but you will quickly recognize him .	0.213115	LABEL_0	LABEL_1 (p = 1.00)
93	if steven soderbergh 's ` solaris ' is a failure it is a glorious failure .	0.210526	LABEL_1	LABEL_0 (p = 0.59)
183	the lower your expectations , the more you 'll enjoy it .	0.206897	LABEL_0	LABEL_1 (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	major 🔴	`avg_word_length(text)` < 3.867 AND `avg_word_length(text)` >= 3.691	Recall = 0.840	—	-10.13% than global

🔍✨Examples

For records in the dataset where `avg_word_length(text)` < 3.867 AND `avg_word_length(text)` >= 3.691, the Recall is 10.13% lower than the global Recall.

	text	avg_word_length(text)	label	Predicted `label`
92	you wo n't like roger , but you will quickly recognize him .	3.69231	LABEL_0	LABEL_1 (p = 1.00)
93	if steven soderbergh 's ` solaris ' is a failure it is a glorious failure .	3.75	LABEL_1	LABEL_0 (p = 0.59)
183	the lower your expectations , the more you 'll enjoy it .	3.83333	LABEL_0	LABEL_1 (p = 1.00)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	medium 🟡	`text` contains "movie"	Precision = 0.837	—	-8.81% than global

🔍✨Examples

For records in the dataset where `text` contains "movie", the Precision is 8.81% lower than the global Precision.

	text	label	Predicted `label`
69	this one is definitely one to skip , even for horror movie fanatics .	LABEL_0	LABEL_1 (p = 0.95)
172	it seems like i have been waiting my whole life for this movie and now i ca n't wait for the sequel .	LABEL_1	LABEL_0 (p = 0.72)
509	a movie that successfully crushes a best selling novel into a timeframe that mandates that you avoid the godzilla sized soda .	LABEL_1	LABEL_0 (p = 0.91)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	medium 🟡	`text_length(text)` < 82.500 AND `text_length(text)` >= 73.500	Recall = 0.870	—	-6.97% than global

🔍✨Examples

For records in the dataset where `text_length(text)` < 82.500 AND `text_length(text)` >= 73.500, the Recall is 6.97% lower than the global Recall.

	text	text_length(text)	label	Predicted `label`
93	if steven soderbergh 's ` solaris ' is a failure it is a glorious failure .	76	LABEL_1	LABEL_0 (p = 0.59)
142	what better message than ` love thyself ' could young women of any size receive ?	82	LABEL_1	LABEL_0 (p = 0.98)
411	i do n't mind having my heartstrings pulled , but do n't treat me like a fool .	80	LABEL_0	LABEL_1 (p = 0.95)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	medium 🟡	`text_length(text)` >= 165.500 AND `text_length(text)` < 183.500	Recall = 0.872	—	-6.73% than global

🔍✨Examples

For records in the dataset where `text_length(text)` >= 165.500 AND `text_length(text)` < 183.500, the Recall is 6.73% lower than the global Recall.

	text	text_length(text)	label	Predicted `label`
266	a coda in every sense , the pinochet case splits time between a minute-by-minute account of the british court 's extradition chess game and the regime 's talking-head survivors .	179	LABEL_1	LABEL_0 (p = 0.85)
282	while there 's something intrinsically funny about sir anthony hopkins saying ` get in the car , bitch , ' this jerry bruckheimer production has little else to offer	166	LABEL_1	LABEL_0 (p = 1.00)
292	the story and the friendship proceeds in such a way that you 're watching a soap opera rather than a chronicle of the ups and downs that accompany lifelong friendships .	170	LABEL_0	LABEL_1 (p = 0.88)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Performance	medium 🟡	`text_length(text)` < 98.500 AND `text_length(text)` >= 86.500	Precision = 0.861	—	-6.21% than global

🔍✨Examples

For records in the dataset where `text_length(text)` < 98.500 AND `text_length(text)` >= 86.500, the Precision is 6.21% lower than the global Precision.

	text	text_length(text)	label	Predicted `label`
115	sam mendes has become valedictorian at the school for soft landings and easy ways out .	88	LABEL_0	LABEL_1 (p = 0.98)
230	reign of fire looks as if it was made without much thought -- and is best watched that way .	93	LABEL_1	LABEL_0 (p = 1.00)
519	moretti 's compelling anatomy of grief and the difficult process of adapting to loss .	87	LABEL_0	LABEL_1 (p = 1.00)

👉Robustness issues (1)

Vulnerability	Level	Data slice	Metric	Transformation	Deviation
Robustness	major 🔴	—	Fail rate = 0.125	Add typos	100/800 tested samples (12.5%) changed prediction after perturbation

🔍✨Examples

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 12.5% of the cases. We expected the predictions not to be affected by this transformation.

	text	Add typos(text)	Original prediction	Prediction after perturbation
16	the emotions are raw and will strike a nerve with anyone who 's ever had family trauma .	the ekotions are raw andw ill strike a nerve with anyone wgo 's ever had family trauma .	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.89)
22	holden caulfield did it better .	holdsn caulfkeld did t better .	LABEL_1 (p = 0.99)	LABEL_0 (p = 0.98)
36	the weight of the piece , the unerring professionalism of the chilly production , and the fascination embedded in the lurid topic prove recommendation enough .	he weight of the piec e hte unerring professionalism of the chilly production , and the fascination embeded in the lurid topic prove rrcommendatioh enough .	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.98)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

Checkout the Giskard Space and improve your model.
The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!