giskardai/giskard-evaluator · Report for cardiffnlp/twitter-roberta-base-offensive

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 3 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset tweet_eval (subset offensive, split test).

👉Performance issues (3)

For records in the dataset where text contains "antifa", the Recall is 50.0% lower than the global Recall.

Level	Data slice	Metric	Deviation
major 🔴	`text` contains "antifa"	Recall = 0.333	-50.00% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
1	@user @user @user I got in a pretty deep debate with my friend and she told me that latinos for Trump and blacks for Trump were paid supporters 😂 then I said you mean antifa are paid domestic terrorist and she said No they are anti-fascist then I said they are the fascist are you kidding me?!	non-offensive	offensive (p = 0.65)
148	#Antifa protest in #Athens for the 5 years after the murder of Pavlos Fyssas by Golden Dawn. Their voters are complicit to their crimes. #KillahP	offensive	non-offensive (p = 0.55)
189	@user @user @user Fascism was also against liberals as is Antifa according to your meme.	offensive	non-offensive (p = 0.80)

For records in the dataset where text contains "conservatives", the Recall is 31.82% lower than the global Recall.

Level	Data slice	Metric	Deviation
major 🔴	`text` contains "conservatives"	Recall = 0.455	-31.82% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
35	#ConstitutionDay It's very odd for the alt right conservatives to say that we are ruining the constitution just because we want #GunControlNow but they are the ones ruining the constitution getting upset because foreigners are coming to this land who are not White wanting to live	offensive	non-offensive (p = 0.58)
62	#Conservatives We mus pray for liberals. They trooly kno knot watt they do. I was brot up socialist democrat, by my wonderful, but, ignorant parents. I served inda military. I learnd I can serve Christ, too. Eye reel -eyesed CONSERVATIVISM was more closely aligned w God.	offensive	non-offensive (p = 0.68)
80	#BlueWave #DumpTrump …give my love 2, or cooperate/compromise w/, conservatives/Trump supporters who constantly vilify marginalized groups, who constantly attack women’s rights 2 make choices about their own bodies,…	offensive	non-offensive (p = 0.62)

For records in the dataset where text contains "trump", the Balanced Accuracy is 10.74% lower than the global Balanced Accuracy.

Level	Data slice	Metric	Deviation
major 🔴	`text` contains "trump"	Balanced Accuracy = 0.714	-10.74% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
1	@user @user @user I got in a pretty deep debate with my friend and she told me that latinos for Trump and blacks for Trump were paid supporters 😂 then I said you mean antifa are paid domestic terrorist and she said No they are anti-fascist then I said they are the fascist are you kidding me?!	non-offensive	offensive (p = 0.65)
80	#BlueWave #DumpTrump …give my love 2, or cooperate/compromise w/, conservatives/Trump supporters who constantly vilify marginalized groups, who constantly attack women’s rights 2 make choices about their own bodies,…	offensive	non-offensive (p = 0.62)
122	#America ... tear down that #Wall! #tcot #partisanship #Trump #thewall #Borderwall #liberty #civilsociety #think #Conservatives #Democrats #Progressives #liberals #Independent #libertarians #GOP #DNC #CriticalThinking	offensive	non-offensive (p = 0.90)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.