Report for cardiffnlp/twitter-roberta-base-irony

#10
by giskard-bot - opened

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 2 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset tweet_eval (subset irony, split validation).

👉Performance issues (2)
Vulnerability Level Data slice Metric Transformation Deviation
Performance major 🔴 text contains "love" Accuracy = 0.333 -36.21% than global
🔍✨Examples For records in the dataset where `text` contains "love", the Accuracy is 36.21% lower than the global Accuracy.
text label Predicted label
120 Dont we all just love those people who message you out of nowhere and act like you guys are close cus they want something from you? irony non_irony (p = 0.86)
133 I love waking up at 4am everyday and then running 5 miles.. favorite part of the Army :) irony non_irony (p = 0.99)
155 I love how today is going irony non_irony (p = 0.98)
Vulnerability Level Data slice Metric Transformation Deviation
Performance medium 🟡 text contains "day" Accuracy = 0.490 -6.26% than global
🔍✨Examples For records in the dataset where `text` contains "day", the Accuracy is 6.26% lower than the global Accuracy.
text label Predicted label
41 Oh, thank GOD - our entire office email system is down... the day of a big event. Santa, you know JUST what to get me for xmas. irony non_irony (p = 0.93)
62 First day skiing #killedit #iwasshit #tignes #seasonlife irony non_irony (p = 0.72)
121 LOL..It was an RSS agenda modi bribed UN officals to make yoga day international.. #secularindia #AntiConversionLaw #hypocrites irony non_irony (p = 0.98)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

  • Checkout the Giskard Space and improve your model.
  • The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!

Sign up or log in to comment