Report for nateraw/bert-base-uncased-emotion
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 2 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset dair-ai/emotion (subset split, split validation).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.182 | Add typos | 182/1000 tested samples (18.2%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 18.2% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1486 | i feel as though my descriptions are skimmable and unimportant | i rfeel as thoughm y descrpitions are skimmable nd jnimportant | sadness (p = 0.98) | joy (p = 0.98) |
| 99 | im not trying to sound sarcastic but only trying to make the point that amid the daily pressures of life as wife and mom we often may find ourselves feeling kind of unimportant or robotic if you will in carrying out our tasks | im not trying t sojnd asrfastic but only trying to make the point that amid the daily pressures of life as wife and mom we often may find ourselves feelkng kind of hnimportant or robotjic if you will in carryig uot ourt asks | sadness (p = 0.99) | joy (p = 0.99) |
| 392 | i remember feeling disheartened one day when we were studying a poem really dissecting it verse by verse stanza by stanza | i remember feeling cishezrtened one day when we were studying a poem really dissecting ot vefse by verse stajza by stanza | sadness (p = 1.00) | joy (p = 0.42) |
👉Performance issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "know" |
Precision = 0.876 | — | -6.45% than global |
🔍✨Examples
For records in the dataset where `text` contains "know", the Precision is 6.45% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 17 | i know what it feels like he stressed glaring down at her as she squeezed more soap onto her sponge | anger | sadness (p = 0.89) |
| 91 | i feel like the people i know are really generous and i have my needs met | joy | love (p = 0.68) |
| 164 | i have stayed at heritage christian because of the fulfillment that i feel in doing christ s work in action by being the hands the eyes the legs and the voice of supporting the individuals that i have been blessed to know and support | joy | love (p = 0.61) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!