Report for bhadresh-savani/distilbert-base-uncased-emotion
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 2 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset dair-ai/emotion (subset split, split validation).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.215 | Add typos | 215/1000 tested samples (21.5%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 21.5% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 781 | i feel that im most amazed still by silent knight which is an instrumental song ala hizaki | i feel tat im osr amazec still by silent knight which is an instrumental song ala hizaki | surprise (p = 0.99) | joy (p = 0.97) |
| 632 | i feel he is an terrific really worth bet | i fel he is an erridic realoy qworth bet | joy (p = 1.00) | anger (p = 0.63) |
| 181 | i feel like waiting for you to be online and you didnt makes me furious | i feel like waiting for you o be oline and you ddnt makes me furius | anger (p = 1.00) | joy (p = 0.98) |
👉Performance issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "know" |
Precision = 0.885 | — | -5.25% than global |
🔍✨Examples
For records in the dataset where `text` contains "know", the Precision is 5.25% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 17 | i know what it feels like he stressed glaring down at her as she squeezed more soap onto her sponge | anger | sadness (p = 0.98) |
| 91 | i feel like the people i know are really generous and i have my needs met | joy | love (p = 0.80) |
| 164 | i have stayed at heritage christian because of the fulfillment that i feel in doing christ s work in action by being the hands the eyes the legs and the voice of supporting the individuals that i have been blessed to know and support | joy | love (p = 0.82) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!