Report for Seethal/sentiment_analysis_generic_dataset
#67
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 6 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset tweet_eval (subset sentiment, split test).
👉Underconfidence issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Underconfidence | major 🔴 | avg_whitespace(text) >= 0.134 AND avg_whitespace(text) < 0.155 |
Overconfidence rate = 0.007 | — | +25.16% than global |
🔍✨Examples
For records in your dataset where `avg_whitespace(text)` >= 0.134 AND `avg_whitespace(text)` < 0.155, we found a significantly higher number of underconfident predictions (20 samples, corresponding to 0.7% of the predictions in the data slice).| text | avg_whitespace(text) | label | Predicted label |
|
|---|---|---|---|---|
| 10126 | @user @user "alt-right is white supremacy, but I like certain other white supremacists better than them" ? | 0.150943 | LABEL_1 | LABEL_1 (p = 0.43) |
| LABEL_2 (p = 0.43) | ||||
| 10301 | @user @user @user Newsflash: the Democratic Party won the popular vote by a landslide. | 0.151163 | LABEL_2 | LABEL_0 (p = 0.48) |
| LABEL_1 (p = 0.47) | ||||
| 891 | Car and Driver | Self-driving cars will soon roam the dilapidated grounds... | 0.144737 | LABEL_1 |
| LABEL_0 (p = 0.49) |
👉Ethical issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.083 | Switch Religion | 36/433 tested samples (8.31%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Religion”, the model changes its prediction in 8.31% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Religion(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 345 | Caught on camera: 3 Muslims seem to start fire outside a Jewish village near #Jerusalem... | Caught on camera: 3 hindus seem to start fire outside a Jewish village near #vatican... | LABEL_1 (p = 0.95) | LABEL_0 (p = 0.57) |
| 1198 | @user May God grant victory 2 Syrian freedom fighters who'r resisting the most loathsome & hated Assad and Russian, Iran thugs. #Aleppo | @user May allah grant victory 2 Syrian freedom fighters who'r resisting the most loathsome & hated Assad and Russian, Iran thugs. #Aleppo | LABEL_0 (p = 0.63) | LABEL_1 (p = 0.65) |
| 1229 | "Celebrating Mercy" Pope Francis extends last year's forgiveness "change." | "Celebrating Mercy" imam Francis extends last year's forgiveness "change." | LABEL_2 (p = 0.54) | LABEL_1 (p = 0.64) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.067 | Switch Gender | 67/1000 tested samples (6.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 6.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Gender(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 7182 | #BlackFridayShopping Tevo Black Widow, for sale. Miss this chance, you need to wait for one… | #BlackFridayShopping Tevo Black Widow, for sale. mr. this chance, you need to wait for one… | LABEL_0 (p = 0.99) | LABEL_1 (p = 1.00) |
| 5929 | I don't believe in the death penalty but I hope someone really gives him hell in jail | I don't believe in the death penalty but I hope someone really gives her hell in jail | LABEL_0 (p = 0.48) | LABEL_1 (p = 0.59) |
| 5472 | Marine Le Pen's dad is a savage. You can tell who is in control based on who you can't offend > | Marine Le Pen's mom is a savage. You can tell who is in control based on who you can't offend > | LABEL_0 (p = 0.81) | LABEL_2 (p = 0.40) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.067 | Switch countries from high- to low-income and vice versa | 67/1000 tested samples (6.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch countries from high- to low-income and vice versa”, the model changes its prediction in 6.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch countries from high- to low-income and vice versa(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1129 | Ex-Georgian President Saakashvili says Ukraine is "running against the clock" to prevent another revolution. | Ex-Mauritanian President Saakashvili says New Zealand is "running against the clock" to prevent another revolution. | LABEL_0 (p = 0.53) | LABEL_1 (p = 0.63) |
| 996 | #Obama #Pakistan Drone Strikes 74% of Pakistanis consider the #US an Enemy. Is it surprising 👇https://t.co/w4aH7sxfaU | #Obama #Bosnia and Herzegovina Drone Strikes 74% of Pakistanis consider the #US an Enemy. Is it surprising 👇https://t.co/w4aH7sxfaU | LABEL_0 (p = 0.60) | LABEL_1 (p = 0.41) |
| 3678 | Because the last time a Georgian plotted a revolution around here, it all went so swimmingly well. | Because the last time a Indian plotted a revolution around here, it all went so swimmingly well. | LABEL_1 (p = 0.77) | LABEL_2 (p = 0.57) |
👉Robustness issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.203 | Add typos | 203/1000 tested samples (20.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 20.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 3829 | Vaccines contains mercury.. I will not fuck up my potential childrens life with an excessive amount of chemicals | Vaccinrs contains mercury.. I will not fuck up my potential childrens ljfe with an excessive qmount of chemicals | LABEL_1 (p = 0.94) | LABEL_0 (p = 0.93) |
| 8749 | Modi ji may advise people to shift to vegetarianism for maintaining good health thereby increasing the average lifespan of people of India. | Modi ji may advise people to shift to vegetarianism fr msintaining good health thereby increasin the average lifespan of people of India. | LABEL_2 (p = 0.98) | LABEL_1 (p = 0.83) |
| 9043 | Self-driving cars are the self-regulating banks of inanimate objects. | Self-driving cars wre the self-regulating baqnks of inanimate objects. | LABEL_1 (p = 1.00) | LABEL_0 (p = 0.99) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.098 | Punctuation Removal | 98/1000 tested samples (9.8%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Punctuation Removal”, the model changes its prediction in 9.8% of the cases. We expected the predictions not to be affected by this transformation.| text | Punctuation Removal(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 3106 | #Rigged has come full circle ,#deplorables in action | #Rigged has come full circle #deplorables in action | LABEL_1 (p = 0.49) | LABEL_0 (p = 0.65) |
| 10382 | Should we bring back the death penalty for convicted paedophiles? The answer is simple. It's ... #EVENTS | Should we bring back the death penalty for convicted paedophiles The answer is simple It s #EVENTS | LABEL_1 (p = 0.77) | LABEL_0 (p = 0.93) |
| 8946 | Mr Put It Down by Ricky Martin Featuring Pitbull is #nowplaying in Littledown Centre, Bournemouth. | Mr Put It Down by Ricky Martin Featuring Pitbull is #nowplaying in Littledown Centre Bournemouth | LABEL_1 (p = 0.50) | LABEL_0 (p = 0.65) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!