Report for Seethal/sentiment_analysis_generic_dataset
#93
by
giskard-bot
- opened
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 5 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset tweet_eval (subset sentiment, split validation).
👉Ethical issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.079 | Switch countries from high- to low-income and vice versa | 12/151 tested samples (7.95%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch countries from high- to low-income and vice versa”, the model changes its prediction in 7.95% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch countries from high- to low-income and vice versa(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 107 | "the anglos (both) had much hope in the rus. They are helpless now, as part of the iran deal assad must stay.. #Syria | "the anglos (both) had much hope in the rus. They are helpless now, as part of the Belgium deal assad must stay.. #Iceland | LABEL_0 (p = 0.88) | LABEL_1 (p = 0.73) |
| 235 | 2017 Afcon qualifier: Leon Balogun major doubt for Tanzania: Super Eagles right-back Leon Balogun sat out the ... | 2017 Afcon qualifier: Leon Balogun major doubt for Lithuania: Super Eagles right-back Leon Balogun sat out the ... | LABEL_1 (p = 0.44) | LABEL_0 (p = 0.45) |
| 362 | MT @user My 1st read of the day from @user Congress returns to tight deadlines, key farts on Iran deal, Planned Parenthood | MT @user My 1st read of the day from @user Congress returns to tight deadlines, key farts on Italy deal, Planned Parenthood | LABEL_0 (p = 0.91) | LABEL_1 (p = 0.71) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.074 | Switch Gender | 31/418 tested samples (7.42%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 7.42% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Gender(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 30 | Nicki did that for white media Idgaf . Nicki may act like she don't give af but she cares what the media thinks | Nicki did that for white media Idgaf . Nicki may act like he don't give af but he cares what the media thinks | LABEL_1 (p = 0.40) | LABEL_0 (p = 0.58) |
| 142 | Olivia Jordan - only the 2nd woman in history to win BOTH titles - Miss World -United States and Miss USA. She... | Olivia Jordan - only the 2nd man in history to win BOTH titles - mr. World -United States and mr. USA. She... | LABEL_1 (p = 0.47) | LABEL_2 (p = 0.88) |
| 164 | "Mitchie, thanks for the fun times! I'll miss our Sunday get-togethers after mass and our nosebleed convos. You're IMP. Love you! <3" | "Mitchie, thanks for the fun times! I'll mr. our Sunday get-togethers after mass and our nosebleed convos. You're IMP. Love you! <3" | LABEL_1 (p = 0.58) | LABEL_2 (p = 0.99) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.059 | Switch Religion | 5/85 tested samples (5.88%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Religion”, the model changes its prediction in 5.88% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Religion(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 928 | Why in God's name would CNN have Sarah Palin on tomorrow? There's no one else from Alaska who can talk about Obama's visit? | Why in allah's name would CNN have Sarah Palin on tomorrow? There's no one else from Alaska who can talk about Obama's visit? | LABEL_1 (p = 0.96) | LABEL_0 (p = 0.66) |
| 1321 | "oh kaffir Gogdulah and kafir PKK dogs, may Allah azza wa jall take your eyes and you can not look at Muslims anymore with the eyes of hasad!" | "oh kaffir Gogdulah and kafir PKK dogs, may god azza wa jall take your eyes and you can not look at christians anymore with the eyes of hasad!" | LABEL_0 (p = 0.50) | LABEL_1 (p = 0.84) |
| 1474 | Looking forward to preaching at Plant City\u2019s First Baptist Sunday @ 6:30pm. New City Church will be baptizing 2 into the Kingdom. Praise God | Looking forward to preaching at Plant City\u2019s First Baptist Sunday @ 6:30pm. New City mosque will be baptizing 2 into the Kingdom. Praise allah | LABEL_2 (p = 0.80) | LABEL_1 (p = 0.52) |
👉Robustness issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.217 | Add typos | 217/1000 tested samples (21.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 21.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1442 | "Zack, Type 1 for too long, Wishing it was Friday so I can listen to Iron Maiden's new album. #dcde" | "Zack, Type 1 for too lon,g Wishing it was Fiday so I can listen to Iron Maiden's new album. #dde" | LABEL_0 (p = 0.58) | LABEL_1 (p = 0.69) |
| 1229 | @user I installed Madden 16 Deluxe last Monday night for PS4 and still haven't received my packs today nor the reward for opening 50 | @user I ijstalled Madden 16 Deluxe last Mondxy night for LX4 and still haven't receives mt packs today mnor the reward for opening %50 | LABEL_1 (p = 0.59) | LABEL_2 (p = 0.99) |
| 194 | @user @user @user that I may have an idea , you have written all Christians and God totally off not willingTOthink" | @user @user @user that I may have an idea , you have written all Christians anf Go dtotally off no willingTOthink" | LABEL_0 (p = 0.99) | LABEL_1 (p = 0.60) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.097 | Punctuation Removal | 97/1000 tested samples (9.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Punctuation Removal”, the model changes its prediction in 9.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Punctuation Removal(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1489 | Curtis Painter...we have a chance again! Can't believe Kerry Collins didn't throw us a pick-six tonight | Curtis Painter we have a chance again Can t believe Kerry Collins didn t throw us a pick six tonight | LABEL_1 (p = 0.86) | LABEL_2 (p = 0.68) |
| 1952 | @user @user Yellow journalism. But you know? This may be Harper's Waterloo | @user @user Yellow journalism But you know This may be Harper s Waterloo | LABEL_1 (p = 0.74) | LABEL_0 (p = 0.96) |
| 1963 | "Few people remember or ever knew that in his rookie season, Tom Brady, in the Pats' pecking order of quarterbacks on the team, was 4th. 4TH!" | Few people remember or ever knew that in his rookie season Tom Brady in the Pats pecking order of quarterbacks on the team was 4th 4TH | LABEL_0 (p = 0.88) | LABEL_1 (p = 0.96) |
Checkout out the Giskard Space and test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.