Report for Sigma/financial-sentiment-analysis
#62
by
inoki-giskard
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 3 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.409 | Transform to title case | 409/1000 tested samples (40.9%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 40.9% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 408 | Ragutis , which is based in Lithuania 's second-largest city Kaunas , boosted its sales last year 22.3 per cent to 36.4 million liters . | Ragutis , Which Is Based In Lithuania 'S Second-Largest City Kaunas , Boosted Its Sales Last Year 22.3 Per Cent To 36.4 Million Liters . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 117 | Finnish department store chain Stockmann Oyj Abp net profit rose to 39.8 mln euro ( $ 56.8 mln ) for the first nine months of 2007 from 37.4 mln euro ( $ 53.4 mln ) for the same period of 2006 . | Finnish Department Store Chain Stockmann Oyj Abp Net Profit Rose To 39.8 Mln Euro ( $ 56.8 Mln ) For The First Nine Months Of 2007 From 37.4 Mln Euro ( $ 53.4 Mln ) For The Same Period Of 2006 . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 2155 | Net sales decreased to EUR 91.6 mn from EUR 109mn in the corresponding period in 2005 . | Net Sales Decreased To Eur 91.6 Mn From Eur 109Mn In The Corresponding Period In 2005 . | LABEL_0 (p = 1.00) | LABEL_1 (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.402 | Transform to uppercase | 402/1000 tested samples (40.2%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 40.2% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 118 | Finnish lifting equipment maker Konecranes Oyj said on July 30 , 2008 that its net profit rose to 71.2 mln euro ( $ 111.1 mln ) for the first half of 2008 from 57.1 mln euro ( $ 89.1 mln ) for the same period of 2007 . | FINNISH LIFTING EQUIPMENT MAKER KONECRANES OYJ SAID ON JULY 30 , 2008 THAT ITS NET PROFIT ROSE TO 71.2 MLN EURO ( $ 111.1 MLN ) FOR THE FIRST HALF OF 2008 FROM 57.1 MLN EURO ( $ 89.1 MLN ) FOR THE SAME PERIOD OF 2007 . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 875 | Net sales will , however , increase from 2005 . | NET SALES WILL , HOWEVER , INCREASE FROM 2005 . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| 823 | In the end of 2006 , the number of outlets will rise to 60-70 . | IN THE END OF 2006 , THE NUMBER OF OUTLETS WILL RISE TO 60-70 . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.057 | Add typos | 57/1000 tested samples (5.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 5.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 781 | After the reporting period , BioTie North American licensing partner Somaxon Pharmaceuticals announced positive results with nalmefene in a pilot Phase 2 clinical trial for smoking cessation . | After the reporting period , BioTie North Xerican licensing partner Somaxon Pharmaceuticals announced psitive results with nakmefene in a piloft Phase 2 clinical trial for smoking cessation . | LABEL_2 (p = 1.00) | LABEL_1 (p = 0.99) |
| 2244 | Due to the rapid decrease in net sales , personnel reductions have been carried out on a wider scale than initially expected . | Due to the rapid dwecrewse ib net sales , personnel reuctions have been carried out on a wider scale than initially expdcted . | LABEL_0 (p = 1.00) | LABEL_1 (p = 0.93) |
| 231 | Marimekko Group 's full-year net sales are estimated to increase by about 10 % . | Mairmekko Grou 's full-year net salws are estimated to increas by about 10 % . | LABEL_2 (p = 1.00) | LABEL_1 (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!