Report for Sigma/financial-sentiment-analysis

#63
by inoki-giskard - opened

Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊

We have identified 3 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_75agree, split train).

👉Robustness issues (3)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness major 🔴 Fail rate = 0.393 Transform to title case 393/1000 tested samples (39.3%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 39.3% of the cases. We expected the predictions not to be affected by this transformation.
text Transform to title case(text) Original prediction Prediction after perturbation
2985 It is a disappointment to see the plan folded . It Is A Disappointment To See The Plan Folded . LABEL_0 (p = 0.98) LABEL_1 (p = 1.00)
201 Operating profit of Kauppalehti group rose to EUR 1.5 mn from EUR 1.3 mn , and that of Marketplaces to EUR 1.3 mn from EUR 1.0 mn in the third quarter of 2006 . Operating Profit Of Kauppalehti Group Rose To Eur 1.5 Mn From Eur 1.3 Mn , And That Of Marketplaces To Eur 1.3 Mn From Eur 1.0 Mn In The Third Quarter Of 2006 . LABEL_2 (p = 1.00) LABEL_1 (p = 1.00)
3223 Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation . Making Matters More Difficult , The Company Said It Has Been Grappling With Higher Oil And Gas Prices , Which Have Pushed Up The Cost Of Energy , Raw Materials And Transportation . LABEL_0 (p = 1.00) LABEL_1 (p = 1.00)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness major 🔴 Fail rate = 0.387 Transform to uppercase 387/1000 tested samples (38.7%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 38.7% of the cases. We expected the predictions not to be affected by this transformation.
text Transform to uppercase(text) Original prediction Prediction after perturbation
202 Operating profit rose from EUR 1.94 mn to EUR 2.45 mn . OPERATING PROFIT ROSE FROM EUR 1.94 MN TO EUR 2.45 MN . LABEL_2 (p = 1.00) LABEL_1 (p = 1.00)
3223 Making matters more difficult , the company said it has been grappling with higher oil and gas prices , which have pushed up the cost of energy , raw materials and transportation . MAKING MATTERS MORE DIFFICULT , THE COMPANY SAID IT HAS BEEN GRAPPLING WITH HIGHER OIL AND GAS PRICES , WHICH HAVE PUSHED UP THE COST OF ENERGY , RAW MATERIALS AND TRANSPORTATION . LABEL_0 (p = 1.00) LABEL_1 (p = 1.00)
1414 Systeemitiimi 's sales and project resources will also be strengthened , director Paul Skogberg said . SYSTEEMITIIMI 'S SALES AND PROJECT RESOURCES WILL ALSO BE STRENGTHENED , DIRECTOR PAUL SKOGBERG SAID . LABEL_2 (p = 1.00) LABEL_1 (p = 1.00)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness medium 🟡 Fail rate = 0.082 Add typos 82/1000 tested samples (8.2%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 8.2% of the cases. We expected the predictions not to be affected by this transformation.
text Add typos(text) Original prediction Prediction after perturbation
551 Finland 's Poyry Energy has won a contract to advise builders of a new cogeneration power plant in Lithuania 's second-biggest city of Kaunas and to supervise the construction process . Finlabnd 's Poyry Enerygy has wkn a contract to advise bylider kf a new cogeneration power planti n Lithuania 's second-biggext city of Kaunas and to supervise the conztruction pfocess . LABEL_2 (p = 1.00) LABEL_1 (p = 0.99)
1450 `` We have analyzed Kaupthing Bank Sweden and found a business which fits well into Alandsbanken , '' said Alandsbanken 's chief executive Peter Wiklof in a statement . `` We have naalyzed Kahupthing Ban Sweden and dound a business which fits well into Alandsbanken , '' said Alabndsbanken 's chkef executive Peter Wiklof in a statement . LABEL_1 (p = 0.50) LABEL_2 (p = 0.64)
477 narrows to EUR2 .8 m 9-mo '09 29 October 2009 - Finnish software and hardware developer Elektrobit Oyj HEL : EBG1V , or EB , said today that its net loss narrowed to EUR2 .8 m for the first nine months of 2009 from EUR35 .6 m for the same period a year ago . narrows to EUR2 .8 m 9-mo '09 29 October 2009 - Finnish software and hardware developer Elektrobit Oyj HEL : EBG1V , or EB , said today that its net loss nzrrowed to EUR2 .8 m for the first nine mnths of 2009 from SUR35 .6 m for the same period a year ago . LABEL_2 (p = 1.00) LABEL_0 (p = 0.95)

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

💡 What's Next?

  • Checkout the Giskard Space and improve your model.
  • The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.

🙌 Big Thanks!

We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!

Sign up or log in to comment