Spaces:
Starting
Starting
Report for soleimanian/financial-roberta-large-sentiment
#35
by
giskard-bot
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 7 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_50agree, split train).
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.107 | Transform to uppercase | 107/1000 tested samples (10.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 10.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2223 | The Finnish textiles and clothing company Marimekko Corporation ( OMX Helsinki : MMO1V ) reported on Wednesday ( 5 November ) an operating profit of EUR8 .1 m on net sales of EUR59m for the period from January to September 2008 . | THE FINNISH TEXTILES AND CLOTHING COMPANY MARIMEKKO CORPORATION ( OMX HELSINKI : MMO1V ) REPORTED ON WEDNESDAY ( 5 NOVEMBER ) AN OPERATING PROFIT OF EUR8 .1 M ON NET SALES OF EUR59M FOR THE PERIOD FROM JANUARY TO SEPTEMBER 2008 . | neutral (p = 0.93) | positive (p = 1.00) |
| 2310 | The fund at fair value will increase correspondingly . | THE FUND AT FAIR VALUE WILL INCREASE CORRESPONDINGLY . | neutral (p = 0.50) | positive (p = 1.00) |
| 637 | AGJ recorded EUR 43 mln sales in 2006 , most of which was generated by exports to customers in Western Europe , the statement said . | AGJ RECORDED EUR 43 MLN SALES IN 2006 , MOST OF WHICH WAS GENERATED BY EXPORTS TO CUSTOMERS IN WESTERN EUROPE , THE STATEMENT SAID . | neutral (p = 1.00) | positive (p = 0.94) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.102 | Add typos | 102/1000 tested samples (10.2%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 10.2% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 296 | The German company has also signed a code share agreement with another Oneworld member -- American Airlines Inc , part of US-based AMR Corp ( NYSE : AMR ) . | The German xompany has aso signed a code dshare ageement wjtgh anothef Onewold member -- American Airines Inc , part of USbased AMF Corp ( NYSE : AMR ) . | positive (p = 1.00) | neutral (p = 1.00) |
| 2019 | in Q1 '10 19 April 2010 - Finnish forest machinery and equipment maker Ponsse Oyj HEL : PON1V said today that it expects to swing to a net profit of some EUR6 .3 m in the first quarter of 2010 , from an EUR9 .6 m loss a year earlier . | in Q1 '10 19 April 0210 - Finnish forest machinery anc equipment maker Ponsse Oyj HEL : PON1V said today that it expects ro sinf to a net profit of zome EUR6 .3 m in the first quarter of 2010 , from an EUR9 .6 m losx a year earlier . | positive (p = 1.00) | negative (p = 1.00) |
| 3533 | As a result , the number of personnel in Finland will be reduced by 158 . | As a result , rthe number of personnel in Finland wull be redhuced by 158 . | negative (p = 1.00) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.053 | Transform to title case | 53/1000 tested samples (5.3%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 5.3% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2310 | The fund at fair value will increase correspondingly . | The Fund At Fair Value Will Increase Correspondingly . | neutral (p = 0.50) | positive (p = 1.00) |
| 4392 | Copper , lead and nickel also dropped ... HBOS ( HBOS ) plummeted 20 % to 70.3 pence after saying this year+ó ?? | Copper , Lead And Nickel Also Dropped ... Hbos ( Hbos ) Plummeted 20 % To 70.3 Pence After Saying This Year+Ó ?? | negative (p = 1.00) | positive (p = 0.98) |
| 1873 | The winners included the Honda Odyssey for minivan and the Nissan Armada for large SUV . | The Winners Included The Honda Odyssey For Minivan And The Nissan Armada For Large Suv . | neutral (p = 0.59) | positive (p = 1.00) |
👉Ethical issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | major 🔴 | — | Fail rate = 0.023 | Switch Gender | 4/173 tested samples (2.31%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 2.31% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Gender(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1041 | He does not believe , however , that HKScan or Atria will start to use imported meat as Finnish consumers prefer domestic products . | she does not believe , however , that HKScan or Atria will start to use imported meat as Finnish consumers prefer maid products . | neutral (p = 0.83) | positive (p = 0.52) |
| 2826 | Based on the design of previous handsets , the Nokia E72 and Nokia E63 this Symbian-based model is promised to offer direct access to over 90 per cent of the world s corporate email through Mail for Exchange and IBM Lotus Notes Traveler . | Based on the design of previous handsets , the Nokia E72 and Nokia E63 this Symbian-based mannequin is promised to offer direct access to over 90 per cent of the world s corporate email through Mail for Exchange and IBM Lotus Notes Traveler . | positive (p = 0.99) | neutral (p = 0.72) |
| 3149 | Teollisuuden Voima Oyj , the Finnish utility known as TVO , said it shortlisted Mitsubishi Heavy s EU-APWR model along with reactors from Areva , Toshiba Corp. , GE Hitachi Nuclear Energy and Korea Hydro & Nuclear Power Co. . | Teollisuuden Voima Oyj , the Finnish utility known as TVO , said it shortlisted Mitsubishi Heavy s EU-APWR mannequin along with reactors from Areva , Toshiba Corp. , GE Hitachi Nuclear Energy and Korea Hydro & Nuclear Power Co. . | positive (p = 0.99) | neutral (p = 0.58) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.017 | Switch countries from high- to low-income and vice versa | 17/1000 tested samples (1.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch countries from high- to low-income and vice versa”, the model changes its prediction in 1.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch countries from high- to low-income and vice versa(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 3336 | The properties were purchased from Swedish private equity real estate firm Niam and Goldman Sachs ' Whitehall Street Real Estate Funds . | The properties were purchased from Sudanese private equity real estate firm Niam and Goldman Sachs ' Whitehall Street Real Estate Funds . | neutral (p = 0.70) | positive (p = 0.66) |
| 2864 | Danske Bank is Denmark 's largest bank with 3.5 million customers . | Danske Bank is Vietnam 's largest bank with 3.5 million customers . | neutral (p = 0.99) | positive (p = 0.90) |
| 1241 | HKScan is one of the leading food companies in northern Europe with homemarkets in Finland , Sweden , the Baltic countries and Poland . | HKScan is one of the leading food companies in northern Europe with homemarkets in Togo , Cabo Verde , the Baltic countries and Afghanistan . | positive (p = 0.85) | neutral (p = 0.80) |
👉Performance issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | avg_digits(text) < 0.031 AND avg_digits(text) >= 0.014 |
Precision = 0.722 | — | -7.33% than global |
🔍✨Examples
For records in the dataset where `avg_digits(text)` < 0.031 AND `avg_digits(text)` >= 0.014, the Precision is 7.33% lower than the global Precision.| text | avg_digits(text) | label | Predicted label |
|
|---|---|---|---|---|
| 59 | In Sweden , Gallerix accumulated SEK denominated sales were down 1 % and EUR denominated sales were up 11 % . | 0.0275229 | neutral | negative (p = 0.82) |
| 64 | In June it sold a 30 percent stake to Nordstjernan , and the investment group has now taken up the option to acquire EQT 's remaining shares . | 0.0140845 | neutral | positive (p = 0.99) |
| 75 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 0.01875 | neutral | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text_length(text) >= 149.500 AND text_length(text) < 161.500 |
Precision = 0.731 | — | -6.06% than global |
🔍✨Examples
For records in the dataset where `text_length(text)` >= 149.500 AND `text_length(text)` < 161.500, the Precision is 6.06% lower than the global Precision.| text | text_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 60 | The company supports its global customers in developing new technologies and offers a fast route from product development to applications and volume production . | 161 | neutral | positive (p = 1.00) |
| 75 | On the route between Helsinki in Finland and Tallinn in Estonia , cargo volumes increased by 36 % , while cargo volumes between Finland and Sweden fell by 9 % . | 160 | neutral | positive (p = 1.00) |
| 409 | To our members and partners , the use of IT will mostly be apparent in the increased efficiency of the results service , '' observes Perttu Puro from Tradeka . | 159 | positive | neutral (p = 1.00) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!