Spaces:
Running
Running
Report for soleimanian/financial-roberta-large-sentiment
#37
by
giskard-bot
- opened
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 7 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_66agree, split train).
👉Performance issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "said" |
Balanced Accuracy = 0.814 | — | -6.91% than global |
🔍✨Examples
For records in the dataset where `text` contains "said", the Balanced Accuracy is 6.91% lower than the global Balanced Accuracy.| text | label | Predicted label |
|
|---|---|---|---|
| 1 | Technopolis plans to develop in stages an area of no less than 100,000 square meters in order to host companies working in computer technologies and telecommunications , the statement said . | neutral | positive (p = 0.99) |
| 371 | Efore 's CEO Reijo Maihaniemi said that the company has developed and already delivered power supply units to respective systems based on this technology to Chinese sector company ZTE Corp SHE : 000063 . | neutral | positive (p = 1.00) |
| 372 | Finnish Talvivaara Mining Co HEL : TLV1V said Thursday it had picked BofA Merrill Lynch and JPMorgan NYSE : JPM as joint bookrunners of its planned issue of convertible notes worth up to EUR250m USD332m . | neutral | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Performance | medium 🟡 | text contains "business" |
Precision = 0.763 | — | -6.55% than global |
🔍✨Examples
For records in the dataset where `text` contains "business", the Precision is 6.55% lower than the global Precision.| text | label | Predicted label |
|
|---|---|---|---|
| 74 | The company still expects its turnover in 2010 to slightly increase from the level of 2009 , adding that `` market predictability is still too poor for trustworthy forecasts on the market development of the contract manufacturing business during the current year '' . | positive | negative (p = 1.00) |
| 506 | `` These tests are part of a larger campaign which includes various customer trials and demonstrations to make LTE on 800 MHz commercially viable by this summer , '' Nokia Siemens head of LTE business line , Reino Tammela , said . | neutral | positive (p = 0.96) |
| 617 | Turnover surged to EUR61 .8 m from EUR47 .6 m due to increasing service demand , especially in the third quarter , and the overall growth of its business . | positive | negative (p = 1.00) |
👉Robustness issues (3)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.116 | Transform to uppercase | 116/1000 tested samples (11.6%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 11.6% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1296 | In the next few years , the ICT sector 's share of electricity consumption will be raised by the increase in the popularity of smartphones . | IN THE NEXT FEW YEARS , THE ICT SECTOR 'S SHARE OF ELECTRICITY CONSUMPTION WILL BE RAISED BY THE INCREASE IN THE POPULARITY OF SMARTPHONES . | negative (p = 0.53) | positive (p = 1.00) |
| 2658 | Salcomp Manufacturing Oy will pay EUR 35 million in cash to Salcomp in connection with the implementation of the sale and transfer of the Business . | SALCOMP MANUFACTURING OY WILL PAY EUR 35 MILLION IN CASH TO SALCOMP IN CONNECTION WITH THE IMPLEMENTATION OF THE SALE AND TRANSFER OF THE BUSINESS . | positive (p = 0.95) | neutral (p = 0.96) |
| 2026 | Besides , Pit-Produkt will enter the new segments . | BESIDES , PIT-PRODUKT WILL ENTER THE NEW SEGMENTS . | positive (p = 0.94) | neutral (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.104 | Add typos | 104/1000 tested samples (10.4%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 10.4% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 2984 | With this , the company will exit the contract manufacturing service segment . | Aith this , the company will exuit the contract manufacturing service segment . | negative (p = 0.89) | neutral (p = 1.00) |
| 3828 | Operating loss landed at EUR39m including one-offs and at EUR27m excluding one-offs . | Operating pos landed at EUR39m includiny one-offzs and at EUR27m excluding pne-offs . | negative (p = 1.00) | neutral (p = 1.00) |
| 1285 | Furthermore , the company will sell the warehouse and office buildings in Loudeac and Saint Marcel and lease new joint premises for these operations in Morvillars . | Furthermore , the comlpany will sell the warehouse and office buildings in Loudeac and Saint Marcel and lezawe new joint premises for thes opedations in Morvillars . | neutral (p = 0.70) | positive (p = 1.00) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | medium 🟡 | — | Fail rate = 0.067 | Transform to title case | 67/1000 tested samples (6.7%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 6.7% of the cases. We expected the predictions not to be affected by this transformation.| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 852 | Sales VAT inclusive expanded by 19 percent , to 351 million euros . | Sales Vat Inclusive Expanded By 19 Percent , To 351 Million Euros . | negative (p = 1.00) | positive (p = 1.00) |
| 1296 | In the next few years , the ICT sector 's share of electricity consumption will be raised by the increase in the popularity of smartphones . | In The Next Few Years , The Ict Sector 'S Share Of Electricity Consumption Will Be Raised By The Increase In The Popularity Of Smartphones . | negative (p = 0.53) | positive (p = 0.94) |
| 3609 | `` We have the most expensive water brand in Finland at the moment . '' | `` We Have The Most Expensive Water Brand In Finland At The Moment . '' | neutral (p = 0.83) | negative (p = 0.99) |
👉Ethical issues (2)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | major 🔴 | — | Fail rate = 0.021 | Switch Gender | 3/141 tested samples (2.13%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 2.13% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch Gender(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 912 | He does not believe , however , that HKScan or Atria will start to use imported meat as Finnish consumers prefer domestic products . | she does not believe , however , that HKScan or Atria will start to use imported meat as Finnish consumers prefer maid products . | neutral (p = 0.83) | positive (p = 0.52) |
| 2403 | Based on the design of previous handsets , the Nokia E72 and Nokia E63 this Symbian-based model is promised to offer direct access to over 90 per cent of the world s corporate email through Mail for Exchange and IBM Lotus Notes Traveler . | Based on the design of previous handsets , the Nokia E72 and Nokia E63 this Symbian-based mannequin is promised to offer direct access to over 90 per cent of the world s corporate email through Mail for Exchange and IBM Lotus Notes Traveler . | positive (p = 0.99) | neutral (p = 0.72) |
| 2693 | Teollisuuden Voima Oyj , the Finnish utility known as TVO , said it shortlisted Mitsubishi Heavy s EU-APWR model along with reactors from Areva , Toshiba Corp. , GE Hitachi Nuclear Energy and Korea Hydro & Nuclear Power Co. . | Teollisuuden Voima Oyj , the Finnish utility known as TVO , said it shortlisted Mitsubishi Heavy s EU-APWR mannequin along with reactors from Areva , Toshiba Corp. , GE Hitachi Nuclear Energy and Korea Hydro & Nuclear Power Co. . | positive (p = 0.99) | neutral (p = 0.58) |
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Ethical | medium 🟡 | — | Fail rate = 0.018 | Switch countries from high- to low-income and vice versa | 18/1000 tested samples (1.8%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Switch countries from high- to low-income and vice versa”, the model changes its prediction in 1.8% of the cases. We expected the predictions not to be affected by this transformation.| text | Switch countries from high- to low-income and vice versa(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1738 | CORPORATE IT UPDATE - ( C ) 1995-2009 M2 COMMUNICATIONS LTD Finnish technology group Teleste Corporation ( OMX Helsinki : TLT1V ) reported on Wednesday ( 4 February ) an operating profit of EUR5 .6 m on net sales of EUR108 .7 m for the year 2008 . | CORPORATE IT UPDATE - ( C ) 1995-2009 M2 COMMUNICATIONS LTD Nepalese technology group Teleste Corporation ( OMX Helsinki : TLT1V ) reported on Wednesday ( 4 February ) an operating profit of EUR5 .6 m on net sales of EUR108 .7 m for the year 2008 . | negative (p = 0.53) | neutral (p = 0.92) |
| 660 | Finnish electronics manufacturing services company Elcoteq signing a cooperation agreement with a Japanese mobile phone manufacturer , bypasses Elcoteq Tallinn , says Jan Kotka , CEO of Elcoteq Tallinn . | Liberian electronics manufacturing services company Elcoteq signing a cooperation agreement with a São Toméan mobile phone manufacturer , bypasses Elcoteq Tallinn , says Jan Kotka , CEO of Elcoteq Tallinn . | neutral (p = 0.83) | positive (p = 0.83) |
| 879 | The inventor was issued U.S. Patent No. . | The inventor was issued Yemen Patent No. . | neutral (p = 0.82) | positive (p = 0.99) |
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- Checkout the Giskard Space and improve your model.
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!