Spaces:
Sleeping
Sleeping
Report for yiyanghkust/finbert-tone
#193
by
giskard-bot
- opened
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 8 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
You can find a full version of scan report here.
👉Performance issues (7)
For records in the dataset where text contains "said", the Precision is 19.67% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| major 🔴 | text contains "said" |
Precision = 0.737 | -19.67% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 115 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| 129 | However , sales returned to growth in April-June 2010 , CEO Pekka Eloholma said . | Positive | Neutral (p = 0.46) |
| 312 | Finnish telecoms software developer Tecnomen Oyj said on December 5 , 2006 it won a $ 3.3 mln ( 2.5 mln euro ) order to supply an expansion of the charging and messaging systems for the mobile and fixed networks of Brasil Telecom . | Positive | Neutral (p = 0.98) |
For records in the dataset where text contains "period", the Precision is 10.49% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| major 🔴 | text contains "period" |
Precision = 0.821 | -10.49% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 47 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 329 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| 330 | 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . | Positive | Negative (p = 0.63) |
For records in the dataset where text contains "finland", the Balanced Accuracy is 8.58% lower than the global Balanced Accuracy.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text contains "finland" |
Balanced Accuracy = 0.810 | -8.58% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 115 | Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . | Positive | Neutral (p = 1.00) |
| 324 | In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . | Positive | Neutral (p = 1.00) |
| 425 | Citigroup , Inc ( NYSE : C ) has announced that its Global Transaction Services ( GTS ) business has been awarded a new mandate by Finland-based Pohjola Bank Group . | Positive | Neutral (p = 0.75) |
For records in the dataset where text contains "year", the Precision is 6.02% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text contains "year" |
Precision = 0.862 | -6.02% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 230 | Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 . | Positive | Neutral (p = 0.63) |
| 321 | LONDON MarketWatch -- Nokia nok said it 's won a five-year services contract to run Hutchison Essar 's network operations in nine locations in India . | Positive | Neutral (p = 1.00) |
| 324 | In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . | Positive | Neutral (p = 1.00) |
For records in the dataset where text contains "eur", the Precision is 5.68% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text contains "eur" |
Precision = 0.865 | -5.68% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 21 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | Positive | Neutral (p = 1.00) |
| 47 | Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . | Positive | Neutral (p = 0.74) |
| 200 | The fair value of the investment properties totaled EUR 2,534.9 mn , up from EUR 2,455.1 mn in 2006 . | Positive | Neutral (p = 1.00) |
For records in the dataset where text contains "quarter", the Precision is 5.57% lower than the global Precision.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text contains "quarter" |
Precision = 0.866 | -5.57% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | label | Predicted label |
|
|---|---|---|---|
| 328 | In the second quarter of 2010 , the group 's pretax loss narrowed to EUR 400,000 from EUR 600,000 . | Positive | Negative (p = 0.98) |
| 329 | 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . | Positive | Negative (p = 0.59) |
| 330 | 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . | Positive | Negative (p = 0.63) |
For records in the dataset where text_length(text) >= 122.500 AND text_length(text) < 133.500, the Balanced Accuracy is 5.28% lower than the global Balanced Accuracy.
| Level | Data slice | Metric | Deviation |
|---|---|---|---|
| medium 🟡 | text_length(text) >= 122.500 AND text_length(text) < 133.500 |
Balanced Accuracy = 0.839 | -5.28% than global |
Taxonomy
avid-effect:performance:P0204🔍✨Examples
| text | text_length(text) | label | Predicted label |
|
|---|---|---|---|---|
| 10 | Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . | 125 | Positive | Neutral (p = 1.00) |
| 21 | The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . | 132 | Positive | Neutral (p = 1.00) |
| 319 | Bilfinger investors cheered the agreement , pushing shares up 7 % , or & euro ; 3.30 , to & euro ; 50.29 , in afternoon trade . | 127 | Positive | Neutral (p = 1.00) |
👉Robustness issues (1)
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.1% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| medium 🟡 | Fail rate = 0.091 | Add typos | 91/1000 tested samples (9.1%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1143 | A huge issue for us is the button placement . | A huge isue for us is the button placment . | Negative (p = 1.00) | Neutral (p = 0.99) |
| 2217 | Talentum 's net sales in September were smaller than expected . | Talentum ' net sales in September were skaler tham expeted . | Negative (p = 1.00) | Neutral (p = 1.00) |
| 956 | Elcoteq has a global network of After Market Service sites which have a long experience in serving Consumer Electronics and Systems Solutions customers . | Elcoteq has a globa network of After Market Service sites which have a long experience in serving Consumer Electrinixs and Systems Solutions ustomers . | Positive (p = 0.55) | Neutral (p = 0.97) |
Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.