Spaces:
Sleeping
Report for ahmedrachid/FinancialBERT-Sentiment-Analysis
Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 3 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
You can find a full version of scan report here.
👉Robustness issues (3)
When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 41.6% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| major 🔴 | Fail rate = 0.416 | Transform to title case | 416/1000 tested samples (41.6%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Transform to title case(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 408 | Ragutis , which is based in Lithuania 's second-largest city Kaunas , boosted its sales last year 22.3 per cent to 36.4 million liters . | Ragutis , Which Is Based In Lithuania 'S Second-Largest City Kaunas , Boosted Its Sales Last Year 22.3 Per Cent To 36.4 Million Liters . | positive (p = 1.00) | neutral (p = 1.00) |
| 785 | The agreement must be approved by the Russian competition authorities before it enters into force . | The Agreement Must Be Approved By The Russian Competition Authorities Before It Enters Into Force . | positive (p = 0.98) | neutral (p = 1.00) |
| 117 | Finnish department store chain Stockmann Oyj Abp net profit rose to 39.8 mln euro ( $ 56.8 mln ) for the first nine months of 2007 from 37.4 mln euro ( $ 53.4 mln ) for the same period of 2006 . | Finnish Department Store Chain Stockmann Oyj Abp Net Profit Rose To 39.8 Mln Euro ( $ 56.8 Mln ) For The First Nine Months Of 2007 From 37.4 Mln Euro ( $ 53.4 Mln ) For The Same Period Of 2006 . | positive (p = 1.00) | neutral (p = 1.00) |
When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 40.8% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| major 🔴 | Fail rate = 0.408 | Transform to uppercase | 408/1000 tested samples (40.8%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Transform to uppercase(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 785 | The agreement must be approved by the Russian competition authorities before it enters into force . | THE AGREEMENT MUST BE APPROVED BY THE RUSSIAN COMPETITION AUTHORITIES BEFORE IT ENTERS INTO FORCE . | positive (p = 0.98) | neutral (p = 0.99) |
| 118 | Finnish lifting equipment maker Konecranes Oyj said on July 30 , 2008 that its net profit rose to 71.2 mln euro ( $ 111.1 mln ) for the first half of 2008 from 57.1 mln euro ( $ 89.1 mln ) for the same period of 2007 . | FINNISH LIFTING EQUIPMENT MAKER KONECRANES OYJ SAID ON JULY 30 , 2008 THAT ITS NET PROFIT ROSE TO 71.2 MLN EURO ( $ 111.1 MLN ) FOR THE FIRST HALF OF 2008 FROM 57.1 MLN EURO ( $ 89.1 MLN ) FOR THE SAME PERIOD OF 2007 . | positive (p = 1.00) | neutral (p = 0.99) |
| 875 | Net sales will , however , increase from 2005 . | NET SALES WILL , HOWEVER , INCREASE FROM 2005 . | positive (p = 1.00) | neutral (p = 1.00) |
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 7.1% of the cases. We expected the predictions not to be affected by this transformation.
| Level | Metric | Transformation | Deviation |
|---|---|---|---|
| medium 🟡 | Fail rate = 0.071 | Add typos | 71/1000 tested samples (7.1%) changed prediction after perturbation |
Taxonomy
avid-effect:performance:P0201🔍✨Examples
| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 1919 | Cash flow from operations totalled EUR 7.4 mn , compared to a negative EUR 68.6 mn in the second quarter of 2008 . | Cash dlow from operations totalled WUR 7.4 mhn , comlared to a negative EUR 68.6 mn in the second wquarter of 2008 . | positive (p = 1.00) | negative (p = 0.99) |
| 2217 | Talentum 's net sales in September were smaller than expected . | Talentum ' net sales in September were skaler tham expeted . | negative (p = 0.99) | neutral (p = 1.00) |
| 235 | Olvi , which controls a 94 percent stake in Ragutis through A. Le Coq , said in its annual report published earlier this year that the Lithuanian brewery 's sales reached 15.04 million euros last year , a rise of 20.4 percent from 12.49 million euros in 2004 . | Olvi , which controls a 94 percent stake in Raygutis through A. Le Coq , said in its annual report published earlier tbhis year taht the Lithuanian brewery 's sales reached 15.04 mlilion euros laxt year , a ise fo 20.4 percent from 12.49 milklion euros in 2004 . | positive (p = 1.00) | neutral (p = 0.89) |
Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.