Report for ProsusAI/finbert

#194
by giskard-bot - opened
Giskard org

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 1 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).

You can find a full version of scan report here.

👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 6.2% of the cases. We expected the predictions not to be affected by this transformation.

Level Metric Transformation Deviation
medium 🟡 Fail rate = 0.062 Add typos 62/1000 tested samples (6.2%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201
🔍✨Examples
text Add typos(text) Original prediction Prediction after perturbation
1919 Cash flow from operations totalled EUR 7.4 mn , compared to a negative EUR 68.6 mn in the second quarter of 2008 . Cash dlow from operations totalled WUR 7.4 mhn , comlared to a negative EUR 68.6 mn in the second wquarter of 2008 . positive (p = 0.86) negative (p = 0.95)
1584 `` These developments partly reflect the government 's higher activity in the field of dividend policy . '' `` These developmsents partly reflect he government 's higher activity in the field of dividend policty . '' positive (p = 0.75) neutral (p = 0.58)
628 The share of the share capital of both above mentioned shareholders remains below 5 % . The share of the share capiral of both above mentioned sharehooders remains below 5 % . neutral (p = 0.72) negative (p = 0.83)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

Sign up or log in to comment