giskardai/giskard-evaluator · Report for ProsusAI/finbert

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 1 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).

You can find a full version of scan report here.

👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 6.2% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
medium 🟡	Fail rate = 0.062	Add typos	62/1000 tested samples (6.2%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
1919	Cash flow from operations totalled EUR 7.4 mn , compared to a negative EUR 68.6 mn in the second quarter of 2008 .	Cash dlow from operations totalled WUR 7.4 mhn , comlared to a negative EUR 68.6 mn in the second wquarter of 2008 .	positive (p = 0.86)	negative (p = 0.95)
1584	`` These developments partly reflect the government 's higher activity in the field of dividend policy . ''	`` These developmsents partly reflect he government 's higher activity in the field of dividend policty . ''	positive (p = 0.75)	neutral (p = 0.58)
628	The share of the share capital of both above mentioned shareholders remains below 5 % .	The share of the share capiral of both above mentioned sharehooders remains below 5 % .	neutral (p = 0.72)	negative (p = 0.83)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.