Add verifyToken field to verify evaluation results are produced by Hugging Face's automatic model evaluator

f6ffac1 over 1 year ago

No virus

10.4 kB

	---
	license: mit
	tags:
	- generated_from_trainer
	datasets:
	- banking77
	metrics:
	- accuracy
	widget:
	- text: 'Can I track the card you sent to me? '
	example_title: Card Arrival Example - English
	- text: 'Posso tracciare la carta che mi avete spedito? '
	example_title: Card Arrival Example - Italian
	- text: Can you explain your exchange rate policy to me?
	example_title: Exchange Rate Example - English
	- text: Potete spiegarmi la vostra politica dei tassi di cambio?
	example_title: Exchange Rate Example - Italian
	- text: I can't pay by my credit card
	example_title: Card Not Working Example - English
	- text: Non riesco a pagare con la mia carta di credito
	example_title: Card Not Working Example - Italian
	model-index:
	- name: xlm-roberta-base-banking77-classification
	results:
	- task:
	type: text-classification
	name: Text Classification
	dataset:
	name: banking77
	type: banking77
	config: default
	split: train
	args: default
	metrics:
	- type: accuracy
	value: 0.9321428571428572
	name: Accuracy
	- task:
	type: text-classification
	name: Text Classification
	dataset:
	name: banking77
	type: banking77
	config: default
	split: test
	metrics:
	- type: accuracy
	value: 0.9321428571428572
	name: Accuracy
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzIxM2VlNzU0MjUzYzEzZWVlN2RmNGE0ZGJmYjU0MTEzNDQwNDIxOWE3NjBkODg5MTVhMDVjYTM1MTdlY2YwZiIsInZlcnNpb24iOjF9.EEQaYJCGNkLPDSkfImiypAmHmhXUebRVyaYFX3zM7scpEJHod7yTuAW6EMRc8_uJFUED0Z7kyY1xQBW9gZfcAw
	- type: precision
	value: 0.9339627666926148
	name: Precision Macro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNmJkMTlhMTRiM2Q4OWExMmFjNmZiMmM4ZjhiYjM5MGIxNzkwNjE5MjBlZjI0YjJiYmM0NmMwM2UxYjMxMzEzOSIsInZlcnNpb24iOjF9.KlDw3IF4zC8cXgoQEXqyz6K7WuP1_fPttH5aY37r6qNeMGyAfu_wfcfZZNuJWClO3-ZYAEf1pJk3JGaCMFghDg
	- type: precision
	value: 0.9321428571428572
	name: Precision Micro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNzNhY2YyYTJlMDE1OTI0ZWY5OTBmM2YzMmNiYmEyYWRlNTQxZjkzNTlmNzkyNTU1MDM4NjMxNWE1OWZiMGJjYyIsInZlcnNpb24iOjF9.pcBkD3BJwy2PuVpb4jEstE9mHUtn5eVZEzNcHafhvzYl8pg-XJc0110f_nGl4m2N2K673HshypWlK1QyGFwFBg
	- type: precision
	value: 0.9339627666926148
	name: Precision Weighted
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMDA1YjljYWZjMDQxMjczNDgxY2YxYTQxNGU3MGJmZDA3YWQ5MzdhOGI4ZGMxYjczNDhlNjRkNzM5ODk4Mjk3MiIsInZlcnNpb24iOjF9.jYcTzy50APKNnVwFeyKahMwzo7JO-BlnXQm8P58CRPMNIpJ2Jt1SbDaeCT113JY595P1cJJrv4wGwuskeEMLBw
	- type: recall
	value: 0.9321428571428572
	name: Recall Macro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNjhiYjhkNzZhNzU0YzM5NWQxYTE4ZGI5ZTVkM2VmNTUyOGEzN2Q1MmVhZmRiYTcwZDJhNGM0YWNjNDg1M2JiZCIsInZlcnNpb24iOjF9.1_bFnT9Q_jJADacbB6vrbeLHzSkUh98KjiNmONVgXCzMZIs4OIk9-n7EQssLr-Pe73iNryniifz5ByyfN8SQBA
	- type: recall
	value: 0.9321428571428572
	name: Recall Micro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYjA2ZjQ4NGUzYTQ5NzlhM2Q2Mjk3OTY0NGQyNDE5ODQzOGNhM2IxNDhjOGMxMTZiZjhiNDFkYmFiNDc2ZWFkNSIsInZlcnNpb24iOjF9.domtuqYppEgM-SK5IWrU0OTajd5yybEzxK8MRmcsP619BNfbfMOrAs4RVXDIbcL2j2gditYYotuKwGZYCYMoDQ
	- type: recall
	value: 0.9321428571428572
	name: Recall Weighted
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2ZkOTZhMDc3NWFhZjY2NmU0MGM0YTVlYjEwZDYyMGNmNzgzZjlkNWQ4ZTM1YTQ1YzZmMzg1YjEzYzA5ZDQ5MyIsInZlcnNpb24iOjF9.g5Wwtg3lyPGgDdd0KQQ-Y4dvEdvw_BvAxJTsujLbYx9QIeRVGzwoFwrBINOT__N5ZYK3bJm8VgePOC8Icj7bBg
	- type: f1
	value: 0.9320514513719953
	name: F1 Macro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDY0MGI2YWFmODJiYzBlZWJiYTBjNmVjOWNmMGQ2MmM1NDA1MzQ0NjAwN2U2YmRmMmI1NTM4MTdkMzI1ZDZkZiIsInZlcnNpb24iOjF9.YyPEz8zOhTiSxEVa0yh9UikvsNcNr58bl2zCMJ9lWZCwaZQazDGQfeTLro_7LkXcfyvVjEy0PLfpySO1AH55Dg
	- type: f1
	value: 0.9321428571428572
	name: F1 Micro
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMzU0N2Y3MmY4NTEyOGFlMTlhYmVlNGQ3NTVjNGUzZmEzNzNjY2QxMzU5NTA2ZGU2ZTJjYjViNmJmMzU2YzAyYyIsInZlcnNpb24iOjF9.s13AfCIOGYu4fK63lZCQwAX7IStK5aK7xUCn_fqiLF7IT3lvhgUzTE-Y7T38NBitEsnupFuD6y-f9224eoGjDQ
	- type: f1
	value: 0.9320514513719956
	name: F1 Weighted
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZjM1NTIxYmQyZWQzOTQ4M2UyNWRlZDRlYzlhODUyNmYyOTY2NWVhOTM1NzhiMDAwMjMwMjU1MWYzYWI4MDExMCIsInZlcnNpb24iOjF9.zuke8f4H8RxhZVlYHgiznsIZsLZEclGzdB5b7llCRh_7kU77TylmcjIdm1eIueqhqZfx5vj2KHLqFThPhs80DA
	- type: loss
	value: 0.30337899923324585
	name: loss
	verified: true
	verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMGNiMmVlYjRlZjUyNDhlMWJlNDRmZjE1NjAzZjQwY2E3NWExZmYwZGZjMWU2NWIyMDhiYWVjOGYzMWM3YmY3NiIsInZlcnNpb24iOjF9.pABIX_sezuJzB8EJ1AQvrnSeKPlaiFSKyF7QxopNrfyy8O26j7r5RmXSDt-r5HqNub8zY1JslxgnQ5v8JPsADw
	---

	<!-- This model card has been generated automatically according to the information the Trainer had access to. You
	should probably proofread and complete it, then remove this comment. -->

	# xlm-roberta-base-banking77-classification

	This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the banking77 dataset.
	It achieves the following results on the evaluation set:
	- Loss: 0.3034
	- Accuracy: 0.9321
	- F1 Score: 0.9321

	## Model description

	Experiment on a cross-language model to assess how accurate the classification is by using for fine tuning an English dataset but later querying the model in Italian.

	## Intended uses & limitations

	The model can be used on text classification. In particular is fine tuned on banking domain for multilingual task.

	## Training and evaluation data

	The dataset used is [banking77](https://huggingface.co/datasets/banking77)

	The 77 labels are:

	\|label\|intent\|
	\|:---:\|:----:\|
	\|0\|activate_my_card\|
	\|1\|age_limit\|
	\|2\|apple_pay_or_google_pay\|
	\|3\|atm_support\|
	\|4\|automatic_top_up\|
	\|5\|balance_not_updated_after_bank_transfer\|
	\|6\|balance_not_updated_after_cheque_or_cash_deposit\|
	\|7\|beneficiary_not_allowed\|
	\|8\|cancel_transfer\|
	\|9\|card_about_to_expire\|
	\|10\|card_acceptance\|
	\|11\|card_arrival\|
	\|12\|card_delivery_estimate\|
	\|13\|card_linking\|
	\|14\|card_not_working\|
	\|15\|card_payment_fee_charged\|
	\|16\|card_payment_not_recognised\|
	\|17\|card_payment_wrong_exchange_rate\|
	\|18\|card_swallowed\|
	\|19\|cash_withdrawal_charge\|
	\|20\|cash_withdrawal_not_recognised\|
	\|21\|change_pin\|
	\|22\|compromised_card\|
	\|23\|contactless_not_working\|
	\|24\|country_support\|
	\|25\|declined_card_payment\|
	\|26\|declined_cash_withdrawal\|
	\|27\|declined_transfer\|
	\|28\|direct_debit_payment_not_recognised\|
	\|29\|disposable_card_limits\|
	\|30\|edit_personal_details\|
	\|31\|exchange_charge\|
	\|32\|exchange_rate\|
	\|33\|exchange_via_app\|
	\|34\|extra_charge_on_statement\|
	\|35\|failed_transfer\|
	\|36\|fiat_currency_support\|
	\|37\|get_disposable_virtual_card\|
	\|38\|get_physical_card\|
	\|39\|getting_spare_card\|
	\|40\|getting_virtual_card\|
	\|41\|lost_or_stolen_card\|
	\|42\|lost_or_stolen_phone\|
	\|43\|order_physical_card\|
	\|44\|passcode_forgotten\|
	\|45\|pending_card_payment\|
	\|46\|pending_cash_withdrawal\|
	\|47\|pending_top_up\|
	\|48\|pending_transfer\|
	\|49\|pin_blocked\|
	\|50\|receiving_money\|
	\|51\|Refund_not_showing_up\|
	\|52\|request_refund\|
	\|53\|reverted_card_payment?\|
	\|54\|supported_cards_and_currencies\|
	\|55\|terminate_account\|
	\|56\|top_up_by_bank_transfer_charge\|
	\|57\|top_up_by_card_charge\|
	\|58\|top_up_by_cash_or_cheque\|
	\|59\|top_up_failed\|
	\|60\|top_up_limits\|
	\|61\|top_up_reverted\|
	\|62\|topping_up_by_card\|
	\|63\|transaction_charged_twice\|
	\|64\|transfer_fee_charged\|
	\|65\|transfer_into_account\|
	\|66\|transfer_not_received_by_recipient\|
	\|67\|transfer_timing\|
	\|68\|unable_to_verify_identity\|
	\|69\|verify_my_identity\|
	\|70\|verify_source_of_funds\|
	\|71\|verify_top_up\|
	\|72\|virtual_card_not_working\|
	\|73\|visa_or_mastercard\|
	\|74\|why_verify_identity\|
	\|75\|wrong_amount_of_cash_received\|
	\|76\|wrong_exchange_rate_for_cash_withdrawal\|


	## Training procedure

	```
	from transformers import pipeline
	pipe = pipeline("text-classification", model="nickprock/xlm-roberta-base-banking77-classification")
	pipe("Non riesco a pagare con la carta di credito")
	```

	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 2e-05
	- train_batch_size: 64
	- eval_batch_size: 64
	- seed: 42
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: linear
	- num_epochs: 20

	### Training results

	\| Training Loss \| Epoch \| Step \| Validation Loss \| Accuracy \| F1 Score \|
	\|:-------------:\|:-----:\|:----:\|:---------------:\|:--------:\|:--------:\|
	\| 3.8002 \| 1.0 \| 157 \| 2.7771 \| 0.5159 \| 0.4483 \|
	\| 2.4006 \| 2.0 \| 314 \| 1.6937 \| 0.7140 \| 0.6720 \|
	\| 1.4633 \| 3.0 \| 471 \| 1.0385 \| 0.8308 \| 0.8153 \|
	\| 0.9234 \| 4.0 \| 628 \| 0.7008 \| 0.8789 \| 0.8761 \|
	\| 0.6163 \| 5.0 \| 785 \| 0.5029 \| 0.9068 \| 0.9063 \|
	\| 0.4282 \| 6.0 \| 942 \| 0.4084 \| 0.9123 \| 0.9125 \|
	\| 0.3203 \| 7.0 \| 1099 \| 0.3515 \| 0.9253 \| 0.9253 \|
	\| 0.245 \| 8.0 \| 1256 \| 0.3295 \| 0.9227 \| 0.9225 \|
	\| 0.1863 \| 9.0 \| 1413 \| 0.3092 \| 0.9269 \| 0.9269 \|
	\| 0.1518 \| 10.0 \| 1570 \| 0.2901 \| 0.9338 \| 0.9338 \|
	\| 0.1179 \| 11.0 \| 1727 \| 0.2938 \| 0.9318 \| 0.9319 \|
	\| 0.0969 \| 12.0 \| 1884 \| 0.2906 \| 0.9328 \| 0.9328 \|
	\| 0.0805 \| 13.0 \| 2041 \| 0.2963 \| 0.9295 \| 0.9295 \|
	\| 0.063 \| 14.0 \| 2198 \| 0.2998 \| 0.9289 \| 0.9288 \|
	\| 0.0554 \| 15.0 \| 2355 \| 0.2933 \| 0.9351 \| 0.9349 \|
	\| 0.046 \| 16.0 \| 2512 \| 0.2960 \| 0.9328 \| 0.9326 \|
	\| 0.04 \| 17.0 \| 2669 \| 0.3032 \| 0.9318 \| 0.9318 \|
	\| 0.035 \| 18.0 \| 2826 \| 0.3061 \| 0.9312 \| 0.9312 \|
	\| 0.0317 \| 19.0 \| 2983 \| 0.3030 \| 0.9331 \| 0.9330 \|
	\| 0.0315 \| 20.0 \| 3140 \| 0.3034 \| 0.9321 \| 0.9321 \|


	### Framework versions

	- Transformers 4.21.1
	- Pytorch 1.12.1+cu113
	- Datasets 2.4.0
	- Tokenizers 0.12.1