--- tags: - Transformers - text-classification - multi-class-classification languages: - af-ZA - am-ET - ar-SA - az-AZ - bn-BD - cy-GB - da-DK - de-DE - el-GR - en-US - es-ES - fa-IR - fi-FI - fr-FR - he-IL - hi-IN - hu-HU - hy-AM - id-ID - is-IS - it-IT - ja-JP - jv-ID - ka-GE - km-KH - kn-IN - ko-KR - lv-LV - ml-IN - mn-MN - ms-MY - my-MM - nb-NO - nl-NL - pl-PL - pt-PT - ro-RO - ru-RU - sl-SL - sq-AL - sv-SE - sw-KE - ta-IN - te-IN - th-TH - tl-PH - tr-TR - ur-PK - vi-VN - zh-CN - zh-TW multilinguality: - af-ZA - am-ET - ar-SA - az-AZ - bn-BD - cy-GB - da-DK - de-DE - el-GR - en-US - es-ES - fa-IR - fi-FI - fr-FR - he-IL - hi-IN - hu-HU - hy-AM - id-ID - is-IS - it-IT - ja-JP - jv-ID - ka-GE - km-KH - kn-IN - ko-KR - lv-LV - ml-IN - mn-MN - ms-MY - my-MM - nb-NO - nl-NL - pl-PL - pt-PT - ro-RO - ru-RU - sl-SL - sq-AL - sv-SE - sw-KE - ta-IN - te-IN - th-TH - tl-PH - tr-TR - ur-PK - vi-VN - zh-CN - zh-TW datasets: - qanastek/MASSIVE widget: - text: "wake me up at five am this week" - text: "je veux écouter la chanson de jacques brel encore une fois" - text: "quiero escuchar la canción de arijit singh una vez más" - text: "olly onde é que á um parque por perto onde eu possa correr" - text: "פרק הבא בפודקאסט בבקשה" - text: "亚马逊股价" - text: "найди билет на поезд в санкт-петербург" license: cc-by-4.0 --- **People Involved** * [LABRAK Yanis](https://www.linkedin.com/in/yanis-labrak-8a7412145/) (1) **Affiliations** 1. [LIA, NLP team](https://lia.univ-avignon.fr/), Avignon University, Avignon, France. ## Model XLM-Roberta : [https://huggingface.co/xlm-roberta-base](https://huggingface.co/xlm-roberta-base) Paper : [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/pdf/1911.02116.pdf) ## Demo: How to use in HuggingFace Transformers Pipeline Requires [transformers](https://pypi.org/project/transformers/): ```pip install transformers``` ```python from transformers import AutoTokenizer, AutoModelForSequenceClassification, TextClassificationPipeline model_name = 'qanastek/51-languages-classifier' tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) classifier = TextClassificationPipeline(model=model, tokenizer=tokenizer) res = classifier("פרק הבא בפודקאסט בבקשה") print(res) ``` Outputs: ```python [{'label': 'he-IL', 'score': 0.9998375177383423}] ``` ## Training data [MASSIVE](https://huggingface.co/datasets/qanastek/MASSIVE) is a parallel dataset of > 1M utterances across 51 languages with annotations for the Natural Language Understanding tasks of intent prediction and slot annotation. Utterances span 60 intents and include 55 slot types. MASSIVE was created by localizing the SLURP dataset, composed of general Intelligent Voice Assistant single-shot interactions. ### Languages Thee model is capable of distinguish 51 languages : - `Afrikaans - South Africa (af-ZA)` - `Amharic - Ethiopia (am-ET)` - `Arabic - Saudi Arabia (ar-SA)` - `Azeri - Azerbaijan (az-AZ)` - `Bengali - Bangladesh (bn-BD)` - `Chinese - China (zh-CN)` - `Chinese - Taiwan (zh-TW)` - `Danish - Denmark (da-DK)` - `German - Germany (de-DE)` - `Greek - Greece (el-GR)` - `English - United States (en-US)` - `Spanish - Spain (es-ES)` - `Farsi - Iran (fa-IR)` - `Finnish - Finland (fi-FI)` - `French - France (fr-FR)` - `Hebrew - Israel (he-IL)` - `Hungarian - Hungary (hu-HU)` - `Armenian - Armenia (hy-AM)` - `Indonesian - Indonesia (id-ID)` - `Icelandic - Iceland (is-IS)` - `Italian - Italy (it-IT)` - `Japanese - Japan (ja-JP)` - `Javanese - Indonesia (jv-ID)` - `Georgian - Georgia (ka-GE)` - `Khmer - Cambodia (km-KH)` - `Korean - Korea (ko-KR)` - `Latvian - Latvia (lv-LV)` - `Mongolian - Mongolia (mn-MN)` - `Malay - Malaysia (ms-MY)` - `Burmese - Myanmar (my-MM)` - `Norwegian - Norway (nb-NO)` - `Dutch - Netherlands (nl-NL)` - `Polish - Poland (pl-PL)` - `Portuguese - Portugal (pt-PT)` - `Romanian - Romania (ro-RO)` - `Russian - Russia (ru-RU)` - `Slovanian - Slovania (sl-SL)` - `Albanian - Albania (sq-AL)` - `Swedish - Sweden (sv-SE)` - `Swahili - Kenya (sw-KE)` - `Hindi - India (hi-IN)` - `Kannada - India (kn-IN)` - `Malayalam - India (ml-IN)` - `Tamil - India (ta-IN)` - `Telugu - India (te-IN)` - `Thai - Thailand (th-TH)` - `Tagalog - Philippines (tl-PH)` - `Turkish - Turkey (tr-TR)` - `Urdu - Pakistan (ur-PK)` - `Vietnamese - Vietnam (vi-VN)` - `Welsh - United Kingdom (cy-GB)` ## Evaluation results ```plain precision recall f1-score support af-ZA 0.9821 0.9805 0.9813 2974 am-ET 1.0000 1.0000 1.0000 2974 ar-SA 0.9809 0.9822 0.9815 2974 az-AZ 0.9946 0.9845 0.9895 2974 bn-BD 0.9997 0.9990 0.9993 2974 cy-GB 0.9970 0.9929 0.9949 2974 da-DK 0.9575 0.9617 0.9596 2974 de-DE 0.9906 0.9909 0.9908 2974 el-GR 0.9997 0.9973 0.9985 2974 en-US 0.9712 0.9866 0.9788 2974 es-ES 0.9825 0.9842 0.9834 2974 fa-IR 0.9940 0.9973 0.9956 2974 fi-FI 0.9943 0.9946 0.9945 2974 fr-FR 0.9963 0.9923 0.9943 2974 he-IL 1.0000 0.9997 0.9998 2974 hi-IN 1.0000 0.9980 0.9990 2974 hu-HU 0.9983 0.9950 0.9966 2974 hy-AM 1.0000 0.9993 0.9997 2974 id-ID 0.9319 0.9291 0.9305 2974 is-IS 0.9966 0.9943 0.9955 2974 it-IT 0.9698 0.9926 0.9811 2974 ja-JP 0.9987 0.9963 0.9975 2974 jv-ID 0.9628 0.9744 0.9686 2974 ka-GE 0.9993 0.9997 0.9995 2974 km-KH 0.9867 0.9963 0.9915 2974 kn-IN 1.0000 0.9993 0.9997 2974 ko-KR 0.9917 0.9997 0.9956 2974 lv-LV 0.9990 0.9950 0.9970 2974 ml-IN 0.9997 0.9997 0.9997 2974 mn-MN 0.9987 0.9966 0.9976 2974 ms-MY 0.9359 0.9418 0.9388 2974 my-MM 1.0000 0.9993 0.9997 2974 nb-NO 0.9600 0.9533 0.9566 2974 nl-NL 0.9850 0.9748 0.9799 2974 pl-PL 0.9946 0.9923 0.9934 2974 pt-PT 0.9885 0.9798 0.9841 2974 ro-RO 0.9919 0.9916 0.9918 2974 ru-RU 0.9976 0.9983 0.9980 2974 sl-SL 0.9956 0.9939 0.9948 2974 sq-AL 0.9936 0.9896 0.9916 2974 sv-SE 0.9902 0.9842 0.9872 2974 sw-KE 0.9867 0.9953 0.9910 2974 ta-IN 1.0000 1.0000 1.0000 2974 te-IN 1.0000 0.9997 0.9998 2974 th-TH 1.0000 0.9983 0.9992 2974 tl-PH 0.9929 0.9899 0.9914 2974 tr-TR 0.9869 0.9872 0.9871 2974 ur-PK 0.9983 0.9929 0.9956 2974 vi-VN 0.9993 0.9973 0.9983 2974 zh-CN 0.9812 0.9832 0.9822 2974 zh-TW 0.9832 0.9815 0.9823 2974 accuracy 0.9889 151674 macro avg 0.9889 0.9889 0.9889 151674 weighted avg 0.9889 0.9889 0.9889 151674 ``` Keywords : language identification ; language identification ; multilingual ; classification