Fill-Mask

Question Answering

Summarization

Table Question Answering

Text Classification

Text Generation

Text2Text Generation

Token Classification

Translation

Zero-Shot Classification

Text-to-Speech

Automatic Speech Recognition

Audio Source Separation

Voice Activity Detection + 4

PyTorch TensorFlow Rust Flair Asteroid TF SavedModel ESPnet TF Lite Pyannote ONNX Timm + 9

wikipedia squad c4 bookcorpus dcep europarl jrc-acquis CLUECorpusSmall oscar squad_v2 cnn_dailymail jrc-acquis PropBank.Br xsum conll2003 OSIAN 1.5B Arabic Corpus gigaword Indo4B OSCAR Arabic Unshuffled natural_questions ontonotes multi_nli librispeech_asr wikisql CoNLL-2012 lince code_search_net OPUS wmt19 sep_clean blended_skill_talk wtq OpenLegalData imdb tab_fact msr_sqa ai-soco mc4 fever common_crawl mnli xtreme enh_single sep_noisy openwebtext snli flaubert piaf DAGW biomedical literature from Scielo and Pubmed arabic_billion_words open_subtitles twitter squad2 SAIL 2017 ag_news trivia_qa quoref docred gap winograd_wsc winogender glue Libri1Mix Libri2Mix Libri3Mix librispeech web_questions brWaC BFD wiki_dpr emotion FQuAD SQuAD-FR PubMed id_newspapers_2018 reddit singapore, malaysia hardwarezone id_liputan6 wmt16 opus100 array of dataset identifiers ComVE wham Universal Dependencies anli go_emotions xnli STSbenchmark MIMIC-III scientific_papers Wikipedia wikipedia-turkish mlqa fquad dindebat.dk hestenettet.dk danish OpenSubtitles common_gen MS MARCO document ranking Indonesian Wikipedia indosum race wmt14 MNLI muchocine NQ Trivia SQuAD MLQA DRCD scancode-rules Twitter IndianPolitics imagenet trec conll2000 dihard jsut ljspeech break_data sst-2 multi_nli_mismatch coqa Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) yahoo-answers msmarco Uniref100 eli5 AI4Bharat IndicNLP Corpora 220M words (IndoWiki, IndoWC, News) squad_v1 SciDocs CommonCrawl emo pubmed nadi Arabic poetry from several eras Wikihow quora triviaqa RuSentiment tydiqa discofuse 100GB Chinese corpus germeval_14 mlsum Arabic Wikipedia Spotify Podcasts Dataset quartz custom-book-corpus tweets_hate_speech_detection mulit_nli ai2_arc openbookqa legal entity recognition Wikipedia (Hindi, Sanskrit, Gujarati) https://github.com/staeiou/arxiv_archive/tree/v1.0.1 arcd custom danish dataset marefa-mt RuReviews CC-aligned sms_spam https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation codexglue RuTweetCorp Squad XQuad Tydiqa yelp_polarity Icelandic portion of the OSCAR corpus from INRIA sail pytorrent qasc socian bangla-sentiment-benchmark augmented_codesearchnet JW300 commonvoice CC100 cc100 coco Tesserae Phi5 Thomas Aquinas algebra_linear_1d algebra_linear_1d_composed measurement_time numbers_gcd kowiki news DaNE legal voxceleb tatoeba setimes csmsc sqa Marefa-NER libri1mix bible_para event2Mind quarel TACDataset urdu-text-news susumu2357/squad_v2_sv parsinlu + 199

en es fr sv fi de multilingual zh ru ar it uk id eo pt nl tr pl da bg fa ja he hi no af el cs ca Chinese ro is ms hu et ko lt vi ht sl tl hr bn gl mt gu ig mk ur sg lv ny rw sn xh ee ts ln mr lg yo si rn eu as be or sm ty to nso fy ha lb sq yi nb fj nn niu crs tn bcl guw gaa co wa ceb ga st te mh fo ilo pon iso bzs pag efi pap pis lua gil cy rm oc an am hy sk zu ti lus kg swc tll tvl loz th gv tw bi bem hil lu war lue ase gd ml so os ps se kqn srn toi ka km kn mg mn ta wo br kw tiv tpi run wls ho zne az jv ber chk kwy mfe rnd tum sc yap ve c++ ne mi tk tt mos sh my roa protein code lo sw sa cv ba na yue cel bo bs pa sr su sla kl io ce ab english dv English ISO 639-1 code for your language, or `multilingual` afa sem gem cpp iir fiu zle zlw inc gmq urj zls umb gmw ine itc kk jap la eng trk ch gn nv mul cnr hbs grk dra aav kj ss bat bnt mkh cus luo cpf phi py kwn ng om alv sal euq nic nyk pqe lun aed fse zai French csg csn mfs prl tzo Cszech Deustch Swedish haw hmn ku ky tg ug uz italian ks sd scientific english pt-br vn [en] scandinavia Deustch English hi-en poz ssp cau tdt zh-tw art tut yua zul ccs map pqw sit sami kab taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch Spanish Deustch French Deustch Italian Deustch Swedish English Cszech English Deustch French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Italian Deustch Italian English Italian Spanish Italian French Italian Swedish Swedish Cszech Swedish Deustch Swedish English Swedish Spanish Swedish French Swedish Italian Go Java javascript php python en de nl es nr sot ven xho esperanto fas + 329

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 cc-by-nc-4.0 public domain notice cc-by-sa-4.0 any valid license identifier cc-by 4.0 apache license 2.0 apache gnu gplv3 cc by-nc-sa 4.0 + 13