Fill-Mask

Question Answering

Summarization

Table Question Answering

Text Classification

Text Generation

Text2Text Generation

Token Classification

Translation

Zero-Shot Classification

Text-to-Speech

Automatic Speech Recognition

Audio Source Separation

Voice Activity Detection + 4

PyTorch TensorFlow Rust Flair Asteroid TF SavedModel ESPnet TF Lite Pyannote ONNX Timm + 9

wikipedia squad c4 bookcorpus dcep europarl jrc-acquis CLUECorpusSmall oscar squad_v2 cnn_dailymail jrc-acquis PropBank.Br xsum conll2003 OSIAN 1.5B Arabic Corpus gigaword Indo4B OSCAR Arabic Unshuffled natural_questions ontonotes multi_nli librispeech_asr wikisql CoNLL-2012 lince code_search_net OPUS wmt19 sep_clean blended_skill_talk wtq OpenLegalData imdb tab_fact msr_sqa ai-soco mc4 fever common_crawl mnli xtreme enh_single sep_noisy openwebtext snli flaubert piaf DAGW biomedical literature from Scielo and Pubmed arabic_billion_words open_subtitles twitter SAIL 2017 squad2 ag_news trivia_qa quoref docred gap winograd_wsc winogender glue librispeech Libri1Mix Libri2Mix Libri3Mix web_questions parsinlu brWaC BFD wiki_dpr emotion FQuAD SQuAD-FR PubMed id_newspapers_2018 reddit singapore, malaysia hardwarezone id_liputan6 array of dataset identifiers wmt16 opus100 xsum_nl ComVE wham Universal Dependencies anli go_emotions xnli STSbenchmark MIMIC-III Wikipedia scientific_papers wikipedia-turkish mlqa fquad dindebat.dk hestenettet.dk danish OpenSubtitles common_gen MS MARCO document ranking Indonesian Wikipedia indosum race wmt14 MNLI muchocine NQ Trivia SQuAD MLQA DRCD Twitter IndianPolitics scancode-rules imagenet trec conll2000 dihard jsut ljspeech break_data sst-2 multi_nli_mismatch coqa Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) yahoo-answers msmarco Uniref100 eli5 AI4Bharat IndicNLP Corpora 220M words (IndoWiki, IndoWC, News) squad_v1 100GB Chinese corpus SciDocs CommonCrawl emo pubmed nadi Arabic poetry from several eras triviaqa Wikihow quora tydiqa RuSentiment discofuse germeval_14 mlsum Arabic Wikipedia tweets_hate_speech_detection Spotify Podcasts Dataset quartz mulit_nli custom-book-corpus ai2_arc openbookqa legal entity recognition Wikipedia (Hindi, Sanskrit, Gujarati) https://github.com/staeiou/arxiv_archive/tree/v1.0.1 arcd custom danish dataset marefa-mt RuReviews CC-aligned RuTweetCorp sms_spam https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation codexglue Squad XQuad Tydiqa yelp_polarity Icelandic portion of the OSCAR corpus from INRIA sail pytorrent socian bangla-sentiment-benchmark qasc augmented_codesearchnet JW300 commonvoice CC100 cc100 coco Tesserae Phi5 Thomas Aquinas algebra_linear_1d algebra_linear_1d_composed measurement_time numbers_gcd kowiki news DaNE legal voxceleb tatoeba setimes csmsc sqa Marefa-NER libri1mix bible_para event2Mind quarel bioASQ TACDataset urdu-text-news susumu2357/squad_v2_sv + 201

en es fr sv fi de multilingual zh ru ar it uk id eo pt nl tr pl fa da bg ja he hi no af el cs ca Chinese ro is ms hu et ko lt vi ht sl tl hr bn gl mt gu ig mk ur sg lv ny rw sn xh ee ts ln mr lg yo si rn eu as be or sm ty to nso fy ha lb sq yi nb fj nn niu crs tn bcl guw gaa co wa ceb ga st te mh fo ilo iso pon bzs pag efi pap pis lua gil cy rm oc an am hy sk zu ti lus kg swc tll loz tvl th gv bi bem hil lu war lue tw ase gd ml so os ps se kqn srn toi jv ka km kn mg mn ta wo br kw tiv tpi run wls ho zne az ber chk kwy mfe rnd tum sc yap ve c++ ne mi tk tt mos sh my roa protein code lo sw sa cv ba na english yue cel bo bs pa sr su sla kl io ce ab dv ISO 639-1 code for your language, or `multilingual` English afa sem gem cpp iir fiu zle zlw inc gmq dutch urj zls umb gmw itc ine kk jap la eng trk ch gn nv mul cnr hbs grk dra aav kj ss bat bnt mkh sal py cus kwn luo phi cpf ng om alv euq nic nyk pqe lun aed fse zai French csg csn mfs prl tzo Cszech Deustch Swedish haw hmn ku ky tg ug uz italian ks sd scientific english pt-br vn [en] Deustch English scandinavia hi-en cau poz ssp zh-tw tdt art tut yua zul ccs map pqw sit sami kab taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch Spanish Deustch French Deustch Italian Deustch Swedish English Cszech English Deustch French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Italian Deustch Italian English Italian Spanish Italian French Italian Swedish Swedish Cszech Swedish Deustch Swedish English Swedish Spanish Swedish French Swedish Italian Go Java javascript php python en de nl es nr sot ven xho esperanto + 329

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 cc-by-nc-4.0 public domain notice cc by-nc-sa 4.0 cc-by-sa-4.0 any valid license identifier cc-by 4.0 apache license 2.0 apache gnu gplv3 + 13