Fill-Mask

Question Answering

Summarization

Table Question Answering

Text Classification

Text Generation

Text2Text Generation

Token Classification

Translation

Zero-Shot Classification

Text-to-Speech

Automatic Speech Recognition

Audio Source Separation

Voice Activity Detection + 4

PyTorch TensorFlow Rust Flair Asteroid TF SavedModel ESPnet TF Lite Pyannote ONNX Timm + 9

wikipedia squad c4 bookcorpus dcep europarl jrc-acquis CLUECorpusSmall oscar squad_v2 cnn_dailymail jrc-acquis PropBank.Br xsum conll2003 OSIAN 1.5B Arabic Corpus gigaword Indo4B OSCAR Arabic Unshuffled natural_questions ontonotes multi_nli librispeech_asr wikisql CoNLL-2012 lince code_search_net wmt19 OPUS parsinlu sep_clean blended_skill_talk wtq OpenLegalData imdb tab_fact msr_sqa ai-soco mc4 fever common_crawl mnli xtreme enh_single sep_noisy openwebtext snli flaubert piaf DAGW biomedical literature from Scielo and Pubmed quoref docred gap winograd_wsc winogender glue arabic_billion_words open_subtitles twitter SAIL 2017 squad2 ag_news trivia_qa librispeech Libri1Mix Libri2Mix Libri3Mix web_questions brWaC BFD wiki_dpr emotion FQuAD SQuAD-FR PubMed id_newspapers_2018 array of dataset identifiers reddit singapore, malaysia hardwarezone id_liputan6 wmt16 opus100 xsum_nl ComVE wham Universal Dependencies anli go_emotions xnli STSbenchmark MIMIC-III Wikipedia scientific_papers wikipedia-turkish dindebat.dk hestenettet.dk danish OpenSubtitles mlqa fquad Indonesian Wikipedia common_gen MS MARCO document ranking indosum race wmt14 MNLI NQ Trivia SQuAD MLQA DRCD muchocine Twitter IndianPolitics scancode-rules imagenet trec conll2000 dihard jsut ljspeech break_data sst-2 multi_nli_mismatch coqa Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) Marefa-NER yahoo-answers squad_v1 msmarco Uniref100 eli5 AI4Bharat IndicNLP Corpora 220M words (IndoWiki, IndoWC, News) nadi SciDocs 100GB Chinese corpus CommonCrawl emo pubmed Arabic poetry from several eras triviaqa Wikihow tweets_hate_speech_detection quora tydiqa RuSentiment discofuse ai2_arc openbookqa Arabic Wikipedia mlsum germeval_14 Spotify Podcasts Dataset custom-book-corpus quartz mulit_nli legal entity recognition Wikipedia (Hindi, Sanskrit, Gujarati) The Pile https://github.com/staeiou/arxiv_archive/tree/v1.0.1 marefa-mt arcd custom danish dataset RuTweetCorp RuReviews bioASQ codexglue CC-aligned sms_spam https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation Squad XQuad Tydiqa sail Icelandic portion of the OSCAR corpus from INRIA yelp_polarity susumu2357/squad_v2_sv pytorrent Oscar Corpus, News, Stories socian bangla-sentiment-benchmark augmented_codesearchnet JW300 commonvoice CC100 cc100 coco Tesserae Phi5 Thomas Aquinas algebra_linear_1d algebra_linear_1d_composed measurement_time numbers_gcd kowiki news DaNE legal voxceleb tatoeba setimes csmsc sqa libri1mix bible_para event2Mind qasc quarel quotes-500K TQUAD TACDataset urdu-text-news + 205

en es fr sv fi de multilingual zh ru ar it uk id eo pt nl tr fa pl bg da ja he hi no af el cs ca Chinese ro is ms hu et ko lt vi ht sl tl hr bn gl mt gu mk ig ur sg lv mr ny rw sn xh ee ts ln lg yo si rn eu be as or sm ty to nso fy ha lb sq yi nb fj nn niu crs bcl guw tn gaa co wa ceb ga st te mh fo ilo pag pon efi iso pis bzs pap lua gil cy rm oc an am hy sk zu ti lus kg swc tvl tll loz th gv bi bem hil lu tw lue war ase gd ml so os ps se kqn toi srn jv ka km kn mg mn ta wo br kw run tiv ho tpi wls zne az ber kwy mfe chk rnd sc tum yap ve c++ ne mi tk tt mos sh my roa protein code lo sw sa cv ba na english yue cel sla bo bs pa sr su kl io ce ab ISO 639-1 code for your language, or `multilingual` dv English fiu afa cpp sem gem iir inc zle gmq zlw dutch umb zls ine urj gmw itc kk jap la eng trk ch gn nv mul grk cnr hbs kj aav dra luo bat kwn py om bnt ng ss cus cpf nyk euq lun alv mkh sal phi pqe nic csg csn fse aed mfs prl tzo zai Cszech Deustch French Swedish haw hmn ku ky tg ug uz italian ks sd scientific english vn pt-br [en] hi-en scandinavia Deustch English zh-tw kab zul cau poz tdt art ssp ccs map pqw sit tut yua sami taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch Spanish Deustch French Deustch Italian Deustch Swedish English Cszech English Deustch French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Italian Deustch Italian English Italian Spanish Italian French Italian Swedish Swedish Cszech Swedish Deustch Swedish English Swedish Spanish Swedish French Swedish Italian Go Java javascript php python en de nl es nr sot ven xho esperanto + 329

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 cc by-nc-sa 4.0 cc-by-nc-4.0 public domain notice cc-by-sa-4.0 any valid license identifier cc-by 4.0 apache license 2.0 apache gnu gplv3 + 13