Fill-Mask

Question Answering

Summarization

Table Question Answering

Text Classification

Text Generation

Text2Text Generation

Token Classification

Translation

Zero-Shot Classification

Text-to-Speech

Automatic Speech Recognition

Audio Source Separation

Voice Activity Detection + 4

PyTorch TensorFlow Rust Flair Asteroid TF SavedModel ESPnet TF Lite Pyannote ONNX Timm + 9

wikipedia squad c4 bookcorpus dcep europarl jrc-acquis CLUECorpusSmall oscar squad_v2 cnn_dailymail jrc-acquis PropBank.Br xsum conll2003 OSIAN 1.5B Arabic Corpus gigaword Indo4B OSCAR Arabic Unshuffled natural_questions ontonotes multi_nli librispeech_asr wikisql CoNLL-2012 lince code_search_net wmt19 OPUS sep_clean blended_skill_talk wtq OpenLegalData imdb tab_fact msr_sqa ai-soco mc4 fever common_crawl mnli xtreme enh_single sep_noisy openwebtext snli flaubert piaf DAGW biomedical literature from Scielo and Pubmed arabic_billion_words open_subtitles twitter SAIL 2017 squad2 ag_news quoref docred gap winograd_wsc winogender glue trivia_qa librispeech Libri1Mix Libri2Mix Libri3Mix web_questions parsinlu brWaC BFD wiki_dpr emotion FQuAD SQuAD-FR PubMed id_newspapers_2018 reddit singapore, malaysia hardwarezone array of dataset identifiers id_liputan6 wmt16 opus100 xsum_nl ComVE wham Universal Dependencies anli go_emotions xnli STSbenchmark MIMIC-III Wikipedia scientific_papers wikipedia-turkish mlqa fquad dindebat.dk hestenettet.dk danish OpenSubtitles common_gen Indonesian Wikipedia MS MARCO document ranking indosum race wmt14 MNLI NQ Trivia SQuAD MLQA DRCD muchocine Twitter IndianPolitics scancode-rules imagenet trec conll2000 dihard jsut ljspeech break_data sst-2 multi_nli_mismatch coqa Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) yahoo-answers msmarco Uniref100 eli5 AI4Bharat IndicNLP Corpora 220M words (IndoWiki, IndoWC, News) squad_v1 100GB Chinese corpus SciDocs CommonCrawl emo pubmed nadi Arabic poetry from several eras triviaqa Wikihow quora tweets_hate_speech_detection tydiqa RuSentiment discofuse germeval_14 ai2_arc openbookqa Arabic Wikipedia mlsum Spotify Podcasts Dataset quartz custom-book-corpus mulit_nli legal entity recognition Wikipedia (Hindi, Sanskrit, Gujarati) https://github.com/staeiou/arxiv_archive/tree/v1.0.1 arcd custom danish dataset RuTweetCorp RuReviews marefa-mt CC-aligned The Pile sms_spam Squad XQuad Tydiqa codexglue bioASQ https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation Icelandic portion of the OSCAR corpus from INRIA yelp_polarity sail pytorrent socian bangla-sentiment-benchmark qasc Oscar Corpus, News, Stories augmented_codesearchnet JW300 commonvoice CC100 cc100 coco Tesserae Phi5 Thomas Aquinas algebra_linear_1d algebra_linear_1d_composed measurement_time numbers_gcd kowiki news DaNE legal voxceleb tatoeba setimes csmsc sqa Marefa-NER libri1mix bible_para event2Mind quarel TACDataset urdu-text-news susumu2357/squad_v2_sv + 203

en es fr sv fi de multilingual zh ru ar it uk id eo pt nl tr pl fa bg da ja he hi no af el cs ca Chinese ro is ms hu et ko lt vi ht sl tl hr bn gl mt gu mk ig ur sg lv mr ny rw sn xh ee ts ln lg yo si rn eu be as or sm ty to nso fy ha lb sq yi nb fj nn niu crs tn bcl guw gaa co wa ceb ga st te mh fo ilo iso pon bzs pag efi pap lua pis gil cy rm oc an am hy sk zu ti lus kg swc tll loz tvl th gv bi bem hil war lu lue tw ase gd ml so os ps se srn kqn toi jv ka km kn mg mn ta wo br kw tiv tpi run wls ho zne az ber chk kwy mfe rnd tum sc yap ve c++ ne mi tk tt mos sh my roa protein code lo sw sa cv ba english na yue cel sla bo bs pa sr su kl io ce ab ISO 639-1 code for your language, or `multilingual` dv English afa sem cpp gem fiu iir zle gmq dutch zlw inc urj umb zls gmw itc ine kk jap la eng trk ch gn nv mul grk cnr hbs dra bat aav kj ss bnt mkh cus sal py cpf kwn luo phi ng om alv euq nic pqe nyk lun aed fse zai French csg csn mfs prl tzo Cszech Deustch Swedish haw hmn ku ky tg ug uz italian scientific english ks sd vn pt-br [en] scandinavia Deustch English hi-en cau tdt poz ssp zh-tw art tut zul yua ccs map pqw sit sami kab taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch Spanish Deustch French Deustch Italian Deustch Swedish English Cszech English Deustch French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Italian Deustch Italian English Italian Spanish Italian French Italian Swedish Swedish Cszech Swedish Deustch Swedish English Swedish Spanish Swedish French Swedish Italian Go Java javascript php python en de nl es nr sot ven xho esperanto + 329

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 cc-by-nc-4.0 public domain notice cc by-nc-sa 4.0 cc-by-sa-4.0 any valid license identifier cc-by 4.0 apache license 2.0 apache gnu gplv3 + 13