Translation

Text Generation

Fill-Mask

Text Classification

Text2Text Generation

Question Answering

Token Classification

Table Question Answering

Zero-Shot Classification

Automatic Speech Recognition

PyTorch TensorFlow Rust Asteroid TF SavedModel Flair TF Lite ESPnet Pyannote ONNX + 8

wikipedia c4 squad bookcorpus dcep europarl jrc-acquis squad_v2 oscar CLUECorpusSmall cnn_dailymail PropBank.Br jrc-acquis OSIAN 1.5B Arabic Corpus gigaword xsum OSCAR Arabic Unshuffled natural_questions wikisql CoNLL-2012 Indo4B lince multi_nli code_search_net wmt19 OPUS sep_clean conll2003 blended_skill_talk OpenLegalData imdb wtq enh_single ai-soco msr_sqa tab_fact mnli fever mc4 xtreme Libri1Mix openwebtext snli piaf common_crawl flaubert biomedical literature from Scielo and Pubmed squad2 arabic_billion_words open_subtitles twitter SAIL 2017 Libri2Mix sep_noisy Libri3Mix trivia_qa web_questions quoref docred gap winograd_wsc winogender glue FQuAD SQuAD-FR wiki_dpr emotion PubMed wmt16 reddit singapore, malaysia hardwarezone ComVE wham Universal Dependencies brWaC xnli BFD MIMIC-III STSbenchmark fquad anli wmt14 mlqa DAGW race wikipedia-turkish indosum scancode-rules MS MARCO document ranking scientific_papers trec Twitter IndianPolitics Indonesian Wikipedia dihard MNLI jsut ljspeech Wikipedia common_gen muchocine opus100 break_data sst-2 eli5 Uniref100 AI4Bharat IndicNLP Corpora yahoo-answers 220M words (IndoWiki, IndoWC, News) quora Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) multi_nli_mismatch pubmed nadi yelp_polarity Wikihow emo event2Mind triviaqa id_newspapers_2018 augmented_codesearchnet Wikipedia (Hindi, Sanskrit, Gujarati) msmarco germeval_14 discofuse Arabic Wikipedia https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation coqa go_emotions ai2_arc openbookqa custom-book-corpus codexglue dindebat.dk hestenettet.dk danish OpenSubtitles custom danish dataset JW300 100GB Chinese corpus CC100 Arabic poetry from several eras cc100 coco RuTweetCorp RuReviews RuSentiment librispeech id_liputan6 https://github.com/staeiou/arxiv_archive/tree/v1.0.1 kowiki news sail voxceleb tatoeba setimes csmsc sqa libri1mix tydiqa bible_para qasc quarel quartz CC-aligned ag_news Icelandic portion of the OSCAR corpus from INRIA Spotify Podcasts Dataset socian bangla-sentiment-benchmark CommonCrawl urdu-text-news SciDocs + 163

en es fr sv fi de zh multilingual ru ar it uk eo pt tr nl id pl bg ja he da fa hi af el ca no ro cs is hu et lt vi ht sl tl ko bn hr mt gu ig mk ur lv ny rw ee sn ts gl ln xh lg mr yo Chinese si rn eu be ty or to sm as nso fy lb sq nb fj nn yi gaa ha guw bcl crs niu tn mh te fo co ga wa ceb efi gil ilo lua bzs iso pag pap pis pon st cy an rm am ti oc hy zu sk kg loz lus swc tll tvl th gv tw bem bi hil ase lu lue war os ps gd so se ml kqn toi srn mn ka br km wo kn ta kw mg ho run tiv tpi wls zne chk az ber sc kwy mfe rnd ve yap tum c++ ne tk tt mi jv mos sh my code roa cv lo na ba yue sa sw protein su kl bo io ce ab pa afa sla english cel cpp fiu gem inc ine gmq gmw iir itc sem umb urj zle zls zlw English kk jap ch gn nv mul eng trk bat bnt cpf cus dra kj fse aav aed alv csg csn euq grk kwn lun luo mkh ng nic nyk om phi pqe sal ss mfs prl tzo zai Cszech Deustch French Swedish dv italian zh-tw ks sd scientific english vn kab scandinavia art cau ccs map poz pqw sit tdt tut yua sami ssp taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch English Deustch Spanish Deustch French Deustch Italian Deustch Swedish French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Go Java javascript php python hi-en nr sot ven xho zul [en] pt-br esperanto bs haw hmn ku ky la sr tg ug uz `en` + 309

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 apache 2.0 cc-by-4.0 public domain notice cc-by-nc-4.0 cc-by-sa-4.0 cc-by 4.0 apache license 2.0 apache gnu gplv3 + 11