Fill-Mask

Question Answering

Summarization

Table Question Answering

Text Classification

Text Generation

Text2Text Generation

Token Classification

Translation

Zero-Shot Classification

Text-to-Speech

Automatic Speech Recognition

Audio Source Separation

Voice Activity Detection + 4

PyTorch TensorFlow Rust Flair Asteroid TF SavedModel ESPnet TF Lite Pyannote ONNX Timm + 9

wikipedia squad c4 bookcorpus dcep europarl jrc-acquis CLUECorpusSmall oscar squad_v2 cnn_dailymail conll2003 PropBank.Br jrc-acquis xsum librispeech_asr OSIAN 1.5B Arabic Corpus gigaword parsinlu ontonotes OSCAR Arabic Unshuffled natural_questions multi_nli wikisql CoNLL-2012 Indo4B lince code_search_net OPUS wmt19 sep_clean blended_skill_talk OpenLegalData wtq imdb tab_fact msr_sqa librispeech ai-soco mc4 fever mnli common_crawl xtreme enh_single sep_noisy openwebtext snli flaubert piaf DAGW biomedical literature from Scielo and Pubmed quoref docred gap winograd_wsc winogender glue arabic_billion_words open_subtitles twitter SAIL 2017 squad2 ag_news trivia_qa Libri1Mix Libri2Mix Libri3Mix web_questions brWaC xnli BFD wiki_dpr FQuAD SQuAD-FR emotion PubMed id_newspapers_2018 wmt16 array of dataset identifiers reddit singapore, malaysia hardwarezone id_liputan6 opus100 xsum_nl ComVE wham Universal Dependencies anli go_emotions STSbenchmark MIMIC-III Wikipedia scientific_papers dindebat.dk hestenettet.dk danish OpenSubtitles wikipedia-turkish mlqa fquad Indonesian Wikipedia common_gen MS MARCO document ranking germeval_14 race indosum wmt14 MNLI NQ Trivia SQuAD MLQA DRCD muchocine Twitter IndianPolitics imagenet scancode-rules trec conll2000 dihard jsut ljspeech break_data cnn_dm_nl sst-2 multi_nli_mismatch coqa Shuffled Dutch section of the OSCAR corpus (https://oscar-corpus.com/) Marefa-NER yahoo-answers squad_v1 Uniref100 msmarco eli5 AI4Bharat IndicNLP Corpora 220M words (IndoWiki, IndoWC, News) SciDocs nadi CommonCrawl 100GB Chinese corpus pubmed emo Arabic poetry from several eras triviaqa Wikihow RuSentiment tydiqa tweets_hate_speech_detection quora discofuse mulit_nli mlsum ai2_arc openbookqa Arabic Wikipedia custom-book-corpus Spotify Podcasts Dataset quartz legal entity recognition The Pile bioASQ Wikipedia (Hindi, Sanskrit, Gujarati) RuReviews arcd https://github.com/staeiou/arxiv_archive/tree/v1.0.1 susumu2357/squad_v2_sv marefa-mt https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-Explanation custom danish dataset RuTweetCorp Squad XQuad Tydiqa yelp_polarity codexglue CC-aligned TQUAD Icelandic portion of the OSCAR corpus from INRIA sail sms_spam Oscar Corpus, News, Stories arxiv_dataset socian bangla-sentiment-benchmark pytorrent augmented_codesearchnet JW300 CC100 cc100 coco Tesserae Phi5 Thomas Aquinas algebra_linear_1d algebra_linear_1d_composed measurement_time numbers_gcd kowiki news DaNE legal DAMP-VSEP voxceleb tatoeba setimes csmsc sqa libri1mix bible_para event2Mind qasc quarel quotes-500K MLSUM TACDataset urdu-text-news + 208

en es fr sv fi de multilingual zh ru ar it id uk eo fa nl pt tr pl ja da bg he hi no af Chinese el cs ca ro is ms et lt hu ko sl vi ht hr tl bn gl mt gu lv mk ig ur sg mr ny rw sn xh ee ts ln lg yo si rn eu be as or sm ty to nso fy ha lb sq yi nb fj nn niu bcl crs guw tn gaa te co wa ceb ga st mh fo ilo efi pon pag iso pis bzs pap gil lua cy rm oc an am hy sk zu ti lus kg swc loz tvl tll th gv hil bem bi lu lue ase tw war gd ml so os ps se kqn toi srn jv ka km kn mg mn ta wo br kw ho run tiv tpi wls zne az ber kwy chk mfe rnd tum sc yap ve c++ ne mi tk tt mos dutch sh my roa protein code lo sw sa cv na ba english yue cel sla bo bs pa sr su kl io ce ab ISO 639-1 code for your language, or `multilingual` dv English fiu afa iir sem cpp gem umb gmq zle inc zlw gmw zls ine urj itc kk jap la eng ch gn nv mul trk grk cnr hbs py bat dra aav kj kwn luo ng alv om cpf nyk ss bnt cus euq lun mkh phi sal nic pqe csg csn fse aed mfs prl tzo zai Cszech Deustch French Swedish vn haw hmn ku ky tg ug uz italian ks sd scientific english pt-br [en] scandinavia hi-en yua cau kab art tdt Deustch English zh-tw poz zul ccs map pqw sit tut sami ssp taw vsl wal ach Cszech Deustch Cszech English Cszech Spanish Cszech French Cszech Italian Cszech Swedish Deustch Cszech Deustch Spanish Deustch French Deustch Italian Deustch Swedish English Cszech English Deustch French Cszech French Deustch French English French Spanish French Italian French Swedish Italian Cszech Italian Deustch Italian English Italian Spanish Italian French Italian Swedish Swedish Cszech Swedish Deustch Swedish English Swedish Spanish Swedish French Swedish Italian Go Java javascript php python en de nl es nr sot ven xho esperanto + 329

apache-2.0 mit gpl-3.0 cc-by-sa-3.0 cc-by-4.0 apache 2.0 cc by-nc-sa 4.0 cc-by-sa-4.0 cc-by-nc-4.0 public domain notice any valid license identifier cc-by 4.0 apache license 2.0 apache gnu gplv3 + 13