Where do I find the full list of labels?
I have to map those labels to actual language names.
Hey @viktorvsk , the README cites:
This LID (Language IDentification) model is used to predict the language of the input text, and the hosted version (lid218e) was released as part of the NLLB project and can detect 217 languages.
NLLB was trained on the FLORES dataset, the languages available in that dataset are visible here: https://github.com/facebookresearch/flores/tree/main/flores200#languages-in-flores-200
I would suspect these are the languages that the LID model can detect.
@lysandre this doesnt contain full list of all the languages the model supports. gom_Deva is "Goan Konkani in devanagari script" that the model outputs a label. but this isnt available in the following URL provided above.
The fasttext CLI lets you query all the training parameters and/or vocabulary from the model. There you go:
./fastertext/build/fasttext dump lid218e.bin dict | grep __label__ | cut -f1 -d' '
Output:
__label__tsn_Latn
__label__xho_Latn
__label__lvs_Latn
__label__als_Latn
__label__tso_Latn
__label__kaz_Cyrl
__label__sna_Latn
__label__amh_Ethi
__label__zsm_Latn
__label__tha_Thai
__label__tah_Latn
__label__nso_Latn
__label__ewe_Latn
__label__urd_Arab
__label__isl_Latn
__label__lin_Latn
__label__bis_Latn
__label__twi_Latn
__label__sin_Sinh
__label__ben_Beng
__label__mya_Mymr
__label__plt_Latn
__label__pan_Guru
__label__azj_Latn
__label__guj_Gujr
__label__glg_Latn
__label__kir_Cyrl
__label__tel_Telu
__label__tpi_Latn
__label__ibo_Latn
__label__tam_Taml
__label__tat_Cyrl
__label__bem_Latn
__label__bel_Cyrl
__label__kin_Latn
__label__npi_Deva
__label__pap_Latn
__label__mar_Deva
__label__smo_Latn
__label__run_Latn
__label__che_Cyrl
__label__fij_Latn
__label__tir_Ethi
__label__ast_Latn
__label__kan_Knda
__label__mlt_Latn
__label__yor_Latn
__label__eus_Latn
__label__lua_Latn
__label__pag_Latn
__label__sag_Latn
__label__oss_Cyrl
__label__khk_Cyrl
__label__tum_Latn
__label__tgk_Cyrl
__label__lug_Latn
__label__mal_Mlym
__label__umb_Latn
__label__hat_Latn
__label__kon_Latn
__label__azb_Arab
__label__hau_Latn
__label__mos_Latn
__label__kal_Latn
__label__nno_Latn
__label__lus_Latn
__label__oci_Latn
__label__bos_Latn
__label__gaz_Latn
__label__bak_Cyrl
__label__chv_Cyrl
__label__cym_Latn
__label__tuk_Latn
__label__luo_Latn
__label__ayr_Latn
__label__ssw_Latn
__label__quy_Latn
__label__uzn_Latn
__label__kik_Latn
__label__kmb_Latn
__label__jav_Latn
__label__ltz_Latn
__label__asm_Beng
__label__ton_Latn
__label__nya_Latn
__label__kam_Latn
__label__ckb_Arab
__label__min_Latn
__label__bod_Tibt
__label__lmo_Latn
__label__gle_Latn
__label__sun_Latn
__label__xmf_Geor
__label__cjk_Latn
__label__nia_Latn
__label__kbp_Latn
__label__ory_Orya
__label__fon_Latn
__label__kmr_Latn
__label__khm_Khmr
__label__ydd_Hebr
__label__abk_Cyrl
__label__san_Deva
__label__uig_Arab
__label__lim_Latn
__label__scn_Latn
__label__mai_Deva
__label__snd_Arab
__label__wes_Latn
__label__pcm_Latn
__label__arn_Latn
__label__vec_Latn
__label__nav_Latn
__label__gom_Deva
__label__gla_Latn
__label__yue_Hant
__label__dyu_Latn
__label__kac_Latn
__label__roh_Latn
__label__udm_Cyrl
__label__lao_Laoo
__label__diq_Latn
__label__som_Latn
__label__kab_Latn
__label__bjn_Latn
__label__bxr_Cyrl
__label__knc_Arab
__label__szl_Latn
__label__kea_Latn
__label__ban_Latn
__label__crh_Latn
__label__knc_Latn
__label__bug_Latn
__label__fur_Latn
__label__ace_Latn
__label__fuv_Latn
__label__prs_Arab
__label__mri_Latn
__label__dik_Latn
__label__taq_Latn
__label__kas_Arab
__label__pbt_Arab
__label__tzm_Tfng
__label__bam_Latn
__label__mag_Deva
__label__hne_Deva
__label__kas_Deva
__label__nus_Latn
__label__krc_Cyrl
__label__bho_Deva
__label__mni_Beng
__label__ltg_Latn
__label__alt_Cyrl
__label__dzo_Tibt
__label__lij_Latn
__label__wol_Latn
__label__sat_Olck
__label__taq_Tfng
__label__jpn_Jpan
__label__shn_Mymr
__label__bjn_Arab
__label__ace_Arab
__label__grn_Latn
__label__fao_Latn
__label__zho_Hans
__label__zho_Hant
__label__awa_Deva
__label__aka_Latn
__label__arb_Latn
__label__ewo_Latn
__label__srd_Latn
__label__ady_Cyrl