🚀 Models 🔥 Datasets ⚡️ Pricing
🔥 Sign in
Back to all models
Model: sagorsarker/codeswitch-hineng-lid-lince
pytorch bert token-classification hi en dataset:lince 🏷 license:mit codeswitching hindi-english language-identification
Model card Files and versions Use in transformers

How to use this model directly from the 🤗/transformers library: 

					
Copy to clipboard

						from transformers import AutoTokenizer, AutoModelForTokenClassification
						
						tokenizer = AutoTokenizer.from_pretrained("sagorsarker/codeswitch-hineng-lid-lince")
						
						model = AutoModelForTokenClassification.from_pretrained("sagorsarker/codeswitch-hineng-lid-lince")
See raw config file

How to clone the model repo 

					
Copy to clipboard

					git lfs install
					git clone https://huggingface.co/sagorsarker/codeswitch-hineng-lid-lince
					

						# if you want to clone without large files – just their pointers
						# prepend your git clone with the following env var:
					

					GIT_LFS_SKIP_SMUDGE=1
Done
token-classification mask_token: [MASK]
Query this model
🔥 This model is currently loaded and running on the Inference API. ⚠️ This model could not be loaded by the inference API. ⚠️ This model can be loaded on the Inference API on-demand.
JSON Output
API endpoint  

⚡️ Upgrade your account to access the Inference API

Share Copied link to clipboard

Contributed by

sagorsarker Sagor Sarker
10 models
Hosted on huggingface.co

codeswitch-hineng-lid-lince

This is a pretrained model for language identification of hindi-english code-mixed data used from LinCE

This model is trained for this below repository.

https://github.com/sagorbrur/codeswitch

To install codeswitch:

pip install codeswitch

Identify Language

  • Method-1

from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("sagorsarker/codeswitch-hineng-lid-lince")

model = AutoModelForTokenClassification.from_pretrained("sagorsarker/codeswitch-hineng-lid-lince")
lid_model = pipeline('ner', model=model, tokenizer=tokenizer)

lid_model("put any hindi english code-mixed sentence")
  • Method-2
from codeswitch.codeswitch import LanguageIdentification
lid = LanguageIdentification('hin-eng') 
text = "" # your code-mixed sentence 
result = lid.identify(text)
print(result)