metadata
license: apache-2.0
pipeline_tag: text-classification
language:
- yue
widget:
- text: 係唔係廣東話？
example_title: Cantonese
- text: 台灣真美!
example_title: Traditional Chinese
Model Description
A BERT-based model trained to classify text as either Cantonese or Traditional Chinese.
Intended Use
- Primary Application: Language classification for Cantonese and Traditional Chinese texts.
- Users: NLP researchers, developers working with Chinese language data.
Training Data
Utilizes the "raptorkwok/cantonese-traditional-chinese-parallel-corpus" from Hugging Face Datasets.
Training Procedure
- Base Model:
bert-base-chinese - Epochs: 3
- Learning Rate: 2e-5
- Evaluation at the end of each epoch, saving the best model.
How to Use
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("ming030890/chinese-langid")
model = AutoModelForSequenceClassification.from_pretrained("ming030890/chinese-langid")
text = "係唔係廣東話？"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# 0 for Cantonese, 1 for Traditional Chinese
prediction = outputs.logits.argmax(-1).item()