Back to all models
Model: huseinzol05/xlnet-base-bahasa-cased

Monthly model downloads

huseinzol05/xlnet-base-bahasa-cased huseinzol05/xlnet-base-bahasa-cased
- downloads
last 30 days



Contributed by

huseinzol05 husein zolkepli
2 models

How to use this model directly from the 🤗/transformers library:

Copy model
tokenizer = AutoTokenizer.from_pretrained("huseinzol05/xlnet-base-bahasa-cased") model = AutoModelWithLMHead.from_pretrained("huseinzol05/xlnet-base-bahasa-cased")

Bahasa XLNet Model

Pretrained XLNet base language model for Malay and Indonesian.

Pretraining Corpus

XLNET-base-bahasa-cased model was pretrained on ~1.8 Billion words. We trained on both standard and social media language structures, and below is list of data we trained on,

  1. dumping wikipedia.
  2. local instagram.
  3. local twitter.
  4. local news.
  5. local parliament text.
  6. local singlish/manglish text.
  7. IIUM Confession.
  8. Wattpad.
  9. Academia PDF.

Preprocessing steps can reproduce from here, Malaya/pretrained-model/preprocess.

Pretraining details

Load Pretrained Model

You can use this model by installing torch or tensorflow and Huggingface library transformers. And you can use it directly by initializing it like this:

from transformers import XLNetTokenizer, XLNetModel

model = XLNetModel.from_pretrained('huseinzol05/xlnet-base-bahasa-cased')
tokenizer = XLNetTokenizer.from_pretrained(
    'huseinzol05/xlnet-base-bahasa-cased', do_lower_case = False

Example using AutoModelWithLMHead

from transformers import AlbertTokenizer, AutoModelWithLMHead, pipeline

model = AutoModelWithLMHead.from_pretrained('huseinzol05/xlnet-base-bahasa-cased')
tokenizer = XLNetTokenizer.from_pretrained(
    'huseinzol05/xlnet-base-bahasa-cased', do_lower_case = False
fill_mask = pipeline('fill-mask', model = model, tokenizer = tokenizer)
print(fill_mask('makan ayam dengan [MASK]'))


For further details on the model performance, simply checkout accuracy page from Malaya,, we compared with traditional models.


Thanks to Im Big, LigBlou, Mesolitica and KeyReply for sponsoring AWS, Google and GPU clouds to train XLNet for Bahasa.