ilos-vigil/bigbird-small-indonesian-nli

Indonesian small BigBird model NLI

Source Code

Source code to create this model and perform benchmark is available at https://github.com/ilos-vigil/bigbird-small-indonesian.

Model Description

This model is based on bigbird-small-indonesian and was finetuned on 2 datasets. It is intended to be used for zero-shot text classification.

How to use

Inference for ZSC (Zero Shot Classification) task

>>> pipe = pipeline(
...     task='zero-shot-classification',
...     model='./tmp/checkpoint-28832'
... )
>>> pipe(
...     sequences='Fakta nomor 7 akan membuat ada terkejut',
...     candidate_labels=['clickbait', 'bukan clickbait'],
...     hypothesis_template='Judul video ini {}.',
...     multi_label=False
... )
{
 'sequence': 'Fakta nomor 7 akan membuat ada terkejut',
 'labels': ['clickbait', 'bukan clickbait'],
 'scores': [0.6102734804153442, 0.38972654938697815]
}
>>> pipe(
...     sequences='Samsung tuntut balik Apple dengan alasan hak paten teknologi.',
...     candidate_labels=['teknologi', 'olahraga', 'bisnis', 'politik', 'kesehatan', 'kuliner'],
...     hypothesis_template='Kategori berita ini adalah {}.',
...     multi_label=True
... )
{
 'sequence': 'Samsung tuntut balik Apple dengan alasan hak paten teknologi.',
 'labels': ['politik', 'teknologi', 'kesehatan', 'bisnis', 'olahraga', 'kuliner'],
 'scores': [0.7390161752700806, 0.6657379269599915, 0.4459509551525116, 0.38407933712005615, 0.3679264783859253, 0.14181996881961823]
}

Inference for NLI (Natural Language Inference) task

>>> pipe = pipeline(
...     task='text-classification',
...     model='./tmp/checkpoint-28832',
...     return_all_scores=True
... )
>>> pipe({
...     'text': 'Nasi adalah makanan pokok.',  # Premise
...     'text_pair': 'Saya mau makan nasi goreng.'  # Hypothesis
... })
[
 {'label': 'entailment', 'score': 0.25495028495788574},
 {'label': 'neutral', 'score': 0.40920916199684143},
 {'label': 'contradiction', 'score': 0.33584052324295044}
]
>>> pipe({
...     'text': 'Python sering digunakan untuk web development dan AI research.',
...     'text_pair': 'AI research biasanya tidak menggunakan bahasa pemrograman Python.'
... })
[
 {'label': 'entailment', 'score': 0.12508109211921692},
 {'label': 'neutral', 'score': 0.22146646678447723},
 {'label': 'contradiction', 'score': 0.653452455997467}
]

Limitation and bias

This model inherit limitation/bias from it's parent model and 2 datasets used for fine-tuning. And just like most language model, this model is sensitive towards input change. Here's an example.

>>> from transformers import pipeline
>>> pipe = pipeline(
...     task='zero-shot-classification',
...     model='./tmp/checkpoint-28832'
... )
>>> text = 'Resep sate ayam enak dan mudah.'
>>> candidate_labels = ['kuliner', 'olahraga']
>>> pipe(
...     sequences=text,
...     candidate_labels=candidate_labels,
...     hypothesis_template='Kategori judul artikel ini adalah {}.',
...     multi_label=False
... )
{
 'sequence': 'Resep sate ayam enak dan mudah.',
 'labels': ['kuliner', 'olahraga'],
 'scores': [0.7711364030838013, 0.22886358201503754]
}
>>> pipe(
...     sequences=text,
...     candidate_labels=candidate_labels,
...     hypothesis_template='Kelas kalimat ini {}.',
...     multi_label=False
... )
{
 'sequence': 'Resep sate ayam enak dan mudah.',
 'labels': ['kuliner', 'olahraga'],
 'scores': [0.7043636441230774, 0.295636385679245]
}
>>> pipe(
...     sequences=text,
...     candidate_labels=candidate_labels,
...     hypothesis_template='{}.',
...     multi_label=False
... )
{
 'sequence': 'Resep sate ayam enak dan mudah.',
 'labels': ['kuliner', 'olahraga'],
 'scores': [0.5986711382865906, 0.4013288915157318]
}

Training, evaluation and testing data

This model was finetuned with IndoNLI and multilingual-NLI-26lang-2mil7. Although multilingual-NLI-26lang-2mil7 dataset is machine-translated, this dataset slightly improve result of NLI benchmark and extensively improve result of ZSC benchmark. Both evaluation and testing data is only based on IndoNLI dataset.

Training Procedure

The model was finetuned on single RTX 3060 with 16 epoch/28832 steps with accumulated batch size 64. AdamW optimizer is used with LR 1e-4, weight decay 0.05, learning rate warmup for first 6% steps (1730 steps) and linear decay of the learning rate afterwards. Take note while model weight on epoch 9 has lowest loss/highest accuracy, it has slightly lower performance on ZSC benchmark. Additional information can be seen on Tensorboard training logs.

Benchmark as NLI model

Both benchmark show result of 2 different model as additional comparison. Additional benchmark using IndoNLI dataset is available on it's paper IndoNLI: A Natural Language Inference Dataset for Indonesian.

Model	bigbird-small-indonesian-nli	xlm-roberta-large-xnli	mDeBERTa-v3-base-xnli-multilingual-nli-2mil7
Parameter	30.6M	559.9M	278.8M
Multilingual		V	V
Finetuned on IndoNLI	V		V
Finetuned on multilingual-NLI-26lang-2mil7	V
Test (Lay)	0.6888	0.2226	0.8151
Test (Expert)	0.5734	0.3505	0.7775

Benchmark as ZSC model

Indonesian-Twitter-Emotion-Dataset is used to perform ZSC benchmark. This benchmark include 4 different parameter which affect performance of each model differently. Hypothesis template for this benchmark is Kalimat ini mengekspresikan perasaan {}. and {}.. Take note F1 score measurement only calculate label with highest probability.

Model	Multi-label	Use template	F1 Score
bigbird-small-indonesian-nli	V	V	0.3574
	V		0.3654
		V	0.3985
			0.4160
xlm-roberta-large-xnli	V	V	0.6292
	V		0.5596
		V	0.5737
			0.5433
mDeBERTa-v3-base-xnli-multilingual-nli-2mil7	V	V	0.5324
	V		0.5499
		V	0.5269
			0.5228