distilBert_NER_finer

This model is a fine-tuned version of distilbert-base-cased on the Finer-139 dataset. It achieves the following results on the evaluation set:

• Loss: 0.0198
• Precision: 0.9445
• Recall: 0.9640
• F1: 0.9541
• Accuracy: 0.9954

Training and evaluation data

The training data consists of the 4 most widely available ner_tags from the Finer-139 dataset. The training and the test data were curated from this source accordingly

Prediction procedure

from transformers import TAutoTokenizer
from optimum.onnxruntime import ORTModelForTokenClassification
import torch

def onnx_inference(checkpoint, test_data, export=False):
  test_text = " ".join(test_data['tokens'])
  print("Test Text: " + test_text)

  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  model = ORTModelForTokenClassification.from_pretrained(checkpoint, export=export)

  inputs = tokenizer(test_text, return_tensors="pt")
  outputs = model(**inputs).logits

  predictions = torch.argmax(outputs, dim=2)

  # Convert each tensor element to a scalar before calling .item()
  predicted_token_class = [label_list[int(t)] for t in predictions[0]]
  ner_tags = [label_list[int(t)] for t in test_data['ner_tags']]

  print("Original Tags: ")
  print(ner_tags)
  print("Predicted Tags: ")
  print(predicted_token_class)

onnx_model_path = "" #add the path

onnx_inference(onnx_model_path, test_data)

"""
Here the test_data should contain "tokens" and "ner_tags". This can be of type Dataset.
"""

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
0.0034	1.0	1620	0.0261	0.9167	0.9668	0.9411	0.9941
0.0031	2.0	3240	0.0182	0.9471	0.9651	0.9561	0.9956
0.0012	3.0	4860	0.0198	0.9445	0.9640	0.9541	0.9954

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2