Edit model card

Training

This model adapter is designed for token classification tasks, enabling it to extract aspect terms and predict the sentiment polarity associated with the extracted aspect terms. The extracted aspect terms will be the span(s) from the input text on which a sentiment is being expressed. It has been created using PEFT framework for LoRA:Low-Rank Adaptation.

Datasets

This model has been trained on the following datasets:

  1. Aspect Based Sentiment Analysis SemEval Shared Tasks (2014, 2015, 2016)
  2. Multi-Aspect Multi-Sentiment MAMS

Use

  • Loading the base model and merging it with LoRA parameters
from transformers import AutoTokenizer, AutoModelForTokenClassification
from peft import PeftModel

# preparing the labels
labels = {"B-neu": 1, "I-neu": 2, "O": 0, "B-neg": 3, "B-con": 4, "I-pos": 5, "B-pos": 6, "I-con": 7, "I-neg": 8, "X": -100}
id2labels = {k:lab for lab, k in labels.items()}
labels2ids = {k:lab for lab, k in id2labels.items()}

# loading tokenizer and base_model
base_id = 'microsoft/deberta-v3-large'
tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForTokenClassification.from_pretrained(base_id, num_labels=len(labels), id2label=id2labels, label2id=labels2ids)

# using this adapter with base model 
model = PeftModel.from_pretrained(base_model, 'gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter', is_trainable=False)

This model can be utilized in the following two methods:

  1. Using pipelines for end to end inference
  2. Making token level inference

Using end-to-end token classification pipeline

from transformers import pipeline
ate_sent_pipeline = pipeline(task='ner', 
                  aggregation_strategy='simple',
                  model=model,
                  tokenizer=tokenizer)

text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
ate_sent_pipeline(text_input)

Expected output

[{'entity_group': 'pos',
  'score': 0.89315575,
  'word': 'food',
  'start': 25,
  'end': 30},
 {'entity_group': 'neg',
  'score': 0.9100719,
  'word': 'service',
  'start': 55,
  'end': 63}]

OR

Making token level inference

# making one prediction at a time (should be padded/batched and truncated for efficiency)
text_input = "Been here a few times and food has always been good but service really suffers when it gets crowded."
tok_inputs = tokenizer(text_input, return_tensors="pt")


y_pred = model(**tok_inputs) # predicting the logits

# since first and the last tokens are excluded ([CLS] and [SEP]) they have to be removed before decoding
y_pred_fin = y_pred.logits.argmax(dim=-1)[0][1:-1] # selecting the most favoured labels for each token from the logits


decoded_pred = [id2lab[logx.item()] for logx in y_pred_fin]


## displaying the input tokens with predictions and skipping [CLS] and [SEP] tokens at the beginning and the end respectively
decoded_toks = tok_inputs['input_ids'][0][1:-1]
tok_levl_pred = list(zip(tokenizer.convert_ids_to_tokens(decoded_toks), decoded_pred))

Expected Results

[('▁Been', 'O'),
 ('▁here', 'O'),
 ('▁a', 'O'),
 ('▁few', 'O'),
 ('▁times', 'O'),
 ('▁and', 'O'),
 ('▁food', 'B-pos'),
 ('▁has', 'O'),
 ('▁always', 'O'),
 ('▁been', 'O'),
 ('▁good', 'O'),
 ('▁but', 'O'),
 ('▁service', 'B-neg'),
 ('▁really', 'O'),
 ('▁suffers', 'O'),
 ('▁when', 'O'),
 ('▁it', 'O'),
 ('▁gets', 'O'),
 ('▁crowded', 'O'),
 ('.', 'O')]

Evaluation on Benchmark Test Datasets

The first evaluation is for token-extraction task without considering the polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens on which the sentiments have been expressed. (scores are expressed as micro-averages of B-I-O labels)

ATE (Aspect Term Extraction Only)

Test Dataset Base Model Fine-tuned Model Precision Recall F1 Score
hotel reviews (SemEval 2015) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 71.16 73.92 71.6
hotel reviews (SemEval 2015) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 70.92 72.28 71.07
hotel reviews (SemEval 2015) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 64.05 79.69 70.0
hotel reviews (SemEval 2015) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 66.29 72.78 68.92
------------ ---------- ---------------- --------- ------ --------
laptop reviews (SemEval 2014) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 70.58 61.52 64.21
laptop reviews (SemEval 2014) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 66.38 50.62 54.31
laptop reviews (SemEval 2014) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 70.82 48.97 52.08
laptop reviews (SemEval 2014) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 73.61 46.38 49.87
------------ ---------- ---------------- --------- ------ --------
MAMS-ATE (2019) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 81.07 79.66 80.35
MAMS-ATE (2019) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 79.91 78.95 79.39
MAMS-ATE (2019) microsoft/deberta-v3-large (this)gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 74.46 84.5 78.75
MAMS-ATE (2019) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 77.8 79.81 78.75
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2014) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 88.59 87.0 87.45
restaurant reviews (SemEval 2014) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 92.26 82.95 86.57
restaurant reviews (SemEval 2014) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 93.07 81.95 86.32
restaurant reviews (SemEval 2014) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 92.94 81.71 86.01
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2015) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 72.91 75.4 72.74
restaurant reviews (SemEval 2015) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 70.54 77.48 72.63
restaurant reviews (SemEval 2015) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 68.32 79.84 72.28
restaurant reviews (SemEval 2015) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 71.94 74.75 71.84
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2016) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 70.22 75.83 71.84
restaurant reviews (SemEval 2016) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 71.54 73.38 71.2
restaurant reviews (SemEval 2016) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 71.35 72.78 70.85
restaurant reviews (SemEval 2016) microsoft/deberta-v3-large (this)gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 66.68 77.97 70.79

Aspect Sentiment Evaluation

This evaluation considers token-extraction task with polarity of the extracted tokens. The tokens expected to be extracted are aspect term tokens on which the sentiments have been expressed along with the polarity of the sentiments. (scores are expressed as macro-averages)

Test Dataset Base Model Fine-tuned Model Precision Recall F1 Score
hotel reviews (SemEval 2015) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 51.92 65.55 54.94
hotel reviews (SemEval 2015) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 54.62 53.65 54.08
hotel reviews (SemEval 2015) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 55.43 56.53 54.03
hotel reviews (SemEval 2015) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 52.88 55.19 53.85
------------ ---------- ---------------- --------- ------ --------
laptop reviews (SemEval 2014) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 44.25 41.55 42.81
laptop reviews (SemEval 2014) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 46.15 33.23 37.09
laptop reviews (SemEval 2014) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 41.7 34.38 36.93
laptop reviews (SemEval 2014) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 44.98 31.87 35.67
------------ ---------- ---------------- --------- ------ --------
MAMS-ATE (2019) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 72.06 72.98 72.49
MAMS-ATE (2019) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 72.97 71.63 72.26
MAMS-ATE (2019) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 69.34 73.3 71.07
MAMS-ATE (2019) microsoft/deberta-v3-large (this)gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 65.74 75.11 69.77
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2014) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 61.15 58.46 59.74
restaurant reviews (SemEval 2014) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 60.13 56.81 58.13
restaurant reviews (SemEval 2014) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 56.79 59.3 57.93
restaurant reviews (SemEval 2014) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 58.99 54.76 56.45
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2015) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 53.89 55.7 54.11
restaurant reviews (SemEval 2015) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 54.36 55.38 53.6
restaurant reviews (SemEval 2015) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 51.67 56.58 53.29
restaurant reviews (SemEval 2015) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 54.55 53.68 53.12
------------ ---------- ---------------- --------- ------ --------
restaurant reviews (SemEval 2016) FacebookAI/roberta-large gauneg/roberta-large-absa-ate-sentiment-lora-adapter 53.7 60.49 55.05
restaurant reviews (SemEval 2016) FacebookAI/roberta-base gauneg/roberta-base-absa-ate-sentiment 52.31 54.58 52.33
restaurant reviews (SemEval 2016) microsoft/deberta-v3-base gauneg/deberta-v3-base-absa-ate-sentiment 52.07 54.58 52.15
restaurant reviews (SemEval 2016) microsoft/deberta-v3-large (this) gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter 49.07 56.5 51.25
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for gauneg/deberta-v3-large-absa-ate-sentiment-lora-adapter

Finetuned
(116)
this model