metadata

library_name: transformers
license: mit
language:
  - en
metrics:
  - f1
pipeline_tag: text2text-generation

Model Card for Model ID

Model Details

Model Description

Developed by: Reforged by nicolay-r, initial credits for implementation to scofield7419
Model type: Flan-T5
Language(s) (NLP): English
License: Apache License 2.0

Model Sources

Repository: Reasoning-for-Sentiment-Analysis-Framework
Paper: https://arxiv.org/abs/2404.12342
Demo: We have a code on Google-Colab for launching the related model

Uses

Direct Use

Downstream Use

Please refer to the related section of the Reasoning-for-Sentiment-Analysis Framework

Out-of-Scope Use

This model represent a fine-tuned version of the Flan-T5 on RuSentNE-2023 dataset. Since dataset represent three-scale output answers (positive, negative, neutral), the behavior in general might be biased to this particular task.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

Please proceed with the code from the related Three-Hop-Reasoning CoT section.

Or following the related section on Google Colab notebook

Training Details

Training Data

We utilize train data which was automatically translated into English using GoogleTransAPI. The initial source of the texts written in Russian, is from the following repository: https://github.com/dialogue-evaluation/RuSentNE-evaluation

The translated version on the dataset in English could be automatically downloaded via the following script: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/rusentne23_download.py

Training Procedure

This model has been trained using the Three-hop-Reasoning framework, proposed in the paper: https://arxiv.org/abs/2305.11255

For training procedure accomplishing, the reforged version of this framework was used: https://github.com/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework

Google-colab notebook for reproduction: https://colab.research.google.com/github/nicolay-r/Reasoning-for-Sentiment-Analysis-Framework/blob/main/Reasoning_for_Sentiment_Analysis_Framework.ipynb

The overall training process took 4 epochs.

Training Hyperparameters

Training regime: All the configuration details were highlighted in the related config file

Evaluation

Testing Data, Factors & Metrics

Testing Data

The direct link to the test evaluation data: https://github.com/dialogue-evaluation/RuSentNE-evaluation/blob/main/final_data.csv

Metrics

For the model evaluation, two metrics were used:

F1_PN -- F1-measure over positive and negative classes;
F1_PN0 -- F1-measure over positive, negative, and neutral classes;

Results

Result: F1_PN = 60.024

Below is the log of the training process that showcases the final peformance on the RuSentNE-2023 test set after 4 epochs (lines 5-6):

    F1_PN  F1_PN0  default   mode
0  45.523  59.375   59.375  valid
1  62.345  70.260   70.260  valid
2  62.722  70.704   70.704  valid
3  62.721  70.671   70.671  valid
4  62.357  70.247   70.247  valid
5  60.024  68.171   68.171   test
6  60.024  68.171   68.171   test