--- '[object Object]': null license: cc language: - en library_name: adapter-transformers pipeline_tag: text-classification --- # Model Card for orYx-models/finetuned-roberta-leadership-sentiment-analysis - **Model Description:** This model is a finetuned version of the RoBERTa text classifier(cardiffnlp/twitter-roberta-base-sentiment-latest). It has been trained on a dataset comprising communications from corporate executives to their therapists. Its primary function is to determine whether statements from corporate executives convey a "Positive," "Negative," or "Neutral" sentiment, accompanied by a confidence level indicating the percentage of sentiment expressed in a statement. Being a prototype tool by orYx Models, all feedbacks and insights will be used to further refine the model. ## Model Details ### Model Information - **Model Type:** Text Classifier - **Language(s):** English - **License:** Creative Commons license family - **Finetuned from Model:** cardiffnlp/twitter-roberta-base-sentiment-latest ### Model Sources - **HuggingFace Model ID:** cardiffnlp/twitter-roberta-base-2021-124m - **Paper:** TimeLMs - [Link](https://arxiv.org/abs/2202.03829) ## Uses - **Use case:** This sentiment analysis tool can analyze text from any user within an organization, such as executives, employees, or clients, and assign a sentiment to it. - **Outcomes:** The tool generates a "Scored sentiment" which can be used to assess the likelihood of events occurring or vice versa. It can also facilitate the creation of a rating system based on the sentiments expressed in texts. ### Direct Use ```python nlp = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer) nlp("The results don't match, but the effort seems to be always high") Out[7]: [{'label': 'Positive', 'score': 0.9996090531349182}] ``` - Based on the text the outcomes can be "Positive, Negative, Neutral" along with their confidence score. - ### Recommendations - **Continuous Monitoring:** Regularly monitor the model's performance on new data to ensure its effectiveness and reliability over time. - **Error Analysis:** Conduct thorough error analysis to identify common patterns of misclassifications and areas for improvement. - **Fine-Tuning:** Consider fine-tuning the model further based on feedback and insights from users, to enhance its domain-specific performance. - **Model Interpretability:** Explore techniques for explaining the model's predictions, such as attention mechanisms or feature importance analysis, to increase trust and understanding of its decisions. ## Training Details ``` X_train, X_val, y_train, y_val = train_test_split(X,y, test_size = 0.2, stratify = y) ``` - **Train data:** 80% of 4396 records = 3516 - **Test data:** 20% of 4396 records = 879 ### Training Procedure - **Dataset Split:** Data divided into 80% training and 20% validation sets. - **Preprocessing:** Input data tokenized into 'input_ids' and 'attention_mask' tensors. - **Training Hyperparameters:** Set for training, evaluation, and optimization, including batch size, epochs, and logging strategies. - **Training Execution:** Model trained with specified hyperparameters, monitored with metrics, and logged for evaluation. - **Evaluation Metrics:** Model evaluated on loss, accuracy, F1 score, precision, and recall for both training and validation sets. #### Preprocessing [optional] ``` 'input_ids': tensor 'attention_mask': tensor 'label': tensor(2) ``` #### Training Hyperparameters ``` args = TrainingArguments( output_dir="output", do_train = True, do_eval = True, num_train_epochs = 1, per_device_train_batch_size = 4, per_device_eval_batch_size = 8, warmup_steps = 50, weight_decay = 0.01, logging_strategy= "steps", logging_dir= "logging", logging_steps = 50, eval_steps = 50, save_strategy = "steps", fp16 = True, #load_best_model_at_end = True ) ``` #### Speeds, Sizes, Times [optional] - **TrainOutput** ``` global_step=879, training_loss=0.1825900522650848, ``` - **Metrics** ``` 'train_runtime': 101.6309, 'train_samples_per_second': 34.596, 'train_steps_per_second': 8.649, 'total_flos': 346915041274368.0, 'train_loss': 0.1825900522650848, 'epoch': 1.0 ``` ## Evaluation Metrics Results ``` # Assuming you have a list of evaluation results q and want to create a DataFrame with it q = [Trainer.evaluate(eval_dataset=df) for df in [train_dataset, val_dataset]] # Create DataFrame with index and select only the first 5 columns result_df = pd.DataFrame(q, index=["train", "val"]).iloc[:,:5] # Display the resulting DataFrame print(result_df) ______________________________________________________________________ eval_loss eval_Accuracy eval_F1 eval_Precision eval_Recall train 0.049349 0.988908 0.987063 0.982160 0.992357 val 0.108378 0.976136 0.972464 0.965982 0.979861 ______________________________________________________________________ ``` **loss** - train 0.049349 - val 0.108378 **Accuracy** - train 0.988908 - **98.8%** - val 0.976136 - **97.6%** **F1** - train 0.987063 - **98.7%** - val 0.972464 - **97.2%** **Precision** - train 0.982160 - **98.2%** - val 0.965982 - **96.5%** **Recall** - train 0.992357 - **99.2%** - val 0.979861 - **97.9%** ## Environmental Impact Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). - **Hardware Type:** T4 GPU - **Hours used:** 2 - **Cloud Provider:** Google - **Compute Region:** India - **Carbon Emitted:** No Information Available ### Compute Infrastructure Google Colab - T4 GPU ### References ``` @inproceedings{camacho-collados-etal-2022-tweetnlp, title = "{T}weet{NLP}: Cutting-Edge Natural Language Processing for Social Media", author = "Camacho-collados, Jose and Rezaee, Kiamehr and Riahi, Talayeh and Ushio, Asahi and Loureiro, Daniel and Antypas, Dimosthenis and Boisson, Joanne and Espinosa Anke, Luis and Liu, Fangyu and Mart{\'\i}nez C{\'a}mara, Eugenio" and others, booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations", month = dec, year = "2022", address = "Abu Dhabi, UAE", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2022.emnlp-demos.5", pages = "38--49" } ``` ## Model Card Authors [optional] Vineedhar, relkino ## Model Card Contact https://khalidalhosni.com/