rcfberg/bert-general-nonmarket-communication

BERT General Non-Market Communication

This repository supplements paper published at Business & Society [Ref]. It contains a BERT (Bidirectional Encoder Representations from Transformers) model trained for text classification to separate general non-market communicaton from social media content, along with robustness tests and instructions for running inference. Notice: the model is trained on Twitter/X posts and may not work as intended on other data or use cases.

Model Specifications

This model is a BERT-based text classifier. It was trained on a proprietary X/Twitter manually annotated dataset (n=1604), with the goal of classifying the post as either general non-market communication (1) or not (0). A researcher labeled every message, while a research assistant performed inter‑coder reliability checks on 10 % - randomly selected - of the sample. An unweighted Cohen’s κ of 0.821 was obtained for the two raters, which is considered excellent. The fine-tuned model had the following datasplit using a seeded random shuffle: training 64% / validation 16% / test 20%. Our reported metrics from an inferred instance of this model were an Accuracy of 0.85, a Weighted F1 score of 0.85, and Cohen's Kappa of 0.665.

Bert-large-cased was used as the base model.

Training Parameters

We used the following parameters in our model training.

training_args = TrainingArguments(
    output_dir=output_dir,
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    warmup_ratio=0.1,
    lr_scheduler_type="linear",
    num_train_epochs=2,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    report_to=[],
    push_to_hub=False,
    logging_dir=f"logs_seed_{SEED}",
    seed=SEED,
    data_seed=42,
)

Two training epochs were deemed sufficient for our small training dataset, given the risk of overfitting, aligning with other studies with small datasets (e.g., Areshey and Mathkour, 2023; Choudhary and Arora, 2024; Devlin et al, 2019). These parameters were consistent across multiple training runs.

Robustness Tests

Deep learning model training involves random processes. To address this, we retrained the model six times using random fixed seeds. With the exact parameters of the trained models being fixed, the overall performance and the main conclusions of our study remained consistent, as shown in the summary table:

Metric	Mean	Standard Deviation	Our reported model
Accuracy	0.8479	0.0207	0.8507
F1_Weighted	0.8478	0.0197	0.8501
F1_Macro	0.8304	0.0209	0.8326
Cohen_Kappa	0.6609	0.0209	0.6651

Using the model for inference

These instructions will guide you through running the model to make predictions on new text data. We have tried to make these steps clear for users who may not well-versed in working with deep learning models.

Prerequisites:

1. Python: You need Python 3.10 or higher installed on your computer.

2. Python Packages (Libraries): Python uses "packages" (also called libraries) to provide extra functionality. You'll need to install the following:

   • transformers (version 4.33.2)
   • torch
   • pandas
   • numpy
   • sklearn
   • os

   You can install these using pip, a package installer for Python. Open your terminal (or command prompt) and run:

   pip install transformers==4.33.2 torch pandas numpy scikit-learn random os
   

Steps:

1. Download the Model: Download the contents of the classifier_output folder from this repository. You can either clone this repository using git or download the folder as a ZIP file. Place this folder on your computer (e.g., your Desktop or a dedicated folder).

2. Locate inference.py: Inside the downloaded classifier_output folder, you'll find a Python script called inference.py.

3. Configure inference.py: Open inference.py in a text editor. You'll need to modify a few lines to tell the script where to find the model and where to save the results.

   • model_dir: Change this to the full path to the folder where you saved the classifier_output folder. For example, if you put it on your Desktop, it might look like:

     model_dir = "/Users/your_username/Desktop/classifier_output"  # macOS/Linux
     # or
     model_dir = "C:\\Users\\your_username\\Desktop\\classifier_output"  # Windows
     

     Important: Replace your_username with your actual username!

   • metrics_path: Choose a location to save the performance metrics (e.g., accuracy, F1-score) that will be calculated on your test data. Provide the full path. Example:

     metrics_path = "/Users/your_username/Documents/metrics.txt"
     

   • predict_from_csv() function: This function reads your input data from a CSV file. You need to tell it where this file is located:

     # Inside the predict_from_csv() function:
     input_csv_path = "/Users/your_username/Documents/input_data.csv"
     

     Important: Your CSV file must have a column named "text" containing the text you want to classify.

   • output_csv_path: Choose a location and filename to save the predictions. The script will create a new CSV file with your original text and the model's predicted labels. Example:

     output_csv_path = "/Users/your_username/Documents/predictions.csv"
     

4. Run the Script: Open your terminal (or command prompt), navigate to the directory containing inference.py (using the cd command), and run the script:

   python inference.py
   

The script will load the model, make predictions on your data, and save the results to the specified output CSV file. It will also print the performance metrics to the console and save them to the metrics_path file.