Hugging Face's logo

nbroad
/
ESG-BERT

Text Classification
Transformers
PyTorch
Safetensors
English
bert
Model card Files Community
15
ESG-BERT / README.md
nazneen's picture
nazneen
model documentation
1ae536b
preview code
|
raw
history blame
10.6 kB
 ---
language:
- en
---
# Model Card for ESG-BERT
Domain Specific BERT Model for Text Mining in Sustainable Investing
# Model Details
## Model Description
- **Developed by:** [Charan Pothireddi](https://www.linkedin.com/in/sree-charan-pothireddi-6a0a3587/) and [Parabole.ai](https://www.linkedin.com/in/sree-charan-pothireddi-6a0a3587/)
- **Shared by [Optional]:** HuggingFace
- **Model type:** Language model
- **Language(s) (NLP):** en
- **License:** More information needed
- **Related Models:**
- **Parent Model:** BERT
- **Resources for more information:**
- [GitHub Repo](https://github.com/mukut03/ESG-BERT)
- [Blog Post](https://towardsdatascience.com/nlp-meets-sustainable-investing-d0542b3c264b?source=friends_link&sk=1f7e6641c3378aaff319a81decf387bf)
# Uses
## Direct Use
Text Mining in Sustainable Investing
## Downstream Use [Optional]
The applications of ESG-BERT can be expanded way beyond just text classification. It can be fine-tuned to perform various other downstream NLP tasks in the domain of Sustainable Investing.
## Out-of-Scope Use
The model should not be used to intentionally create hostile or alienating environments for people.
# Bias, Risks, and Limitations
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
## Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recomendations.
# Training Details
## Training Data
More information needed
## Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
### Preprocessing
More information needed
### Speeds, Sizes, Times
More information needed
# Evaluation
## Testing Data, Factors & Metrics
### Testing Data
The fine-tuned model for text classification is also available [here](https://drive.google.com/drive/folders/1Qz4HP3xkjLfJ6DGCFNeJ7GmcPq65_HVe?usp=sharing). It can be used directly to make predictions using just a few steps. First, download the fine-tuned pytorch_model.bin, config.json, and vocab.txt
### Factors
More information needed
### Metrics
More information needed
## Results
ESG-BERT was further trained on unstructured text data with accuracies of 100% and 98% for Next Sentence Prediction and Masked Language Modelling tasks. Fine-tuning ESG-BERT for text classification yielded an F-1 score of 0.90. For comparison, the general BERT (BERT-base) model scored 0.79 after fine-tuning, and the sci-kit learn approach scored 0.67.
# Model Examination
More information needed
# Environmental Impact
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** More information needed
- **Hours used:** More information needed
- **Cloud Provider:** information needed
- **Compute Region:** More information needed
- **Carbon Emitted:** More information needed
# Technical Specifications [optional]
## Model Architecture and Objective
More information needed
## Compute Infrastructure
More information needed
### Hardware
More information needed
### Software
JDK 11 is needed to serve the model
# Citation
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
More information needed
**APA:**
More information needed
# Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
More information needed
# More Information [optional]
More information needed
# Model Card Authors [optional]
[Charan Pothireddi](https://www.linkedin.com/in/sree-charan-pothireddi-6a0a3587/) and [Parabole.ai](https://www.linkedin.com/in/sree-charan-pothireddi-6a0a3587/), in collaboration with the Ezi Ozoani and the HuggingFace Team
# Model Card Contact
More information needed
# How to Get Started with the Model
Use the code below to get started with the model.
<details>
<summary> Click to expand </summary>
```
pip install torchserve torch-model-archiver
pip install torchvision
pip install transformers
```
Next up, we'll set up the handler script. It is a basic handler for text classification that can be improved upon. Save this script as "handler.py" in your directory. [1]
```
from abc import ABC
import json
import logging
import os
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from ts.torch_handler.base_handler import BaseHandler
logger = logging.getLogger(__name__)
class TransformersClassifierHandler(BaseHandler, ABC):
"""
Transformers text classifier handler class. This handler takes a text (string) and
as input and returns the classification text based on the serialized transformers checkpoint.
"""
def __init__(self):
super(TransformersClassifierHandler, self).__init__()
self.initialized = False
def initialize(self, ctx):
self.manifest = ctx.manifest
properties = ctx.system_properties
model_dir = properties.get("model_dir")
self.device = torch.device("cuda:" + str(properties.get("gpu_id")) if torch.cuda.is_available() else "cpu")
# Read model serialize/pt file
self.model = AutoModelForSequenceClassification.from_pretrained(model_dir)
self.tokenizer = AutoTokenizer.from_pretrained(model_dir)
self.model.to(self.device)
self.model.eval()
logger.debug('Transformer model from path {0} loaded successfully'.format(model_dir))
# Read the mapping file, index to object name
mapping_file_path = os.path.join(model_dir, "index_to_name.json")
if os.path.isfile(mapping_file_path):
with open(mapping_file_path) as f:
self.mapping = json.load(f)
else:
logger.warning('Missing the index_to_name.json file. Inference output will not include class name.')
self.initialized = True
def preprocess(self, data):
""" Very basic preprocessing code - only tokenizes.
Extend with your own preprocessing steps as needed.
"""
text = data[0].get("data")
if text is None:
text = data[0].get("body")
sentences = text.decode('utf-8')
logger.info("Received text: '%s'", sentences)
inputs = self.tokenizer.encode_plus(
sentences,
add_special_tokens=True,
return_tensors="pt"
)
return inputs
def inference(self, inputs):
"""
Predict the class of a text using a trained transformer model.
"""
# NOTE: This makes the assumption that your model expects text to be tokenized
# with "input_ids" and "token_type_ids" - which is true for some popular transformer models, e.g. bert.
# If your transformer model expects different tokenization, adapt this code to suit
# its expected input format.
prediction = self.model(
inputs['input_ids'].to(self.device),
token_type_ids=inputs['token_type_ids'].to(self.device)
)[0].argmax().item()
logger.info("Model predicted: '%s'", prediction)
if self.mapping:
prediction = self.mapping[str(prediction)]
return [prediction]
def postprocess(self, inference_output):
# TODO: Add any needed post-processing of the model predictions here
return inference_output
_service = TransformersClassifierHandler()
def handle(data, context):
try:
if not _service.initialized:
_service.initialize(context)
if data is None:
return None
data = _service.preprocess(data)
data = _service.inference(data)
data = _service.postprocess(data)
return data
except Exception as e:
raise e
```
TorcheServe uses a format called MAR (Model Archive). We can convert our PyTorch model to a .mar file using this command:
```
torch-model-archiver --model-name "bert" --version 1.0 --serialized-file ./bert_model/pytorch_model.bin --extra-files "./bert_model/config.json,./bert_model/vocab.txt" --handler "./handler.py"
```
Move the .mar file into a new directory:
```
mkdir model_store && mv bert.mar model_store
```
Finally, we can start TorchServe using the command:
```
torchserve --start --model-store model_store --models bert=bert.mar
```
We can now query the model from another terminal window using the Inference API. We pass a text file containing text that the model will try to classify.
```
curl -X POST http://127.0.0.1:8080/predictions/bert -T predict.txt
```
This returns a label number which correlates to a textual label. This is stored in the label_dict.txt dictionary file.
```
__label__Business_Ethics : 0
__label__Data_Security : 1
__label__Access_And_Affordability : 2
__label__Business_Model_Resilience : 3
__label__Competitive_Behavior : 4
__label__Critical_Incident_Risk_Management : 5
__label__Customer_Welfare : 6
__label__Director_Removal : 7
__label__Employee_Engagement_Inclusion_And_Diversity : 8
__label__Employee_Health_And_Safety : 9
__label__Human_Rights_And_Community_Relations : 10
__label__Labor_Practices : 11
__label__Management_Of_Legal_And_Regulatory_Framework : 12
__label__Physical_Impacts_Of_Climate_Change : 13
__label__Product_Quality_And_Safety : 14
__label__Product_Design_And_Lifecycle_Management : 15
__label__Selling_Practices_And_Product_Labeling : 16
__label__Supply_Chain_Management : 17
__label__Systemic_Risk_Management : 18
__label__Waste_And_Hazardous_Materials_Management : 19
__label__Water_And_Wastewater_Management : 20
__label__Air_Quality : 21
__label__Customer_Privacy : 22
__label__Ecological_Impacts : 23
__label__Energy_Management : 24
__label__GHG_Emissions : 25
```
<\details>