Public commit

README.md

@@ -1,12 +1,52 @@
----
-title: LLMGeneLinker LGL V1
-emoji: 👀
-colorFrom: indigo
-colorTo: pink
-sdk: gradio
-sdk_version: 3.39.0
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# LLMGeneLinker (LGL): a Fine-Tuned SciBERT Model for Named Entity Recognition
+
+LLMGeneLinker uses a domain-specific transformer like SciBERT finetuned on AllenAI drug dataset, BC5CDR disease, NCBI disease, DrugProt and GeneTAG datasets. The resulting SciBERT model performs Named Entity Recognition to tag drug, protein, gene, diseases in input text. Sentence embedding of SciBERT is then fed into BERT 
+
+## Table of Contents
+
+- [Model Overview](#model-overview)
+- [Usage](#usage)
+- [Installation](#installation)
+
+- [Dataset](#dataset)
+
+- [Contributing](#contributing)
+- [License](#license)
+
+## Model Overview
+
+The model is based on the [SciBERT](https://github.com/allenai/scibert) architecture, which is a pre-trained language model specifically designed for the biomedical domain. By fine-tuning SciBERT on a labeled dataset, we have created a specialized NER model that can accurately recognize drugs, genes, and diseases in biomedical texts.
+
+## Usage
+You can access an interactive web interface for querying the fine-tuned LGL model [here](spacelink). If you prefer to load the model yourself, you can check out [Installation](#installation) below.
+
+## Installation
+To use LGL, you need to install the required dependencies and download the model files. Follow the steps below to set up the environment:
+
+1. Clone this repository to your local machine.
+1.1 If you do not have Python installed, download python via the official sources. Anaconda is recommended if you use scientific packages often.
+
+If using anaconda, after installation setup a new conda environment via the following (replace *myname* with your own choice of environment name):
+```conda create --name *myname* python==3.8```
+
+2. Activate your venv/ conda env (if using) and install the required Python packages using `pip`:
+
+```pip install -r requirements.txt```
+
+3. To utilize the fine-tuned NER model for recognizing drugs, genes, and diseases, you can open `demo.ipynb` in Jupyter Lab by starting Jupyter Lab via ```jupyter lab```. The script takes text input as a string and returns the identified entities along with their respective labels.
+
+## Dataset
+
+The following datasets were processed and used for training and evaluation:
+Most datasets were sourced from `BigBIO` [GitHub] (https://github.com/bigscience-workshop/biomedical/blob/main/README.md) [HF] (https://huggingface.co/bigbio)
+
+| Task Type | Dataset       | Links ||
+|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
+| NER       | NCBI-disease  | [Link](https://huggingface.co/datasets/bigbio/ncbi_disease)| 
+| NER       | BC5-disease   | [Link](https://huggingface.co/datasets/bigbio/bc5cdr)| 
+| NER       | Genetag    | [Link](https://huggingface.co/datasets/bigbio/genetag)|
+| NER/RE | Drugprot      | [Link](https://huggingface.co/datasets/bigbio/drugprot)|
+| NER/RE | AllenAI Drug-Combo-Extraction        | [Link](https://huggingface.co/datasets/allenai/drug-combo-extraction)|
+
+
+

app.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+# In[1]:
+
+
+from datasets import Dataset, ClassLabel, Sequence, load_dataset, load_metric
+import numpy as np
+import pandas as pd
+import bioc
+from spacy import displacy
+import transformers
+#import evaluate
+from transformers import (AutoModelForTokenClassification, 
+                          AutoTokenizer, 
+                          DataCollatorForTokenClassification,
+                          pipeline,
+                          TrainingArguments, 
+                          Trainer)
+
+
+# In[2]:
+
+
+label_list = ['O', 'B-DRUG', 'I-DRUG', 'B-DISEASE', 'I-DISEASE', 'B-GENE', 'I-GENE']
+model_checkpoint = './trainedSB2'
+tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
+model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=len(label_list))
+effect_ner_model = pipeline(task="ner", model=model, tokenizer=tokenizer)
+
+
+# In[21]:
+
+
+def visualize_entities(sentence):
+    tokens = effect_ner_model(sentence)
+    entities = []
+    # ['O', 'B-DRUG', 'I-DRUG', 'B-DISEASE', 'I-DISEASE', 'B-GENE', 'I-GENE']
+    for token in tokens:
+        label = int(token["entity"][-1])
+        if label != 0:
+            token["label"] = label_list[label]
+            entities.append(token)
+    
+    params = [{"text": sentence,
+               "ents": entities,
+               "title": None}]
+    
+    html = displacy.render(params, style="ent", manual=True, options={
+        "colors": {
+                    
+                   "B-DRUG": "#f08080",
+                   "I-DRUG": "#f08080",
+                   "B-DISEASE": "#9bddff",
+                   "I-DISEASE": "#9bddff",
+                    "B-GENE": "#008080",
+                   "I-GENE": "#008080",
+               },
+    })
+    return html
+    
+
+
+# In[25]:
+
+
+import gradio as gr
+
+exampleList = [
+    'Famotidine is a histamine H2-receptor antagonist used in inpatient settings for prevention of stress ulcers and is showing increasing popularity because of its low cost.',
+    'A randomized Phase III trial demonstrated noninferiority of APF530 500 mg SC ( granisetron 10 mg ) to intravenous palonosetron 0.25 mg in preventing CINV in patients receiving MEC or HEC in acute ( 0 - 24 hours ) and delayed ( 24 - 120 hours ) settings , with activity over 120 hours .',
+    'What are the known interactions between Aspirin and the COX-1 enzyme?',
+    'Can you explain the mechanism of action of Metformin and its effect on the AMPK pathway?',
+    'Are there any genetic variations in the CYP2C9 gene that may influence the response to Warfarin therapy?',
+    'I am curious about the role of Herceptin in targeting the HER2/neu protein in breast cancer treatment. How does it work?',
+    'What are the common side effects associated with Lisinopril, an angiotensin-converting enzyme (ACE) inhibitor?',
+    'Can you explain the significance of the BCR-ABL fusion protein in the context of Imatinib therapy for chronic myeloid leukemia (CML)?',
+    'How does Ibuprofen affect the COX-2 enzyme compared to COX-1?',
+    'Are there any recent studies exploring the use of Pembrolizumab as an immune checkpoint inhibitor targeting PD-1?',
+    'I have heard about the SLC6A4 gene and its association with serotonin reuptake inhibitors (SSRIs) like Fluoxetine.',
+    'Could you provide insights into the BRAF mutation and its relevance in response to Vemurafenib treatment in melanoma patients?'
+]
+
+footer = """
+LLMGeneLinker uses a domain-specific transformer like SciBERT finetuned on AllenAI drug dataset, BC5CDR disease, NCBI disease, DrugProt and GeneTAG datasets. The resulting SciBERT model performs Named Entity Recognition to tag drug, protein, gene, diseases in input text. Sentence embedding of SciBERT is then fed into BERT 
+This was made during the <a target="_blank" href =https://www.sginnovate.com/event/hackathon-large-language-models-bio> LLMs for Bio Hackathon</a> organised by 4Catalyzer and SGInnovate.
+<br>
+Made by Team GeneLink (<a target="_blank" href=https://www.linkedin.com/in/ntkb/>Nicholas</a>, <a target="_blank" href=https://www.linkedin.com/in/yewchong-sim/>Yew Chong</a>, <a target="_blank" href=https://www.linkedin.com/in/lim-ting-wei-021383175/>Ting Wei</a>,  <a target="_blank" href=https://www.linkedin.com/in/brendan-lim-ciwen/>Brendan</a>
+<hr>
+Note: Performance is noted to be poorer on genes, acronyms, and receptors (named entities that may be targets for drugs or genes)
+Original notebook adapted from <a target="_blank" href=https://huggingface.co/jsylee/scibert_scivocab_uncased-finetuned-ner>jsylee/scibert_scivocab_uncased-finetuned-ner</a>
+"""
+
+with gr.Blocks() as demo:
+    gr.Markdown("## LLMGeneLinker (LGL)")
+    gr.Markdown(footer)
+    
+    txt = gr.Textbox(label="Input", lines=2)
+    txt_3 = gr.HTML(label="Output")
+    btn = gr.Button(value="Submit")
+    btn.click(visualize_entities, inputs=txt, outputs=txt_3)
+
+    gr.Markdown("## Text Examples")
+    gr.Examples(
+        [[x] for x in exampleList],
+        txt,
+        txt_3,
+        visualize_entities,
+        cache_examples=False,
+        run_on_click=True
+    )
+
+
+if __name__ == "__main__":
+    demo.launch()
+
+
+# In[ ]:
+
+
+
+

requirements.txt

@@ -0,0 +1,9 @@
+huggingface_hub
+jupyterlab
+datasets
+transformers[torch]
+seqeval
+spacy
+evaluate
+bioc
+gradio

trainedSB2/config.json

@@ -0,0 +1,43 @@
+{
+  "_name_or_path": "allenai/scibert_scivocab_uncased",
+  "architectures": [
+    "BertForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2",
+    "3": "LABEL_3",
+    "4": "LABEL_4",
+    "5": "LABEL_5",
+    "6": "LABEL_6"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2,
+    "LABEL_3": 3,
+    "LABEL_4": 4,
+    "LABEL_5": 5,
+    "LABEL_6": 6
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.31.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 31090
+}

trainedSB2/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b09b5b25e4643e98d3404b02ad8e83801ef9c7025d1904e59d2f8d76e65eb883
+size 437400745

trainedSB2/special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

trainedSB2/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

trainedSB2/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a97024dafc1a94923419f85c63ab33f440e46391ec27dfec53864654217b0a3
+size 4027