End of training

README.md

@@ -1,66 +1,74 @@
 ---
-license: mit
-datasets:
-- mfarrington/biobert-ner-fda-recalls-dataset
+base_model: dmis-lab/biobert-base-cased-v1.2
+tags:
+- generated_from_trainer
 metrics:
+- precision
+- recall
+- f1
 - accuracy
-pipeline_tag: token-classification
-tags:
-- medical
+model-index:
+- name: devicebert-base-cased-v1.0
+  results: []
 ---
-# Model Card for Model ID
-
-Pretrained BioBERT Model for Performing Named Entity Recognition (NER) of Medical Device Names, Components and Part Numbers. 
-
-## Model Details
-
-### Abstract
-*FDA Medical Device recalls are critical and time-sensitive events, requiring
-swift identification of impacted devices to inform the public of a recall event and
-ensure patient safety. The OpenFDA device recall dataset contains valuable
-information about ongoing device recall actions, but manually extracting relevant
-device information from the recall action summaries is a time-consuming task.
-Named Entity Recognition (NER) is a task in Natural Language Processing
-(NLP) that involves identifying and categorizing named entities in unstructured text.
-Existing NER models, including domain-specific models like BioBERT, struggle
-to correctly identify medical device trade names, part numbers and component terms within these
-summaries. To address this, we propose DeviceBERT, a medical device annotation, pre-processing and enrichment pipeline, which builds on BioBERT to identify and label medical device terminology in the device recall summaries with improved accuracy. Furthermore, we demonstrate that our approach can be applied effectively for performing entity recognition tasks where training data is limited or sparse.*
-
-### Model Description
 
-This model was created as part of a student final project for Stanford CS224N Spring 2024.
-Project Title: "Optimizing BioBERT to Perform Named Entity Recognition of Medical Devices Using FDA Device Recall Summaries"
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-DeviceBERT utilizes BioBERT (dmis-lab/biobert-base-cased-v1.2) which has been trained on PubMed and PMC and subsequently finetuned to accurately identify industry specific medical device trade names, component parts and part numbers. 
+# devicebert-base-cased-v1.0
 
-The model was trained utilizing a processed and annotated NER dataset created using the OpenFDA Device Recalls Dataset (https://open.fda.gov/apis/device/recall/), and further
-tokenized using the DistilBERT AutoTokenizer. It can be used to perform inferencing to accurately identfiy and label medical device, device component, part number and trade name related terms in a variety of downstream applications. 
+This model is a fine-tuned version of [dmis-lab/biobert-base-cased-v1.2](https://huggingface.co/dmis-lab/biobert-base-cased-v1.2) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: nan
+- Precision: 0.6816
+- Recall: 0.6691
+- F1: 0.6753
+- Accuracy: 0.8547
 
-- **Developed by:** Miriam Farrington for CS224N
-- **Model type:** Deep Learning Language Model/LLM
-- **Language(s) (NLP):** Python, TensorFlow
-- **License:** MIT
-- **Finetuned from model [optional]:** BioBERT (dmis-lab/biobert-base-cased-v1.2)
+## Model description
 
-### Model Sources [optional]
+More information needed
 
-- **Repository:** https://github.com/mmfarrington/devicebert-ner-project
-- **Paper:** "DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summaries"
+## Intended uses & limitations
 
-## Uses
+More information needed
 
-### Direct Use
+## Training and evaluation data
 
-This model can be directly used to perform NER inferencing on text to identify medical device related terms, trade names, components and part numbers in a variety of downstream tasks.
-The model can be applied further to generate human feedback loops using inferencing to generate additional NER data for more complex downstream tasks or additional finetuning.
+More information needed
 
-## Training Details
+## Training procedure
 
-### Training Data
+### Training hyperparameters
 
-mfarrington/biobert-ner-fda-recalls-dataset
+The following hyperparameters were used during training:
+- learning_rate: 1e-05
+- train_batch_size: 16
+- eval_batch_size: 16
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 10
 
+### Training results
 
+| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
+|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
+| No log        | 1.0   | 101  | nan             | 0.5981    | 0.5740 | 0.5858 | 0.8131   |
+| No log        | 2.0   | 202  | nan             | 0.6673    | 0.6197 | 0.6427 | 0.8424   |
+| No log        | 3.0   | 303  | nan             | 0.6926    | 0.6673 | 0.6797 | 0.8498   |
+| No log        | 4.0   | 404  | nan             | 0.686     | 0.6271 | 0.6552 | 0.8473   |
+| 0.3891        | 5.0   | 505  | nan             | 0.6853    | 0.6490 | 0.6667 | 0.8539   |
+| 0.3891        | 6.0   | 606  | nan             | 0.6857    | 0.7020 | 0.6938 | 0.8563   |
+| 0.3891        | 7.0   | 707  | nan             | 0.6900    | 0.6673 | 0.6784 | 0.8580   |
+| 0.3891        | 8.0   | 808  | nan             | 0.6795    | 0.6782 | 0.6789 | 0.8514   |
+| 0.3891        | 9.0   | 909  | nan             | 0.6906    | 0.6691 | 0.6797 | 0.8571   |
+| 0.1315        | 10.0  | 1010 | nan             | 0.6816    | 0.6691 | 0.6753 | 0.8547   |
 
 
+### Framework versions
 
+- Transformers 4.41.2
+- Pytorch 2.3.0+cu121
+- Datasets 2.19.2
+- Tokenizers 0.19.1

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fcc925315e0e239dddfa9cfee10132c820ce176fba0f68ed7368e2b7b3ef8bcf
+oid sha256:5d5ac909e05c9af2fedd3552dc8c5b4c4357d3155f34bac3d225d04bce007404
 size 928741200

runs/Jun07_18-40-05_6d0acad3a460/events.out.tfevents.1717785615.6d0acad3a460.17459.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dc79f61278c6cdde8a6c17db07e1156aebe41e73cf7c99782bb8a46aee780bf
-size 9453
+oid sha256:c6fbb28ea6c9a34b9f30d89c36c94159ccd47563a6dd8cd9486ab0aed78bf260
+size 10490