nazneen commited on
Commit
5396bc1
1 Parent(s): 53d4525

model documentation

Browse files
Files changed (1) hide show
  1. README.md +172 -0
README.md ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - feature-extraction
4
+ - bert
5
+ ---
6
+
7
+ # Model Card for biosyn-sapbert-bc5cdr-disease
8
+
9
+ # Model Details
10
+
11
+ ## Model Description
12
+
13
+ More information needed
14
+ - **Developed by:** DMIS-lab (Data Mining and Information Systems Lab, Korea University)
15
+ - **Shared by [Optional]:** Jinhyuk Lee
16
+ - **Model type:** Feature Extraction
17
+ - **Language(s) (NLP):** More information needed
18
+ - **License:** More information needed
19
+ - **Parent Model:** BERT
20
+ - **Resources for more information:**
21
+ - [GitHub Repo](https://github.com/jhyuklee/biobert)
22
+ - [Associated Paper](https://arxiv.org/abs/1901.08746)
23
+
24
+
25
+
26
+
27
+
28
+ # Uses
29
+
30
+
31
+ ## Direct Use
32
+ This model can be used for the task of feature extraction.
33
+
34
+ ## Downstream Use [Optional]
35
+
36
+ More information needed.
37
+
38
+ ## Out-of-Scope Use
39
+
40
+ The model should not be used to intentionally create hostile or alienating environments for people.
41
+
42
+ # Bias, Risks, and Limitations
43
+
44
+
45
+ Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
46
+
47
+ ## Recommendations
48
+
49
+
50
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
51
+
52
+ # Training Details
53
+
54
+ ## Training Data
55
+
56
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
57
+ > We used the BERTBASE model pre-trained on English Wikipedia and BooksCorpus for 1M steps. BioBERT v1.0 (þ PubMed þ PMC) is the version of BioBERT (þ PubMed þ PMC) trained for 470 K steps. When using both the PubMed and PMC corpora, we found that 200K and 270K pre-training steps were optimal for PubMed and PMC, respectively. We also used the ablated versions of BioBERT v1.0, which were pre-trained on only PubMed for 200K steps (BioBERT v1.0 (þ PubMed)) and PMC for 270K steps (BioBERT v1.0 (þ PMC))
58
+
59
+
60
+ ## Training Procedure
61
+
62
+
63
+ ### Preprocessing
64
+
65
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
66
+ > We pre-trained BioBERT using Naver Smart Machine Learning (NSML) (Sung et al., 2017), which is utilized for large-scale experiments that need to be run on several GPUs
67
+
68
+ ### Speeds, Sizes, Times
69
+
70
+ The model creators note in the [associated paper](https://arxiv.org/pdf/1901.08746.pdf):
71
+ > The maximum sequence length was fixed to 512 and the mini-batch size was set to 192, resulting in 98 304 words per iteration.
72
+
73
+ # Evaluation
74
+
75
+
76
+ ## Testing Data, Factors & Metrics
77
+
78
+ ### Testing Data
79
+
80
+ More information needed
81
+
82
+ ### Factors
83
+ More information needed
84
+
85
+ ### Metrics
86
+
87
+ More information needed
88
+
89
+ ## Results
90
+
91
+ More information needed
92
+
93
+ # Model Examination
94
+
95
+ More information needed
96
+
97
+ # Environmental Impact
98
+
99
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
100
+
101
+ - **Hardware Type:** More information needed
102
+ - **Training:** Eight NVIDIA V100 (32GB) GPUs [ for training],
103
+ - **Fine-tuning:** a single NVIDIA Titan Xp (12GB) GPU to fine-tune BioBERT on each task
104
+ - **Hours used:** More information needed
105
+ - **Cloud Provider:** More information needed
106
+ - **Compute Region:** More information needed
107
+ - **Carbon Emitted:** More information needed
108
+
109
+ # Technical Specifications [optional]
110
+
111
+ ## Model Architecture and Objective
112
+
113
+ More information needed
114
+
115
+ ## Compute Infrastructure
116
+
117
+ More information needed
118
+
119
+ ### Hardware
120
+
121
+
122
+ More information needed
123
+
124
+ ### Software
125
+
126
+ More information needed.
127
+
128
+ # Citation
129
+
130
+
131
+ **BibTeX:**
132
+ ```bibtex
133
+ @article{lee2019biobert,
134
+ title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
135
+ author={Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},
136
+ journal={arXiv preprint arXiv:1901.08746},
137
+ year={2019}
138
+ }
139
+ ```
140
+
141
+
142
+ # Glossary [optional]
143
+
144
+ More information needed
145
+
146
+ # More Information [optional]
147
+ For help or issues using BioBERT, please submit a GitHub issue. Please contact Jinhyuk Lee(`lee.jnhk (at) gmail.com`), or Wonjin Yoon (`wonjin.info (at) gmail.com`) for communication related to BioBERT.
148
+
149
+ # Model Card Authors [optional]
150
+
151
+ DMIS-lab (Data Mining and Information Systems Lab, Korea University) in collaboration with Ezi Ozoani and the Hugging Face team
152
+
153
+ # Model Card Contact
154
+
155
+ More information needed
156
+
157
+ # How to Get Started with the Model
158
+
159
+ Use the code below to get started with the model.
160
+
161
+ <details>
162
+ <summary> Click to expand </summary>
163
+
164
+ ```python
165
+ from transformers import AutoTokenizer, AutoModel
166
+
167
+ tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biosyn-sapbert-bc5cdr-disease")
168
+
169
+ model = AutoModel.from_pretrained("dmis-lab/biosyn-sapbert-bc5cdr-disease")
170
+ ```
171
+ </details>
172
+