|
--- |
|
license: mit |
|
widget: |
|
- text: "MERVAVVGVPMDLGANRRGVDMGPSALRYARLLEQLEDLGYTVEDLGDVPVSLARASRRRGRGLAYLEEIRAAALVLKERLAALPEGVFPIVLGGDHSLSMGSVAGAARGRRVGVVWVDAHADFNTPETSPSGNVHGMPLAVLSGLGHPRLTEVFRAVDPKDVVLVGVRSLDPGEKRLLKEAGVRVY" |
|
--- |
|
|
|
## Label Semantics: |
|
|
|
Label 0: Non-crystallizable (Negative) |
|
|
|
Label 1: Crystallizable (Positive) |
|
|
|
## Dataset |
|
|
|
1. [DeepCrystal Train](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/blob/main/Datasets/train.csv) |
|
2. [DeepCrystal Test](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/blob/main/Datasets/test.csv) |
|
3. [BCrystal Test](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/tree/main/Datasets/BCrystal_Balanced_Test_set) |
|
4. [SP Test](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/tree/main/Datasets/SP_Final_set) |
|
5. [TR Test](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/tree/main/Datasets/TR_Final_set) |
|
|
|
|
|
## Model |
|
|
|
### ESMCrystal_t6_8M_v1 |
|
|
|
ESMCrystal_t6_8M_v1 is a state-of-the-art protein crystallization prediction model finetuned on [esm2_t6_8M_UR50D](https://huggingface.co/facebook/esm2_t6_8M_UR50D), |
|
having 6 layers and 8M parameters with the size of [approx. 31.4MB](https://huggingface.co/jaykmr/ESMCrystal_t6_8M_v1/blob/main/pytorch_model.bin) |
|
using transfer learning to predict whether an input protein sequence will crystallize or not. |
|
|
|
## Accuracy : |
|
|
|
|
|
| Dataset | Accuracy | |
|
|------------------|--------------------| |
|
| DeepCrystal Test | 0.7913593256059009 | |
|
| BCrystal test | 0.7811975377728035 | |
|
| SP test | 0.6962025316455697 | |
|
| TR test | 0.8191699604743083 | |
|
|
|
## Comparision Table: |
|
|
|
| | Count | Positives | Negatives | TP | FP | FN | TN | Precision | Recall | F1 | Accuracy | ROC | Mathew's Coefficient | PPV | NPV | |
|
|---------------|-------|-----------|-----------|-----|-----|----|-----|------------|------------|------------|------------|--------|----------------------|------------|------------| |
|
| | | | | | | | | | | | | | | | | |
|
| Test | 1898 | 898 | 1000 | 532 | 362 | 34 | 966 | 0.5950783 | 0.93992933 | 0.72876712 | 0.79091869 | 0.9467 | 0.611906376 | 0.5950783 | 0.966 | |
|
| | | | | | | | | | | | | | | | | |
|
| BCrystal Test | 1787 | 891 | 896 | 531 | 360 | 31 | 865 | 0.5959596 | 0.94483986 | 0.73090158 | 0.78119754 | 0.9396 | 0.604504011 | 0.5959596 | 0.96540179 | |
|
| | | | | | | | | | | | | | | | | |
|
| SP Test | 237 | 148 | 89 | 80 | 68 | 4 | 85 | 0.54054054 | 0.95238095 | 0.68965517 | 0.69620253 | 0.9328 | 0.501728679 | 0.54054054 | 0.95505618 | |
|
| | | | | | | | | | | | | | | | | |
|
| TR Test | 1012 | 374 | 638 | 207 | 167 | 16 | 622 | 0.55347594 | 0.92825112 | 0.69346734 | 0.81916996 | 0.9562 | 0.615341231 | 0.55347594 | 0.97492163 | |
|
| | | | | | | | | | | | | | | | | |
|
|
|
## Graphs |
|
|
|
### ROC-AUC Curve |
|
|
|
* DeepCrystal Test |
|
 |
|
|
|
* BCrystal Test |
|
 |
|
|
|
* SP Test |
|
 |
|
|
|
* TR Test |
|
 |
|
|
|
### PR-AUC Curve |
|
|
|
* DeepCrystal Test |
|
 |
|
|
|
* BCrystal Test |
|
 |
|
|
|
* SP Test |
|
 |
|
|
|
* TR Test |
|
 |
|
|
|
|
|
|
|
## Final scores : |
|
|
|
|
|
* on DeepCrystal test: |
|
|
|
| | precision | recall | f1-score | support | |
|
|--------------------|-----------|--------|----------|---------| |
|
| non-crystallizable | 0.73 | 0.97 | 0.83 | 1000 | |
|
| crystallizable | 0.94 | 0.60 | 0.73 | 898 | |
|
| accuracy | | | 0.79 | 1898 | |
|
| macro avg | 0.83 | 0.78 | 0.78 | 1898 | |
|
| weighted avg | 0.83 | 0.79 | 0.78 | 1898 | |
|
|
|
* on BCrystal test: |
|
|
|
| | precision | recall | f1-score | support | |
|
|--------------------|-----------|--------|----------|---------| |
|
| non-crystallizable | 0.71 | 0.97 | 0.82 | 896 | |
|
| crystallizable | 0.94 | 0.60 | 0.73 | 891 | |
|
| accuracy | | | 0.78 | 1787 | |
|
| macro avg | 0.83 | 0.78 | 0.77 | 1787 | |
|
| weighted avg | 0.83 | 0.78 | 0.77 | 1787 | |
|
|
|
* on SP test: |
|
|
|
| | precision | recall | f1-score | support | |
|
|--------------------|-----------|--------|----------|---------| |
|
| non-crystallizable | 0.56 | 0.96 | 0.70 | 89 | |
|
| crystallizable | 0.95 | 0.54 | 0.69 | 148 | |
|
| accuracy | | | 0.70 | 237 | |
|
| macro avg | 0.75 | 0.75 | 0.70 | 237 | |
|
| weighted avg | 0.80 | 0.70 | 0.69 | 237 | |
|
|
|
* on TR test: |
|
|
|
| | precision | recall | f1-score | support | |
|
|--------------------|-----------|--------|----------|---------| |
|
| non-crystallizable | 0.79 | 0.97 | 0.87 | 638 | |
|
| crystallizable | 0.93 | 0.55 | 0.69 | 374 | |
|
| accuracy | | | 0.82 | 1012 | |
|
| macro avg | 0.86 | 0.76 | 0.78 | 1012 | |
|
| weighted avg | 0.84 | 0.82 | 0.81 | 1012 | |
|
|
|
|
|
## Confusion matrix: |
|
|
|
* on DeepCrystal test: |
|
|
|
``` |
|
| 532 | 362 | |
|
| 34 | 966 | |
|
``` |
|
|
|
* on BCrystal test: |
|
|
|
``` |
|
| 531 | 360 | |
|
| 31 | 865 | |
|
``` |
|
|
|
* on SP test: |
|
|
|
``` |
|
| 80 | 68 | |
|
| 4 | 85 | |
|
``` |
|
|
|
* on TR test: |
|
|
|
``` |
|
| 207 | 167 | |
|
| 16 | 622 | |
|
``` |
|
|
|
## Metrics |
|
|
|
|
|
roc score: |
|
|
|
* on DeepCrystal test: 0.9467594654788418 |
|
|
|
* on BCrystal test: 0.946546316337983 |
|
|
|
* on SP test: 0.9328120255086547 |
|
|
|
* on TR test: 0.9562804888270497 |
|
|
|
Mathews Coefficient: |
|
|
|
* on DeepCrystal test: 0.6130826598876417 |
|
|
|
* on BCrystal test: 0.6045040114572474 |
|
|
|
* on SP test: 0.5017286791304684 |
|
|
|
* on TR test: 0.6153412305503776 |
|
|
|
NPV: |
|
|
|
* on DeepCrystal test: 0.966 |
|
|
|
* on BCrystal test: 0.9654017857142857 |
|
|
|
* on SP test: 0.9550561797752809 |
|
|
|
* on TR test: 0.9749216300940439 |
|
|
|
PPV: |
|
|
|
* on DeepCrystal test: 0.5968819599109132 |
|
|
|
* on BCrystal test: 0.5959595959595959 |
|
|
|
* on SP test: 0.5405405405405406 |
|
|
|
* on TR test: 0.553475935828877 |
|
|
|
|
|
Researchers: |
|
|
|
* [Jayanth Kumar](https://jaykmr.com) |
|
* [Kavya Jaykumar](https://www.linkedin.com/in/kavya-jayakumar-6390271b5/) |
|
|
|
Credits: |
|
|
|
* [Meta ESMFold2](https://github.com/facebookresearch/esm) |
|
* [Huggingface](https://huggingface.co/jaykmr) |
|
* [Paperspace Compute Cloud](https://www.paperspace.com/) |
