Upload folder using huggingface_hub

CRISPR_transformer_model/README.md

@@ -0,0 +1,88 @@
+---
+library_name: transformers
+tags:
+- generated_from_trainer
+datasets:
+- crispr_data
+model-index:
+- name: SX_ispymac_CRISPR_transformer
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# SX_ispymac_CRISPR_transformer
+
+This model is a fine-tuned version of [](https://huggingface.co/) on the crispr_data dataset.
+It achieves the following results on the evaluation set:
+- Loss: 514639.75
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 100
+- eval_batch_size: 100
+- seed: 63036
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_ratio: 0.05
+- num_epochs: 30.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 5601914.5031  | 1.0   | 326  | 1894021.75      |
+| 1328778.8957  | 2.0   | 652  | 670578.75       |
+| 669992.0982   | 3.0   | 978  | 528672.8125     |
+| 549468.5644   | 4.0   | 1304 | 528315.75       |
+| 528034.8957   | 5.0   | 1630 | 517330.3438     |
+| 521024.6871   | 6.0   | 1956 | 516302.25       |
+| 518358.3804   | 7.0   | 2282 | 516195.9375     |
+| 517674.454    | 8.0   | 2608 | 516099.6875     |
+| 516467.3865   | 9.0   | 2934 | 515717.6562     |
+| 516283.1411   | 10.0  | 3260 | 516509.0        |
+| 515811.4356   | 11.0  | 3586 | 515362.1875     |
+| 515361.1779   | 12.0  | 3912 | 517239.9375     |
+| 515421.5951   | 13.0  | 4238 | 515716.4688     |
+| 515266.2577   | 14.0  | 4564 | 514972.875      |
+| 514957.9877   | 15.0  | 4890 | 514995.125      |
+| 514826.3558   | 16.0  | 5216 | 515276.4688     |
+| 514742.8221   | 17.0  | 5542 | 514949.9375     |
+| 514670.9693   | 18.0  | 5868 | 515125.5        |
+| 514493.3988   | 19.0  | 6194 | 514852.7812     |
+| 514367.3129   | 20.0  | 6520 | 514896.3125     |
+| 514363.1411   | 21.0  | 6846 | 515543.625      |
+| 514226.3067   | 22.0  | 7172 | 514779.0        |
+| 514206.135    | 23.0  | 7498 | 514666.75       |
+| 514089.4233   | 24.0  | 7824 | 514983.75       |
+| 514020.908    | 25.0  | 8150 | 514966.5625     |
+| 513915.8282   | 26.0  | 8476 | 514978.0938     |
+| 513902.9202   | 27.0  | 8802 | 514722.125      |
+| 513746.454    | 28.0  | 9128 | 514688.625      |
+| 513719.0675   | 29.0  | 9454 | 514638.1562     |
+| 513652.319    | 30.0  | 9780 | 514639.75       |
+
+
+### Framework versions
+
+- Transformers 4.44.2
+- Pytorch 2.4.0+cu124
+- Datasets 2.21.0
+- Tokenizers 0.19.1

CRISPR_transformer_model/config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "/home/ljw/sdc1/CRISPR_results/CRISPR_transformer/SX_ispymac_CRISPR_transformer",
   "architectures": [
     "CRISPRTransformerModel"
   ],

CRISPR_transformer_model/model.py

@@ -0,0 +1,85 @@
+from transformers import PreTrainedModel, RoFormerConfig, RoFormerModel
+import torch.nn as nn
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+class CRISPRTransformerConfig(RoFormerConfig):
+    model_type = "CRISPR_transformer"
+    label_names = ["observation"]
+
+    def __init__(
+        self,
+        vocab_size = 4, # ACGT
+        hidden_size = 256, # model embedding dimension
+        num_hidden_layers = 3, # number of EncoderLayer
+        num_attention_heads = 4, # number of attention heads
+        intermediate_size = 1024, # FeedForward intermediate dimension size
+        hidden_dropout_prob = 0.1, # The dropout probability for all fully connected layers in the embeddings, encoder, and pooler
+        attention_probs_dropout_prob = 0.1, # The dropout ratio for the attention probabilities
+        max_position_embeddings = 256, # The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 1536).
+        ref1len = 127, # length of reference 1
+        ref2len = 127, # length of reference 2
+        seed = 63036, # random seed for intialization
+        **kwargs
+    ):
+        self.ref1len = ref1len
+        self.ref2len = ref2len
+        self.seed = seed
+        super().__init__(
+            vocab_size = vocab_size,
+            hidden_size = hidden_size,
+            num_hidden_layers = num_hidden_layers,
+            num_attention_heads = num_attention_heads,
+            intermediate_size = intermediate_size,
+            hidden_dropout_prob = hidden_dropout_prob,
+            attention_probs_dropout_prob = attention_probs_dropout_prob,
+            max_position_embeddings = max_position_embeddings,
+            **kwargs
+        )
+
+class CRISPRTransformerModel(PreTrainedModel):
+    config_class = CRISPRTransformerConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.generator = torch.Generator().manual_seed(config.seed)
+        self.model = RoFormerModel(config)
+        self.mlp = nn.Linear(
+            in_features=config.hidden_size,
+            out_features=(config.ref1len + 1) * (config.ref2len + 1)
+        )
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        for m in self.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.normal_(m.weight, mean=0, std=1, generator=self.generator)
+                if m.bias is not None:
+                    nn.init.constant_(m.bias, 0)
+
+    def forward(self, refcode: torch.Tensor, observation: torch.Tensor=None):
+        # refcode (batch_size X sequence_length)
+        # model(refcode) (batch_size X sequence_length X hidden_size)
+        # model(refcode)[:, -1, :] arbitrary choose the last position to predict the logits
+        batch_size = refcode.shape[0]
+        logit = self.mlp(
+            self.model(
+                input_ids=refcode,
+                attention_mask=torch.ones(
+                    batch_size,
+                    self.config.ref1len + self.config.ref2len,
+                    dtype=torch.int64,
+                    device=self.model.device
+                )
+            ).last_hidden_state[:, -1, :]
+        ).view(batch_size, self.config.ref2len + 1, self.config.ref1len + 1)
+        if observation is not None:
+            return {
+                "logit": logit,
+                "loss": - (
+                    observation.flatten(start_dim=1) *
+                    F.log_softmax(logit.flatten(start_dim=1), dim=1)
+                ).sum()
+            }
+        return {"logit": logit}

CRISPR_transformer_model/runs/Nov18_13-19-23_ljw-System-Product-Name/events.out.tfevents.1731907165.ljw-System-Product-Name.35960.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcaba17bb30e7ec94f7abe72eb17745d1bede8b4937023a5812a31f5e26c42d4
+size 19851

CRISPR_transformer_model/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74fb56f007b2e91b3fac96d9bc9f26ea19800479595e6b2d6891946b34402dc9
+size 5304