Create README.md

README.md

@@ -0,0 +1,35 @@
+<h1>Model descriptions</h1>
+
+PAIR ([paper](https://www.biorxiv.org/content/10.1101/2024.07.22.604688)) is a flexible fine-tuning framework to improve the quality of protein representations for function predictions. PAIR uses a text decoder to guide the fine-tuning process of a protein encoder so that the learned representations could extract information contained within the diverse set of annotations in Swiss-Prot. This model fine-tunes ESM2-650M ([repo](https://huggingface.co/facebook/esm2_t33_650M_UR50D)) with PAIR.  
+
+<h1>Intended use</h1>
+
+The model can be used for feature extractions in protein function prediction tasks. 
+
+<h1>How to load the model?</h1>
+
+```python
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained("facebook/esm2_t33_650M_UR50D")
+model = AutoModel.from_pretrained("h4duan/PAIR-esm2")
+```
+
+<h1>How to extract the features?</h1>
+
+```python
+proteins = ["AETCZAO","SKTZP"]
+def extract_features(proteins):
+  ids = tokenizer(proteins, return_tensors="pt", padding=True, max_length=1024, truncation=True, return_attention_mask=True)
+  input_ids = torch.tensor(ids['input_ids']).to(self.device)
+  attention_mask = torch.tensor(ids['attention_mask']).to(self.device)
+  with torch.no_grad():
+    embedding_repr = self.model(output_hidden_states=True, input_ids=input_ids,attention_mask=attention_mask).hidden_states
+    embedding_repr = embedding_repr[self.hidden_layer]
+  attention_mask = attention_mask.unsqueeze(-1)
+  attention_mask = attention_mask.expand(-1, -1, embedding_repr.size(-1))
+  masked_embedding_repr = embedding_repr * attention_mask
+  sum_embedding_repr = masked_embedding_repr.sum(dim=1)
+  non_zero_count = attention_mask.sum(dim=1) 
+  mean_embedding_repr = sum_embedding_repr / non_zero_count
+  return mean_embedding_repr
+```