protein-information-system 3.1.2
Comprehensive Python Module for Protein Data Management: Designed for streamlined integration and processing of protein information from both UniProt and PDB. Equipped with features for concurrent data fetching, robust error handling, and database synchronization.
Requires Python: <4.0,>=3.10

[![PyPI - Version](https://img.shields.io/pypi/v/protein-information-system)](https://pypi.org/project/protein-information-system/)
[![Documentation Status](https://readthedocs.org/projects/protein-information-system/badge/?version=latest)](https://protein-information-system.readthedocs.io/en/latest/?badge=latest)
![Linting Status](https://github.com/CBBIO/protein-information-system/actions/workflows/test-lint.yml/badge.svg?branch=main)
[![codecov](https://codecov.io/gh/CBBIO/protein-information-system/branch/main/graph/badge.svg)](https://codecov.io/gh/CBBIO/protein-information-system)

# **Protein Information System (PIS)**

**Protein Information System (PIS)** is an integrated biological information system focused on extracting, processing, and managing protein-related data. PIS consolidates data from **UniProt**, **PDB**, and **GOA**, enabling the efficient retrieval and organization of protein sequences, structures, and functional annotations.

The primary goal of PIS is to provide a robust framework for large-scale protein data extraction, facilitating downstream functional analysis and annotation transfer. The system is designed for **high-performance computing (HPC) environments**, ensuring scalability and efficiency.


## 📈 **Current State of the Project**

### **FANTASIA: Functional Annotation Toolkit**


> 🧠 **FANTASIA** was built on top of the Protein Information System (PIS) as an advanced tool for **functional protein annotation** using embeddings generated by protein language models.
>
> [🔗 FANTASIA Repository](https://github.com/CBBIO/FANTASIA)
>
> The pipeline supports high-performance computing (HPC) environments and integrates tools such as ProtT5, ESM, and CD-HIT. These models can be extended or replaced with new variants **without modifying the core software structure**, simply by adding the new model to the PIS. This design enables scalable, modular, and reproducible GO term annotation from FASTA sequence files.


### **Protocol for Large-Scale Metamorphism and Multifunctionality Search**

> 🔍 In addition, a systematic protocol has been developed for the **large-scale identification of structural metamorphisms** and **protein multifunctionality**.
>
> [🔗 Metamorphic and multifunctionality Search Repository](https://github.com/CBBIO/metamorphic_multifunctional_search)
> 
> This protocol leverages the full capabilities of PIS to uncover non-obvious relationships between structure and function. **Structural metamorphisms** are detected by filtering large-scale structural alignments between proteins with high sequence identity, identifying divergent conformations. **Multifunctionality** is addressed through a semantic analysis of GO annotations, computing a functional distance metric to determine the two most divergent terms within each GO category per protein.

---

## **📡 Installing the BioData Lookup Table (Two Options)**

This guide shows two ways to load and use the **BioData** lookup table:

1. **Option A** - Manually download the PostgreSQL backup from **Zenodo** and restore it yourself (no PIS required).
2. **Option B** - Clone the **Protein Information System (PIS)** repository and let its helper script set everything up.

Both options end with the same result: a PostgreSQL database called `BioData` running with the `pgvector` extension enabled.

---

## 📚 Prerequisites

- A machine with:
        - Docker installed and running.
        - At least ~25-30 GB of free disk space (the backup itself is large).
- PostgreSQL client tools installed on your host:
        - `psql`, `createdb`, `dropdb`, `pg_restore`
        - Recommended: PostgreSQL 16+ client tools.
- Credentials used in this guide:
        - PostgreSQL user: `usuario`
        - PostgreSQL password: `clave`
        - Database name: `BioData`

> Adjust credentials if you use different ones.

---

## Option A - Manual Setup from Zenodo (without PIS)

### 1. Start the pgvector PostgreSQL container

```bash
docker run -d --name pgvectorsql \
    -e POSTGRES_USER=usuario \
    -e POSTGRES_PASSWORD=clave \
    -e POSTGRES_DB=BioData \
    -p 5432:5432 \
    pgvector/pgvector:pg16
```

This starts PostgreSQL with pgvector on `localhost:5432`.

---

### 2. Download the BioData backup from Zenodo

1. Open the Zenodo record in your browser, for example:
        - Final-layer table: `https://zenodo.org/records/17795871`
        - Early+final layers table: `https://zenodo.org/records/17793273`
2. In the **Files** section, locate the `.backup` file you want, e.g.:
        - `BioData_Dec25_esm2_prott5_prostt5_ankh3_large_esm3c_Layer0.backup`
        - or
        - `BioData_Dec25_esm2_prott5_prostt5_ankh3_large_esm3c_Layers_3Frist_3Last.backup`
3. Click **Download** and **save the file** to a known location, for example:

```bash
~/biodata_backups/BioData_Dec25_esm2_prott5_prostt5_ankh3_large_esm3c_Layers_3Frist_3Last.backup
```

> Do this via the browser to avoid Zenodo's cookie/redirect issues. The file should be multi-GB in size, not a few KB.

---

### 3. Drop and recreate the BioData database

On your host, using the PostgreSQL client tools (connecting to the Docker container):

```bash
export PGPASSWORD="clave"

# 1) Try to drop the database if it exists
dropdb -h localhost -U usuario BioData --if-exists

# 2) If there are still active connections, terminate them
psql -h localhost -U usuario -d postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'BioData' AND pid <> pg_backend_pid();"

dropdb -h localhost -U usuario BioData --if-exists

# 3) Final termination attempt (if needed) and drop
psql -h localhost -U usuario -d postgres \
    -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'BioData';"

sleep 2
dropdb -h localhost -U usuario BioData --if-exists

# 4) Recreate BioData
createdb -h localhost -U usuario BioData
```

---

### 4. Enable pgvector extension

```bash
psql -h localhost -U usuario -d BioData \
    -c "CREATE EXTENSION IF NOT EXISTS vector;"
```

---

### 5. Restore the BioData backup

```bash
export PGPASSWORD="clave"

pg_restore -h localhost -U usuario \
    -d BioData \
    ~/biodata_backups/BioData_Dec25_esm2_prott5_prostt5_ankh3_large_esm3c_Layers_3Frist_3Last.backup
```

If restore succeeds, you now have the BioData database ready to use.

---

### 6. Connecting to BioData

- Using `psql`:

```bash
PGPASSWORD="clave" psql -h localhost -U usuario -d BioData
```

- Typical connection URL for applications:

```
postgresql://usuario:clave@localhost:5432/BioData
```

Use this string in your tools, notebook, or pipeline that needs to query the lookup table.

---

## Option B - Using the PIS Repository and Helper Script

If you also want the **Protein Information System** (PIS) and its automation around the database, use this method.

### 1. Clone the repository

```bash
cd /path/where/you/want/the/repo
git clone https://github.com/CBBIO/protein-information-system.git
cd protein-information-system
```

---

### 2. Set the Zenodo URL in pis_launcher_script.sh

At the top of pis_launcher_script.sh, set:

```bash
ZENODO_URL="https://zenodo.org/records/17793273/files/BioData_Dec25_esm2_prott5_prostt5_ankh3_large_esm3c_Layers_3Frist_3Last.backup?download=1"
```

(or the URL of the specific `.backup` you want from the **Files** section.)

The script will:

- Derive the filename from this URL.
- Download to the configured backup folder if it does not exist.
- Reuse the local file on subsequent runs (no re-download).

---

### 3. Run the self-check with rebase from Zenodo

From the repository root:

```bash
bash pis_launcher_script.sh --rebase-from-zenodo
```

This script will:

1. Check that Docker is running.
2. Ensure the `pgvectorsql` container (PostgreSQL + pgvector) and `rabbitmq` container exist and are running.
3. Download the BioData backup from Zenodo (or reuse the existing file in the configured backup folder).
4. **Drop and recreate** the `BioData` database on `localhost:5432`.
5. Enable the `vector` extension.
6. Run `pg_restore` from the downloaded backup.

If the size check fails (file looks too small), it will stop and tell you to correct `ZENODO_URL` or manually download the backup into the configured backup folder.

> With `--rebase-from-zenodo`, the script focuses on the DB rebase and then exits, so you get a clean BioData database ready to use.

---

### Script Options

Common flags for `pis_launcher_script.sh`:

- `--rebase-from-zenodo`: Download (or reuse) the Zenodo backup and restore it.
- `--rebase-from-backup`: Restore from a local backup file.
- `--zenodo-url=...`: Override the Zenodo URL used for download.
- `--backup-folder=...`: Folder where backups are stored/loaded.
- `--backup-file-name=...`: Backup filename to use inside the backup folder.
- `--database-name=...`: Target database name (default: `BioData`).
- `--check-services` or `--check-services-only`: Only check Docker and container status without a restore.

---

### 4. Use the database

After the script completes successfully:

- Connect with `psql` as in Option A:

```bash
PGPASSWORD="clave" psql -h localhost -U usuario -d BioData
```

- Or point your applications to:

```
postgresql://usuario:clave@localhost:5432/BioData
```

PIS itself can then use this database for its embedding and lookup workflows.

---

If you want, I can also draft a short "Troubleshooting" section for Notion (e.g. `pg_restore` version issues, port conflicts on 5432, etc.).

---

## **Get started:**

To execute the full extraction process, install dependencies and run from project root:

```bash
pis
```

This command will trigger the complete workflow, starting from the initial data preprocessing stages and continuing through to the final data organization and storage.

## **Customizing the Workflow:**

You can customize the sequence of tasks executed by modifying `main.py` or adjusting the relevant parameters in the `config.yaml` file. This allows you to tailor the extraction process to meet specific research needs or to experiment with different data processing configurations.