MiGenPro 0.1.4
Microbial Genome Prospecting (MiGenPro) combines phenotype and genomic linked data. Migenpro serves as a framework for the generation of machine learning models that predict microbial traits from genome sequences. 
License: MIT  license
Requires Python: <3.16,>=3.11.4

![Coverage](https://migenpro-53cfa3.gitlab.io/coverage.svg)
![codequality](https://migenpro-53cfa3.gitlab.io/pylint.svg)
<img src="https://gitlab.com/wurssb/migenpro/-/raw/43d2f687701d09801a57505945a6fdbd9534697f/docs/MIT_logo.png?inline=false" alt="MIT Logo" width="50" height="20">

# MiGenPro - Microbial Genome Processing Toolkit
MiGenPro: A flexible linked data framework for phenotype-genotype prediction of microbial traits using  machine learning. 

## Functionalities
- Genome annotation based on taxonomy identifiers.
- Data formatting and cleaning for microbial genome datasets.
- Conversion of raw query data into structured feature and phenotype matrices.
- Advanced filtering options to remove low-frequency features or phenotypes.
- Parallel processing support for efficient handling of large datasets.
- Easy training and prediction with machine learning models on microbial characteristics.

![workflow_overview.svg](./data_visualisation/workflow_overview.svg)

## Quickstart

## Installation with pip in a special conda environment
```bash
conda create -n migenpro -c bioconda ;   
conda activate migenpro ;
pip install migenpro
```

Run the workflow from phenotype graph to phenotype prediction using the following command:
```bash
 migenpro --df --gq --ml --annotation  --train --predict  \
          --sapp_jar ./binaries/SAPP-2.0.jar  \
          --phenotype_query_file sparql_phenotype:demo_gram.sparql   \
          --phenotype_hdt_file ./data/bacdive.hdt.gz   \
          --genome_query_file sparql_genome:DomainCopyNumber.sparql   \
          --abs_frequency 1   --threads 20   \
          --sampling_type SMOTEN --output ./demo_output  \
          --genome_dir ./demo_output/genomes

```
The --param_grids flag can be used to optimise the parameters of the machine learning models.
An example json file is available at: `tests/resources/param_grid.json`

The individual steps:
1. Querying phenotype graphs
2. Annotating genomes
3. Querying the annotated genomes
4. Training machine learning models
5. Predicting phenotypes with existing models
6. Feature importance analysis
7. Summarising the results


#### 1. Querying phenotype graphs
```bash  
migenpro --df \
    --phenotype_query_file sparql_phenotype:demo_gram.sparql \
    --phenotype_hdt_file ./data/bacdive.hdt.gz \
    --abs_frequency 1 \
    --sapp_jar binaries/SAPP-2.0.jar \
    --output ./demo_output/
```

#### 2. Annotating genomes
Genome annotation is done by default using the workflow: https://workflowhub.eu/workflows/1170/
this can be changed using the `--cwl_file` flag with a workflow of your choice granted that it takes a fasta file as input.
You can speed up this process with the `--threads` flag. 
```bash
migenpro --annotation \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar ./binaries/SAPP-2.0.jar \
    --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
    --output ./demo_output/ \
    --genome_dir ./demo_output/genomes
```

#### 3. Querying the annotated genomes

```bash
migenpro --gq \
    --genome_query_file sparql_genome:DomainCopyNumber.sparql \
    --sapp_jar ./binaries/SAPP-2.0.jar \
    --output ./demo_output/ \
    --genome_dir ./demo_output/genomes
```

#### 4. Training machine learning models 
We will now use the default parameters for training the models.
If you wish to optimise the parameters you can do this using the `--param_grids` flag.
Here we set an absolute frequency of 1 as a per phenotype minimum frequency for the demo. 
```bash
migenpro --ml --train --predict \
      --feature_matrix ./demo_output/feature_matrix.tsv \
      --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
      --output ./demo_output/ \
      --abs_frequency 1 \
      --species_frequency 1
```

#### 5. Predicting phenotypes with existing models
You can do this through the docker container or from the source code. 
1. You will need to obtain a protein domain matrix of the desired genomes you can do this using the java code. 
2. For ease of use we will use the python scripts that were made with the following command. The default output directory is "output/mloutput" if desired you can change this using the --output [output\_directory\_location]

```bash
migenpro --ml --predict \
      --feature_matrix ./demo_output/feature_matrix.tsv \
      --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
      --output ./demo_output/ \
      --abs_frequency 1 \
      --model ./demo_output/protein_domains/DecisionTreeClassifier/DecisionTreeClassifier_protein_domains.pkl
```


#### 6. Feature importance analysis
```bash
migenpro --fi \
        --models  ./demo_output/protein_domains/ \
        --feature_matrix ./demo_output/feature_matrix.tsv \
        --phenotype_matrix ./demo_output/phenotype_matrix.tsv \
        --output ./demo_output/
```

#### 7. Summarising the results

Wait for the script to finish and retrieve the results of your prediction from the output directory. 
There the predictions are given in the following format: 

| Genome | Phenotype | Prediction | Confidence |
|--------|-----------|------------|------------|
| GCA123 | Temperature | mesophilic | 0.96 |

```bash
migenpro --summarise \
        --output ./demo_output/
```

## Contributing 
### Pull the git repo:
```bash
git pull git@gitlab.com:pig-paradigm/migenpro.git
cd migenpro
```

### Installing the needed dependencies. 
A pip requirements.txt file is located in the installation directory which you can install using the following command.

```bash 
conda create -n migenpro python=3.12.5 --file installation/requirements.txt
```

## Recreating the results from the study
The files needed to recreate our results are located on https://zenodo.org/records/16995284. 
Apply the steps from this tutorial namely the `/data_visualisation/construct_all_graphs_from_summaries.ipynb` to recreate the graphs. 

## Maintainers
Jasper J. Koehorst (@jjkoehorst) and Mike Loomans (@MikeLoomans1999)