spatialx-connector 0.2.21
BioTuring SpatialX Connector

### BioTuring SpatialX Connector

### Version Update: April 2025

This update introduces several enhancements to data management and analysis capabilities:

1.  **Enhanced Cell Filtering:** Users can now refine cell selection by adjusting filtering parameters based on both the number of genes and transcripts. [Check 1.7.5. Parsing Submission Information for Multiple Samples](#ParsingSubmissionInformationforMultipleSamples:)

2.  **Modular Study Creation:** Studies can be created and data elements submitted independently, providing greater flexibility in data organization. [[Check 1.7.2. Submitting a New Study with a Single Sample and Data](#SubmittingaNewStudywithaSingleSampleandData:) and [Check 1.7.6. Data Details and Element Management](#DataDetailsandElementManagement)]

3.  **Improved Study Information Access:** Enhanced features for accessing and reviewing comprehensive study information. [Check 1.7.6. Data Details and Element Management](#DataDetailsandElementManagement)

4.  **Metadata and Embeddings Upload Functionality:** Users can now upload new metadata files, enriching their datasets with additional contextual information. [Check 1.7.6. Data Details and Element Management](#DataDetailsandElementManagement)

5.  **Flexible Data Structure Customization:** Users gain increased control over the data structure of submissions, allowing for tailored data organization. [Check 1.7.7. Data Details and Element Management](#CustomSubmission)

6.  **Expanded Visualization Options:** The analysis section now includes a wider range of visualization tools for improved data exploration. [Check 1.8.5. Interact with elements](#SelectElements:)

**IMPORTANT:** To utilize all new functionalities, please re-execute the installation step: [1.1. Installation SpatialX Connector](#InstallationSpatialXConnector).

###  1.1. <a name='InstallationSpatialXConnector'></a>Installation SpatialX Connector


```python
!pip uninstall spatialx_connector -y -q
!pip install spatialx_connector -U -q
```


###  1.2. <a name='Importtherelatedpackages'></a>Import the related packages


```python
import warnings
warnings.filterwarnings("ignore")
```


```python
import os
import pandas as pd
from matplotlib import pyplot as plt

import spatialx_connector
from spatialx_connector import SpatialXConnector
from spatialx_connector import Technologies
from spatialx_connector import DefaultGroup
from spatialx_connector import Species
from spatialx_connector import SpatialAttrs
from spatialx_connector import ConnectorKeys
from spatialx_connector import SubmissionElementKeys
from spatialx_connector import ImagesSubmission
from spatialx_connector import SegmentationSubmission
from spatialx_connector import ExpressionSubmission
```

###  1.3. <a name='DomainandToken'></a>Domain and Token
To obtain your domain URL and personal token, navigate to "SpatialX SDK" in the left panel of your SpatialX interface. Then, enter the information in the fields below. For example:

DOMAIN = "https://example.bioturing.com/spatialx/"

TOKEN = "000000000000000000000000NM"


```python
DOMAIN = ""
TOKEN = ""
```

###  1.4. <a name='ExploreYourAccount'></a>Explore Your Account
With your domain and token added, you can now connect to your SpatialX account and workspace and explore your account details.

####  1.4.1. <a name='UsersInformation:'></a>User's Information:


```python
connector = SpatialXConnector(DOMAIN, TOKEN)
spatialx_connector.format_print(connector.info)
```

    {
        email: techsupport@bioturing.com
        sub_dir: 649e75177eb042a6638f319e35e4959f
        name: ub-techsu-332f83917f3b7d3
        app_base_url: ...
        routing_table: {
            services: {}
        }
    }


####  1.4.2. <a name='Groups:'></a>Groups:


```python
spatialx_connector.format_print(connector.groups)
```

    {
        Personal workspace: 40d094152b98ba25151936c18becb267
        All members: GLOBAL_GROUP
        My submitted data: MY_SUBMITTED_GROUP
    }


###  1.5. <a name='ListofStorages'></a>List of Storages
If you have configured your cloud storages or would like to check your data list in SpatialX, the list of functions below can help you to get the information:

####  1.5.1. <a name='AWSbuckets'></a> AWS buckets


```python
spatialx_connector.format_print(connector.s3)
```

    {
        bioturingpublic: <USER_PATH>/public_cloud/bioturingpublic
    }


####  1.5.2. <a name='PersonalandSharedFolders'></a> Personal and Shared Folders


```python
spatialx_connector.format_print(connector.folders)
```

    {
        Upload: <USER_PATH>/upload
        Submitted: <USER_PATH>/study
        Converted: <USER_PATH>/converted
    }


####  1.5.3. <a name='BrowsingStorage'></a> Browsing Storage


```python
connector.listdir(connector.s3["bioturingpublic"])
```




    ['SpatialX_datasets', 'mount']




```python
connector.listdir(os.path.join(connector.s3["bioturingpublic"], "SpatialX_datasets"))
```




    ['COSMX_VER1', 'Human_Colon_Cancer_P2']




```python
connector.listdir(os.path.join(connector.s3["bioturingpublic"], "SpatialX_datasets/COSMX_VER1"))
```




    ['Lung6', 'Lung9_Rep1', 'Lung9_Rep2']



###  1.6. <a name='Uploadingfiles'></a>Uploading files
To upload files to your personal folders within your SpatialX account, execute the code below. Be sure to replace the placeholder `file_path` with the complete path to the file you wish to upload.

####  1.6.1. <a name='CreateData:'></a> Create data for testing upload:


```python
os.makedirs("./upload_folder", exist_ok=True)
with open("./upload_folder/upload_file", "w") as f:
    f.write("This is file for upload")
with open("./upload_folder/upload_big_file", "w") as f:
    f.write("This is big file for upload")
```

####  1.6.2. <a name='Upload:'></a> Upload files and folders:


```python
uploading_results = connector.upload_file(file_path="./upload_folder/upload_file")
spatialx_connector.format_print(uploading_results)
```

    {
        folder_name:
        unique_id: 01JRVQJQZMGJA377FPFZHZZZSC
        path: <USER_PATH>/upload/upload_file
        url: <USER_URL>/upload/upload_file
    }



```python
uploading_results = connector.upload_big_file(file_path="./upload_folder/upload_big_file", debug_mode=True)
spatialx_connector.format_print(uploading_results)
```

    {
        folder_name:
        unique_id: 01JRVQJV5BKGF3MDSBY3DH57VH
        path: <USER_PATH>/upload/upload_big_file
        url: <USER_URL>/upload/upload_big_file
    }



```python
uploading_results = connector.upload_folder(dir_path="./upload_folder", debug_mode=True)
spatialx_connector.format_print(uploading_results)
```

    <USER_PATH>/upload/upload_folder.v0


###  1.7. <a name='Submission'></a>Submission

####  1.7.1. <a name='ParsingDataInformationforSubmission:'></a> Parsing Data Information for Submission:
* **`data_name`:** Name of the dataset.
* **`technology`:** Technology used for the dataset.
* **`data_path`:** Path to the dataset.


```python
Visium_V2_Human_Colon_Cancer_P2_submission_information = connector.parse_data_information(
    data_name="Visium_V2_Human_Colon_Cancer_P2",
    technology=Technologies.VISIUM.value,
    data_path=os.path.join(
        connector.s3["bioturingpublic"],
        "SpatialX_datasets/Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2"
    )
)
spatialx_connector.format_print(Visium_V2_Human_Colon_Cancer_P2_submission_information)
```

    [
        {
            name: Visium_V2_Human_Colon_Cancer_P2
            submission_type: SUBMIT_SPATIAL_BULK
            technology: VISIUM
            files: [
                {
                    key: images
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2_tissue_image.btf
                }
                {
                    key: matrix
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2_raw_feature_bc_matrix.h5
                }
                {
                    key: tissue_positions
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2/spatial/tissue_positions.csv
                }
                {
                    key: scalefactors
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Visium_V2_Human_Colon_Cancer_P2/spatial/scalefactors_json.json
                }
            ]
            folders: []
            args: []
            kwargs: [
                {
                    key: min_genes
                    value: 1
                }
                {
                    key: min_counts
                    value: 1
                }
                {
                    key: mito_controls_percentage
                    value: 0.25
                }
            ]
            identities: []
        }
    ]



```python
Xenium_V1_Human_Colon_Cancer_P2_submission_information = connector.parse_data_information(
    data_name="Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE",
    technology=Technologies.XENIUM.value,
    data_path=os.path.join(
        connector.s3["bioturingpublic"],
        "SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE"
    )
)
spatialx_connector.format_print(Xenium_V1_Human_Colon_Cancer_P2_submission_information)
```

    [
        {
            name: Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE
            submission_type: SUBMIT_SPATIAL_TRANSCRIPTOMICS
            technology: XENIUM
            files: [
                {
                    key: experiment
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE/experiment.xenium
                }
                {
                    key: images
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE/morphology.ome.tif
                }
                {
                    key: alignment
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE_he_imagealignment.csv
                }
                {
                    key: segmentation
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE/cell_boundaries.parquet
                }
                {
                    key: transcripts
                    value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/Human_Colon_Cancer_P2/Xenium_V1_Human_Colon_Cancer_P2_CRC_Add_on_FFPE/transcripts.parquet
                }
            ]
            folders: []
            args: []
            kwargs: [
                {
                    key: min_genes
                    value: 1
                }
                {
                    key: min_counts
                    value: 1
                }
                {
                    key: mito_controls_percentage
                    value: 0.25
                }
            ]
            identities: []
        }
    ]


####  1.7.2. <a name='SubmittingaNewStudywithaSingleSampleandData:'></a> Submitting a New Study with a Single Sample and Data:
* **`group`:** User's group.
* **`species`:** Species of the dataset.
* **`title`:** Title of the new study.
* **`sample_name`:** Name of the new sample.
* **`sample_data`:** Data information, obtained from `connector.parse_data_information` or a combination of its results.


```python
submission_results = connector.submit(
    group=DefaultGroup.PERSONAL_WORKSPACE.value,
    species=Species.HUMAN.value,
    title="Human Colon Cancer - 10xgenomics",
    sample_name="Human_Colon_Cancer_P2",
    sample_data=Xenium_V1_Human_Colon_Cancer_P2_submission_information + Visium_V2_Human_Colon_Cancer_P2_submission_information,
)
spatialx_connector.format_print(submission_results)
```

####  1.7.3. <a name='AddingaNewSampletoanExistingStudy:'></a> Adding a New Sample to an Existing Study:
* **`study_id`:** ID of the study to which the sample is added.
* **`name`:** Name of the new sample.
* **`sample_data`:** Data information, obtained from `connector.parse_data_information` or a combination of its results.


```python
adding_sample_results = connector.add_sample(
    study_id=submission_results[ConnectorKeys.STUDY_ID.value],
    sample_name="Human_Colon_Cancer_P2 - New Sample",
    sample_data=Visium_V2_Human_Colon_Cancer_P2_submission_information,
)
spatialx_connector.format_print(adding_sample_results)
```

####  1.7.4. <a name='AddingNewDatatoanExistingSample:'></a> Adding New Data to an Existing Sample:

* **`study_id`:** ID of the study containing the sample.
* **`sample_id`:** ID of the existing sample.
* **`sample_data`:** Data information, obtained from `connector.parse_data_information` or a combination of its results.


```python
adding_sample_data_results = connector.add_sample_data(
    study_id=adding_sample_results[ConnectorKeys.STUDY_ID.value],
    sample_id=adding_sample_results[ConnectorKeys.SAMPLE_ID.value],
    sample_data=Xenium_V1_Human_Colon_Cancer_P2_submission_information,
)
spatialx_connector.format_print(adding_sample_data_results)
```

####  1.7.5. <a name='ParsingSubmissionInformationforMultipleSamples:'></a> Parsing Submission Information for Multiple Samples:

* **`technology`:** Technology used for all samples (supports a single technology).
* **`data_path`:** Path to the directory containing multiple dataset subfolders (each subfolder represents a dataset).
* **`sample_name_mapping`:** Mapping of subfolder names to sample names.
* **`min_genes`:** Filtering cells by number of genes.
* **`min_counts`:** Filtering cells by number of transcripts.


```python
multiple_cosmx_samples_submission_information = connector.parse_multiple_samples_information(
    technology=Technologies.COSMX_VER1.value,
    data_path=os.path.join(connector.s3["bioturingpublic"], "SpatialX_datasets/COSMX_VER1"),
    sample_name_mapping={
        "Lung6": "Human Lung Cancer - Sample 6",
        "Lung9_Rep1": "Human Lung Cancer - Sample 9 Rep 1",
        "Lung9_Rep2": "Human Lung Cancer - Sample 9 Rep 2",
    },
    data_name_mapping={
        "Lung6": "Sample 6",
        "Lung9_Rep1": "Sample 9 Rep 1",
        "Lung9_Rep2": "Sample 9 Rep 2",
    },
    min_genes=10,
    min_counts=10,
)
spatialx_connector.format_print(multiple_cosmx_samples_submission_information)
```

    [
        {
            sample_name: Human Lung Cancer - Sample 6
            data: [
                {
                    name: Sample 6
                    submission_type: SUBMIT_SPATIAL_TRANSCRIPTOMICS
                    technology: COSMX_VER1
                    files: [
                        {
                            key: fov_positions
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung6/Lung6-Flat_files_and_images/Lung6_fov_positions_file.csv
                        }
                        {
                            key: transcripts
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung6/Lung6-Flat_files_and_images/Lung6_tx_file.csv
                        }
                    ]
                    folders: [
                        {
                            key: images
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung6/Lung6-RawMorphologyImages
                        }
                        {
                            key: segmentation
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung6/Lung6-Flat_files_and_images/CellLabels
                        }
                    ]
                    args: [
                        {
                            key: mpp
                            value: 0.18
                        }
                    ]
                    kwargs: [
                        {
                            key: min_genes
                            value: 10
                        }
                        {
                            key: min_counts
                            value: 10
                        }
                        {
                            key: mito_controls_percentage
                            value: 0.25
                        }
                    ]
                    identities: []
                }
            ]
        }
        {
            sample_name: Human Lung Cancer - Sample 9 Rep 1
            data: [
                {
                    name: Sample 9 Rep 1
                    submission_type: SUBMIT_SPATIAL_TRANSCRIPTOMICS
                    technology: COSMX_VER1
                    files: [
                        {
                            key: fov_positions
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep1/Lung9_Rep1-Flat_files_and_images/Lung9_Rep1_fov_positions_file.csv
                        }
                        {
                            key: transcripts
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep1/Lung9_Rep1-Flat_files_and_images/Lung9_Rep1_tx_file.csv
                        }
                    ]
                    folders: [
                        {
                            key: images
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep1/Lung9_Rep1-RawMorphologyImages
                        }
                        {
                            key: segmentation
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep1/Lung9_Rep1-Flat_files_and_images/CellLabels
                        }
                    ]
                    args: [
                        {
                            key: mpp
                            value: 0.18
                        }
                    ]
                    kwargs: [
                        {
                            key: min_genes
                            value: 10
                        }
                        {
                            key: min_counts
                            value: 10
                        }
                        {
                            key: mito_controls_percentage
                            value: 0.25
                        }
                    ]
                    identities: []
                }
            ]
        }
        {
            sample_name: Human Lung Cancer - Sample 9 Rep 2
            data: [
                {
                    name: Sample 9 Rep 2
                    submission_type: SUBMIT_SPATIAL_TRANSCRIPTOMICS
                    technology: COSMX_VER1
                    files: [
                        {
                            key: fov_positions
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep2/Lung9_Rep2-Flat_files_and_images/Lung9_Rep2_fov_positions_file.csv
                        }
                        {
                            key: transcripts
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep2/Lung9_Rep2-Flat_files_and_images/Lung9_Rep2_tx_file.csv
                        }
                    ]
                    folders: [
                        {
                            key: images
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep2/Lung9_Rep2-RawMorphologyImages
                        }
                        {
                            key: segmentation
                            value: <USER_PATH>/public_cloud/bioturingpublic/SpatialX_datasets/COSMX_VER1/Lung9_Rep2/Lung9_Rep2-Flat_files_and_images/CellLabels
                        }
                    ]
                    args: [
                        {
                            key: mpp
                            value: 0.18
                        }
                    ]
                    kwargs: [
                        {
                            key: min_genes
                            value: 10
                        }
                        {
                            key: min_counts
                            value: 10
                        }
                        {
                            key: mito_controls_percentage
                            value: 0.25
                        }
                    ]
                    identities: []
                }
            ]
        }
    ]


```python
multiple_samples_submission_results = connector.submit_multiple_samples(
    group=DefaultGroup.PERSONAL_WORKSPACE.value,
    species=Species.HUMAN.value,
    title="Multiple Human Lung Cancer - CosMX Ver1",
    sample_data=multiple_cosmx_samples_submission_information,
)
spatialx_connector.format_print(multiple_samples_submission_results)
```

####  1.7.6. <a name='DataDetailsandElementManagement'></a> Data Details and Element Management

This section provides tools for accessing and modifying data within existing studies. Users can perform the following actions:

* **Retrieve Data:**
    * Fetch data identifiers (IDs).
    * List existing elements associated with a dataset.
    * Download recently submitted AnnData objects.
* **Append Data Elements:**
    * Add individual data components to a dataset, including:
        * Protein images.
        * Segmentation masks.
        * Expression matrices.
        * Metadata.
        * Embeddings.

- **Submit Protein Images**: Submit protein images as a study.


```python
human_pancreas_codex_information = connector.parse_data_information(
    data_name="human_pancreas_codex",
    technology=Technologies.PROTEIN_QPTIFF.value,
    data_path=os.path.join(connector.s3["bioturingpublic"], "mount/examples/spatialx/human_pancreas_codex"),
    args={SubmissionElementKeys.MPP.value: 1},
)
spatialx_connector.format_print(human_pancreas_codex_information)
```

    [
        {
            name: human_pancreas_codex
            submission_type: SUBMIT_SPATIAL_PROTEOMICS
            technology: PROTEIN_QPTIFF
            files: [
                {
                    key: protein_images
                    value: <USER_PATH>/public_cloud/bioturingpublic/mount/examples/spatialx/human_pancreas_codex/human_pancreas_codex.qptiff
                }
            ]
            folders: []
            args: [
                {
                    key: mpp
                    value: None
                }
                {
                    key: mpp
                    value: 1
                }
            ]
            kwargs: [
                {
                    key: min_genes
                    value: 1
                }
                {
                    key: min_counts
                    value: 1
                }
                {
                    key: mito_controls_percentage
                    value: 0.25
                }
            ]
            identities: []
        }
    ]



```python
proteomics_submission_results = connector.submit(
    group=DefaultGroup.PERSONAL_WORKSPACE.value,
    species=Species.HUMAN.value,
    title="Human Pancreas CODEX",
    sample_name="Human Pancreas CODEX",
    sample_data=human_pancreas_codex_information,
    tracking=True,
)
spatialx_connector.format_print(proteomics_submission_results)
```

    [2025-04-15 03:24:41] [3992384] [INFO] SUBMISSION JOB IS WAITING AT ID 15.
    [2025-04-15 03:24:41] [3991797] [INFO] RUN SUBMISSION JOB ID 15: {
        study_path: <USER_PATH>/study/ST-01JRVQMFQVB2JYPM27C331SXKB
        sample_id: SP-01JRVQMG6SD4EGW2VNYJ6RVP0E
        submit_id: SB-01JRVQMG6SD4EGW2VNYHDJ6GRG
        data_id: DA-01JRVQMG70A1G37MZBB6E538C8
        submission_name: human_pancreas_codex
        submission_info: {
            submission_type: SUBMIT_SPATIAL_PROTEOMICS
            technology: PROTEIN_QPTIFF
            files: [
                {
                    key: protein_images
                    value: <USER_PATH>/public_cloud/bioturingpublic/mount/examples/spatialx/human_pancreas_codex/human_pancreas_codex.qptiff
                }
            ]
            folders: []
            args: [
                {
                    key: mpp
                    value: None
                }
                {
                    key: mpp
                    value: 1
                }
            ]
            kwargs: [
                {
                    key: min_genes
                    value: 1
                }
                {
                    key: min_counts
                    value: 1
                }
                {
                    key: mito_controls_percentage
                    value: 0.25
                }
            ]
            identities: []
        }
    }.
    [2025-04-15 03:24:41] [3991797] [INFO] SUBMIT: protein_images
    [2025-04-15 03:24:41] [3991797] [INFO] Fail to extracting channel information of OME-Tiff: 'NoneType' object has no attribute 'encode'
    [2025-04-15 03:24:41] [3991797] [INFO] Write protein_images.
    [2025-04-15 03:24:41] [3991797] [INFO] Process channel ACTA2
    [2025-04-15 03:24:41] [3991797] [INFO] Write image channel
    [2025-04-15 03:25:05] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:25:05] [3991797] [INFO] Process channel BETA-CATENIN
    [2025-04-15 03:25:05] [3991797] [INFO] Write image channel
    [2025-04-15 03:25:24] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:25:24] [3991797] [INFO] Process channel CGC
    [2025-04-15 03:25:24] [3991797] [INFO] Write image channel
    [2025-04-15 03:25:43] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:25:43] [3991797] [INFO] Process channel CHGA
    [2025-04-15 03:25:43] [3991797] [INFO] Write image channel
    [2025-04-15 03:26:04] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:26:04] [3991797] [INFO] Process channel CPEP
    [2025-04-15 03:26:04] [3991797] [INFO] Write image channel
    [2025-04-15 03:26:22] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:26:22] [3991797] [INFO] Process channel E-CADHERIN
    [2025-04-15 03:26:22] [3991797] [INFO] Write image channel
    [2025-04-15 03:26:40] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:26:40] [3991797] [INFO] Process channel HOECHST
    [2025-04-15 03:26:40] [3991797] [INFO] Write image channel
    [2025-04-15 03:27:02] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:27:02] [3991797] [INFO] Process channel IAPP
    [2025-04-15 03:27:02] [3991797] [INFO] Write image channel
    [2025-04-15 03:27:21] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:27:21] [3991797] [INFO] Process channel KRT19
    [2025-04-15 03:27:21] [3991797] [INFO] Write image channel
    [2025-04-15 03:27:37] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:27:37] [3991797] [INFO] Process channel PECAM-1
    [2025-04-15 03:27:37] [3991797] [INFO] Write image channel
    [2025-04-15 03:27:59] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:27:59] [3991797] [INFO] Process channel SST
    [2025-04-15 03:27:59] [3991797] [INFO] Write image channel
    [2025-04-15 03:28:17] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:28:17] [3991797] [INFO] Process channel VIM
    [2025-04-15 03:28:17] [3991797] [INFO] Write image channel
    [2025-04-15 03:28:39] [3991797] [INFO]   - progress: 100%
    [2025-04-15 03:28:39] [3991797] [INFO] SUBMISSION DONE: {
        protein_images: {
            59656ac1f77c445286ac2b4b3bbef5d2: human_pancreas_codex_protein_images
        }
    }.
    [2025-04-15 03:28:39] [3991797] [INFO] DONE.
    {
        study_id: ST-01JRVQMFQVB2JYPM27C331SXKB
        sample_id: SP-01JRVQMG6SD4EGW2VNYJ6RVP0E
        sample_data: [
            {
                data_id: DA-01JRVQMG70A1G37MZBB6E538C8
                submit_id: SB-01JRVQMG6SD4EGW2VNYHDJ6GRG
                submit_name: human_pancreas_codex
            }
        ]
        submit_id: SB-01JRVQMG6SD4EGW2VNYHDJ6GRG
        job_id: 15
        err_message:
    }


- **Retrieving Data ID**: Obtain the `data_id` for adding extended elements and running analyses.


```python
STUDY_ID = proteomics_submission_results[ConnectorKeys.STUDY_ID.value]
SAMPLE_ID = proteomics_submission_results[ConnectorKeys.SAMPLE_ID.value]
DATA_ID = proteomics_submission_results[ConnectorKeys.SAMPLE_DATA.value][0][ConnectorKeys.DATA_ID.value]
```

- **Adding New Segmentation:** Add a new segmentation layer to the dataset.


```python
add_segmentation_result = connector.add_sample_data_element(
    title="Proteomics Segmentation",
    data_id=DATA_ID,
    adding_types=[SegmentationSubmission.PARQUET.value],
    paths={
        SubmissionElementKeys.SEGMENTATION.value: os.path.join(
            connector.s3["bioturingpublic"],
            "mount/examples/spatialx/human_pancreas_codex/human_pancreas_segmentation.parquet",
        )
    },
    tracking=True,
)
spatialx_connector.format_print(add_segmentation_result)
```

    [2025-04-15 03:28:39] [3992386] [INFO] SUBMISSION JOB IS WAITING AT ID 16.
    [2025-04-15 03:28:39] [3991788] [INFO] RUN SUBMISSION JOB ID 16: {
        study_path: <USER_PATH>/study/ST-01JRVQMFQVB2JYPM27C331SXKB
        sample_id: SP-01JRVQMG6SD4EGW2VNYJ6RVP0E
        submit_id: SB-01JRVQVSE1WBZN6FC94MZPY7KD
        data_id: DA-01JRVQMG70A1G37MZBB6E538C8
        submission_name: Proteomics Segmentation
        submission_info: {
            submission_type: SUBMIT_SPATIAL_PROTEOMICS
            technology: PROTEIN_QPTIFF
            files: [
                {
                    key: segmentation
                    value: <USER_PATH>/public_cloud/bioturingpublic/mount/examples/spatialx/human_pancreas_codex/human_pancreas_segmentation.parquet
                }
            ]
            folders: []
            args: [
                {
                    key: ignore_technology_elements
                    value: True
                }
            ]
            kwargs: []
            identities: [
                SEGMENTATION_PARQUET
            ]
        }
    }.
    [2025-04-15 03:28:40] [3991788] [INFO] SUBMIT: segmentation
    [2025-04-15 03:28:41] [3991788] [INFO] Write segmentation.
    [2025-04-15 03:28:49] [3991788] [INFO] SUBMIT: cell_centers
    [2025-04-15 03:28:52] [3991788] [INFO] SUBMISSION DONE: {
        segmentation: {
            a84b2978677c49c18395825a16ce9738: Proteomics Segmentation_segmentation
        }
        cell_centers: {
            589255595512465c9b49d882385b6c0a: Proteomics Segmentation_cell_centers
        }
    }.
    [2025-04-15 03:28:52] [3991788] [INFO] DONE.
    {
        study_id: ST-01JRVQMFQVB2JYPM27C331SXKB
        sample_id: SP-01JRVQMG6SD4EGW2VNYJ6RVP0E
        sample_data: None
        submit_id: SB-01JRVQVSE1WBZN6FC94MZPY7KD
        job_id: 16
        err_message:
    }


- **Retrieving Existing Elements:** List the existing elements associated with the dataset.


```python
sample_data_elements = connector.get_sample_data_elements(DATA_ID)
spatialx_connector.format_print(sample_data_elements)
```

    {
        protein_images: [
            human_pancreas_codex_protein_images
        ]
        cell_centers: [
            Proteomics Segmentation_cell_centers
        ]
        segmentation: [
            Proteomics Segmentation_segmentation
        ]
    }


- **Adding New Expression Matrix:** Add a new expression matrix to the dataset.


```python
add_expression_result = connector.add_sample_data_element(
    title="Proteomics Expression",
    data_id=DATA_ID,
    adding_types=[ExpressionSubmission.IMPORT_ANNDATA.value],
    paths={
        SubmissionElementKeys.EXPRESSION.value: os.path.join(
            connector.s3["bioturingpublic"],
            "mount/examples/spatialx/human_pancreas_codex/human_pancreas_protein.h5ad",
        ),
    },
    args={
        SubmissionElementKeys.SPATIAL_ID.value: sample_data_elements[SubmissionElementKeys.CELL_CENTERS.value][0],
    },
    tracking=True,
)
spatialx_connector.format_print(add_expression_result)
```

    [2025-04-15 03:28:52] [3992386] [INFO] SUBMISSION JOB IS WAITING AT ID 17.
    [2025-04-15 03:28:52] [3991797] [INFO] RUN SUBMISSION JOB ID 17: {
        study_path: <USER_PATH>/study/ST-01JRVQMFQVB2JYPM27C331SXKB
        sample_id: SP-01JRVQMG6SD4EGW2VNYJ6RVP0E
        submit_id: SB-01JRVQW63HR24Q8NBCPAWFGWPZ
        data_id: DA-01JRVQMG70A1G37MZBB6E538C8
        submission_name: Proteomics Expression
        submission_info: {
            submission_type: SUBMIT_SPATIAL_PROTEOMICS
            technology: PROTEIN_QPTIFF
            files: [
                {
                    key: annotated_data
                    value: <USER_PATH>/public_cloud/bioturingpublic/mount/examples/spatialx/human_pancreas_codex/human_pancreas_protein.h5ad
                }
            ]
            folders: []
            args: [
                {
                    key: spatial_id
                    value: Proteomics Segmentation_cell_centers
                }
                {
                    key: ignore_technology_elements
                    value: True
                }
            ]
            kwargs: []
            identities: [
                IMPORT_ANNDATA
            ]
        }
    }.
    [2025-04-15 03:28:52] [3991797] [INFO] SUBMIT: annotated_data


    [2025-04-15 03:28:53] [3991797] [INFO] Start importing
    [2025-04-15 03:28:53] [3991797] [INFO] Merging expression
    [2025-04-15 03:28:53] [3991797] [INFO] Processing obsm
    [2025-04-15 03:28:53] [3991797] [INFO] Processing obs
    [2025-04-15 03:28:54] [3991797] [INFO] Add to table
    [2025-04-15 03:28:54] [3991797] [INFO] Run scVI.
    [2025-04-15 03:28:55] [3991797] [INFO] Parameters: {
        batch_key: None
        n_latents: 20
        encode_covariates: True
        n_layers: 2
        train_size: 0.9
        dropout_rate: 0.2
        n_top_genes: 10
    }.
    [2025-04-15 03:28:55] [3991797] [INFO] Parameters: {
        batch_key: None
        n_latents: 20
        encode_covariates: True
        n_layers: 2
        train_size: 0.9
        dropout_rate: 0.2
        n_top_genes: 10
    }.
    [2025-04-15 03:28:56] [3991797] [INFO] Run highly_variable_genes.
    [2025-04-15 03:28:56] [3991797] [INFO] Parameters: {
        n_top_genes: 10
        flavor: seurat
        normalize_method: raw
    }.
    [2025-04-15 03:28:59] [3991797] [INFO] Run highly_variable_genes successfully.
    [2025-04-15 03:29:00] [3991797] [INFO] Training scVI...
    [2025-04-15 03:29:00] [3991797] [WARNING] /home/nhat/BioTuring/spatialx/runtimes/pyapps/lib/python3.10/site-packages/scvi/data/fields/_base_field.py:64: UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
      self.validate_field(adata)

    [2025-04-15 03:29:00] [3991797] [WARNING] /home/nhat/BioTuring/spatialx/runtimes/pyapps/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=47` in the `DataLoader` to improve performance.

    [2025-04-15 03:29:00] [3991797] [WARNING] /home/nhat/BioTuring/spatialx/runtimes/pyapps/lib/python3.10/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=47` in the `DataLoader` to improve performance.

    [2025-04-15 03:29:51] [3991797] [INFO] Inferring scVI...
    [2025-04-15 03:29:52] [3991797] [INFO] Run scVI successfully.
    [2025-04-15 03:29:52] [3991797] [INFO] Run neighbors_graph.
    [2025-04-15 03:29:52] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 90
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
    }.
    [2025-04-15 03:29:53] [3991797] [INFO] Run nearest_neighbors.
    [2025-04-15 03:29:53] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 90
    }.
    [2025-04-15 03:29:53] [3991797] [INFO] Run nearest_neighbors successfully.
    [2025-04-15 03:29:54] [3991797] [INFO] Run neighbors_graph successfully.
    [2025-04-15 03:29:54] [3991797] [INFO] Run t-SNE.
    [2025-04-15 03:29:54] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 90
        perplexity: 30
        init: pca
    }.
    [2025-04-15 03:29:55] [3991797] [INFO] Run nearest_neighbors.
    [2025-04-15 03:29:55] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 90
    }.
    [2025-04-15 03:29:55] [3991797] [INFO] Found analytic results for nearest_neighbors with same parameters.
    [2025-04-15 03:29:57] [3991797] [INFO] Run t-SNE successfully.
    [2025-04-15 03:29:57] [3991797] [INFO] Run UMAP.
    [2025-04-15 03:29:57] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 15
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
        init: pca
        deterministic: False
    }.
    [2025-04-15 03:29:57] [3991797] [INFO] Run neighbors_graph.
    [2025-04-15 03:29:57] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 15
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
    }.
    [2025-04-15 03:29:58] [3991797] [INFO] Run nearest_neighbors.
    [2025-04-15 03:29:58] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 15
    }.
    [2025-04-15 03:29:58] [3991797] [INFO] Run nearest_neighbors successfully.
    [2025-04-15 03:29:58] [3991797] [INFO] Run neighbors_graph successfully.
    [2025-04-15 03:29:59] [3991797] [INFO] Run UMAP successfully.
    [2025-04-15 03:29:59] [3991797] [INFO] Run Louvain.
    [2025-04-15 03:29:59] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        resolution: 0.1
        max_clusters: 100
        n_neighbors: 90
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
    }.
    [2025-04-15 03:29:59] [3991797] [INFO] Run neighbors_graph.
    [2025-04-15 03:29:59] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        n_neighbors: 90
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
    }.
    [2025-04-15 03:29:59] [3991797] [INFO] Found analytic results for neighbors_graph with same parameters.
    [2025-04-15 03:29:59] [3991797] [INFO] Run Louvain successfully.
    [2025-04-15 03:30:00] [3991797] [INFO] Run Louvain.
    [2025-04-15 03:30:00] [3991797] [INFO] Parameters: {
        embedding_key: f7078b35e2834e389a0ed2cbd9726135
        is_global: True
        resolution: 0.5
        max_clusters: 100
        n_neighbors: 90
        local_connectivity: 1
        bandwidth: 1
        mix_ratio: 1
    }.
    [2025-04-15 03:30:00] [3991797] [INFO] Run neighbors_graph.
    [2025-04-15 03:30:00] [399179