vanvalenlab/torch-spots

Model Card for Model ID

This is a model that detects sub-pixel coordinates for spots in spatial transcriptomics experiments. It is based on a FeatureNet backbone with semantic heads predicting spot probability and subpixel offset.

Model Details

Model Description

• Developed by: The Van Valen Lab
• Model type: weakly-supervised deep learning spot detection
• Language(s) (NLP): en
• License: Modified Apache 2.0 Noncommercial

Model Sources

• Repository: Github
• Paper: Accurate single-molecule spot detection for image-based spatial transcriptomics with weakly supervised deep learning

Uses

This model is useful for fast and accurate detection of spots generated in spatial transcriptomics datasets. Spots constitute single "reads" that, over many rounds, code for individual gene products. This model takes in an image of any shape and identifies the spots at sub-pixel resolution.

Direct Use

This model can be used out of the box in the following way:

1. pip install the package from the github repository.
2. Ensure you have a Hugging Face API key connected to the computer you want to do inference on.
3. When you instantiate the SpotDetection application, the model weights will be downloaded directly to your computer in the canonical ~/.deepcell location.
4. Run the predict_points method by supplying an image of shape (B, 1, H, W)
5. The points will be returned as an (N,2) array containing the point coordinates.

Out-of-Scope Use

Spot detection has been trained, evaluated, and benchmarked on spatial transcriptomics data. Attempting to detect spots from immunofluorescence or any other modality may yield incorrect results.

Bias, Risks, and Limitations

As stated previously, the spot detection has been trained, evaluated and benchmarked on spatial transcriptomic data. This means that, although other data may form foci that look similar, the results will not be accurate.

The model tiles the input image under the hood, conducts inference on each (128x128) tile of the image. This is done batchwise, so one should limit the batch number if vRAM is at a premium. In addition, the current implementation requires the user to read the whole image into memory before inference, so if RAM is also at a premium, consider tiling the full image before sending a smaller tile through the predict_points method.

How to Get Started with the Model

Use the code below to get started with the model.

import tifffile
from torch_spots.detection.inference import SpotDetection
app = SpotDetection() # This will download the model weights if necessary

img = tifffile.imread('your_image.tiff')

print(img.shape) 

# should be shape 1, 1, H, W. 
# If multi-channel, process each channel separately

spots = app.predict_points(img)

Training Details

Training Data

The ground truth data was generated in a weakly-supervised fashion. A set of fluorescence images containing spots were analyzed using manual fine-tuning of classical spot detection algorithms. This manually-tuned algorithms were referred to as "annotators". Using generative modeling of the spots predicted by these annotators, we found a "consensus spot location," which served as our ground truth.

The training images were extracted from 128 px x 128 px crops from several spatial transcriptomics datasets and paired with their respective spot locations for training.

Training Procedure

Preprocessing

Each image was preprocessed by min-max normalization, followed by zero-centering so that the data ranged from -0.5 to 0.5.

Each image was then passed through an augmentation pipeline that added random rotations, translations and zooms to each image. The corresponding points were also rotated, translated and zoomed with the same parameters.

Training Hyperparameters

• Training regime: fp32
• Learning rate: 0.001
• Batch size: 10
• Epochs: 20
• Loss: Custom
• Optimizer: SGD with Nesterov momentum

Loss function

The loss function for training is composed to two separate losses that are summed together to yield the final loss. The first is weighted categorical cross-entropy for the classification heads. The second is mean-squared error (MSE) loss for the regression heads. We then took the weighted average of these two losses, biasing the classification loss five-fold more than the regression loss.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on 94 test images from a mixture of seqFISH datasets, MERFISH datasets, and SunTag datasets. These images were cropped to 128 px x 128 px and min-maxed zero-centered scaled. We did not augment these test datasets.

Metrics

We used a mutual nearest neighbors approach to match the points from the ground truth points to those we predicted. From this, we calculated the precision, recall, F1 score, Jaccard index, and root mean standard error (RMSE) between the list of points.

Environmental Impact

• Hardware Type: RTX A6000
• Hours used: 1
• Cloud Provider: Private infrastructure
• Carbon Efficiency: 0.432 kg/kWh
• Carbon Emitted: 0.13 kg eq.

Citation

BibTeX: in progress

APA: in progress

Model Card Contact