Model Card for Model ID

This model takes in an image of a fish and segments out traits, as described below

Model Details

Trained_model_SM.pth is the fish segmentation model.

se_resnext50_32x4d-a260b3a4.pth is a pretrained ConvNets for pytorch ResNeXt used by BGNN-trait-segmentation. See github.com/Cadene/pretrained-models.pytorch#resnext for documentation about the source.

The segmentation model was first trained on ImageNet (Deng et al., 2009), and then the model was fine-tuned on a specific set of image data relevant to the domain: Illinois Natural History Survey Fish Collection (INHS Fish). The Feature Pyramid Network (FPN) architecture was used for fine-tuning, since it is a CNN-based architecture designed to handle multi-scale feature maps (Lin et al., 2017: IEEE, arXiv). The FPN uses SE-ResNeXt as the base network (Hu et al., 2018: IEEE, arXiv).

Model Description

PyTorch implementation of Fish trait segmentation model. This segmentation model is based on pretrained model using the segementation models torch. Then the model is fine tuned on fish images in order to identify (segment) the different traits.

Trait list:

background     
dorsal_fin
adipos_fin
caudal_fin
anal_fin
pelvic_fin
pectoral_fin
head
eye
caudal_fin_ray
alt_fin_ray
trunk

Developed by: M. Maruf and Anuj Karpatne
License: MIT

Model Sources

Repository: BGNN-trait-segmentation
Paper: [In progress]

How to Get Started with the Model

See instructions for use here.

Training Details

The image data were annotated using SlicerMorph (Rolfe et al., 2021) by collaborators W. Dahdul and K. Diamond.

Training Data

To increase the size and diversity of the training dataset (originally 295 images), we employed data augmentation techniques such as flipping, shifting, rotating, scaling, and adding noise to the original image data to increase the dataset 10-fold. We developed 12 target classes, or trait masks, for our segmentation problem, each representing different morphological traits of a fish specimen. The segmentation classes are: dorsal fin, adipose fin, caudal fin, anal fin, pelvic fin, pectoral fin, head minus the eye, eye, caudal fin-ray, alt fin-ray, alt fin-spine, and trunk. Although minnows do not have adipose fins, the segmentation model was trained on a variety of fish image data, some of which had adipose fins. We retained this class because the segmentation model may erroneously assign an adipose fin to a minnow (Fig. S1), and a domain scientist examining these outputs may want to analyze the accuracy of the model.


Figure S1. Image of a segmented minnow with an erroneously marked anal fin.
The segmentation has 13 classes: background (black), dorsal fin (red), adipose fin (green), caudal fin (blue), anal fin (yellow), pelvic fin (light blue), pectoral fin (pink), head minus the eye (white), eye (bright green), trunk (teal), caudal fin-ray (light red), alt fin-ray (light pink), and alt fin-spine (peach).

The training dataset utilized the image files listed in training_dataset_INHS.txt.

The validation dataset utilized the image files listed in validation_dataset_INHS.txt.

Training Procedure

We prepared the model by using the Segmentation Model PyTorch library (Iakubovskii, 2019) to load an FPN segmentation model that was pretrained on the Imagenet dataset. We used SE-ResNeXt as the base network/encoder to extract features (embedding) from the input image data and replaced the last decoder layer with 12 target classes. During the fine-tuning procedure, the encoder of the pre-trained model was frozen as these layers already contain useful features that we can leverage. We only tuned the decoder weights of our segmentation model during this fine-tuning procedure.

We then trained the prepared model for 120 epochs, updating the weights using dice loss as a measure of similarity between the predicted and ground-truth segmentation. The Adam optimizer (Kingma & Ba, 2014) with a small learning rate (1e-4) was used to update the model weights.

Evaluation

We evaluated the performance of the fine-tuned segmentation model on the test set using the Intersection over Union (IoU) score with a 0.5 threshold. (The IoU score ranges from 0 to 1, with 1 indicating a perfect overlap between the predicted segmentation and the ground-truth segmentation and 0 indicating no overlap.) Our segmentation model achieved a 0.90 mIoU score on the test dataset.

Testing Data, Factors & Metrics

We had 99 testing images and 98 validation images.

More Information

Research supported by NSF Office of Advanced Cyberinfrastructure (OAC) Awards #2022042, #1940233, #1940322, and #1940247, with additional support from NSF Award #2118240. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.