Paper: FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling (ICLR-2024)

Reviewer comment (Evaluation):
There are many evaluation ways to assess the fairness problem. The selected metrics might not be the most suitable one. Please elaborate more on the motivation of baseline setting and evaluation.

Author response:
We introduced a novel metric for assessing fairness in medical segmentation, called Equity-Scaled Segmentation Performance (ESSP). ESSP offers a more direct and clinician-friendly evaluation compared to traditional fairness metrics like DPD and DEOdds. Unlike DPD and DEOdds, which may overlook overall performance, leading to a situation where a model with uniformly lower performance across all demographics could falsely appear fairer. This misalignment is particularly problematic in safety-critical medical applications, which demand high accuracy.  In contrast, our proposed ESSP addresses this issue. It evaluates not only the disparity across different demographic groups but also measures the extent to which overall segmentation accuracy is compromised to achieve a fairer model. A higher ESSP score indicates a model that achieves both fairness and high accuracy.

Alongside ESSP, according to the reviewer’s suggestion, we have also updated the DPD and DEOdds results in the supplementary material, providing a comprehensive fairness assessment of various segmentation approaches. As the table below illustrates, our FEBS (Fairness-Enhanced Biased Segmentation) approach outperforms previous methods in terms of DPD and DEOdds, highlighting its effectiveness in achieving fairer outcomes in medical imaging. From the table below, we show that our method achieves comparable DPD and DEodds performances when compared against other segmentation methods. 

|      | Method            | Overall DPD | Overall DEodds |
|------|-------------------|-------------|----------------|
| Cup  | SAMed             | 0.0085      | 0.0196         |
|      | SAMed+ADV         | 0.0079      | 0.0071         |
|      | SAMed+GroupDRO    | 0.0078      | 0.0154         |
|      | Ours (SAMed)      | 0.0079      | 0.0020         |
|      | TransUNet         | 0.0083      | 0.0430         |
|      | TransUNet+ADV     | 0.0074      | 0.0389         |
|      | TransUNet+GroupDRO| 0.0081      | 0.0317         |
|      | Ours (TransUNet)  | 0.0085      | 0.0492         |
| Rim  | SAMed             | 0.0005      | 0.0670         |
|      | SAMed+ADV         | 0.0004      | 0.0657         |
|      | SAMed+GroupDRO    | 0.0009      | 0.0792         |
|      | Ours (SAMed)      | 0.0002      | 0.0715         |
|      | TransUNet         | 0.0014      | 0.0877         |
|      | TransUNet+ADV     | 0.0025      | 0.0708         |
|      | TransUNet+GroupDRO| 0.0018      | 0.0699         |
|      | Ours (TransUNet)  | 0.0014      | 0.0822         |

Reviewer comment (Experiments):
Since most of the attributes are only for the patient level, why use the pixel-wise weights?

Author response:
Patients from various demographic groups may exhibit different anatomical characteristics. For example, Black people often have a larger cup-to-disc ratio and cup area compared to other races. These anatomical differences can influence segmentation accuracy. To address this, we employ pixel-wise weights to accommodate these underlying anatomical variations within the fundus images.

Reviewer comment (Evaluation):
Dice and IoU are equivalent (https://www.sciencedirect.com/science/article/pii/S1361841521000815), which are not necessary to be reported simultaneously. Instead, please add NSD which is suggested by metrics reloaded (https://arxiv.org/abs/2206.01653).

Author response:
Dice and IoU are commonly used by previous research [1,2,3]. For the sake of future comparison and completeness, we will keep the Dice and IoU in the table, and add NSD as the additional metric. Furthermore, we have included the NSD results for race as table below. The table indicates that unfairness commonly exists in our proposed fair segmentation task, which further strengthens the contribution of our FairSeg dataset. We have added those results in the supplementary material.  

|      |                       | **Overall NSD↑** | **Asian NSD↑** | **Black NSD↑** | **White NSD↑** |
|------|-----------------------|------------------|----------------|----------------|----------------|
| **Cup**|                     |                   |                  |                |                |
| |SAMed                       | 0.7222            | 0.6825           | 0.6871         | 0.7338         |
| |SAMed+ADV                   | 0.7405            | 0.7045           | 0.6954         | 0.7537         |
| |SAMed+GroupDRO              | 0.7415            | 0.7093           | 0.6942         | 0.7547         |
| |**Ours (SAMed)**            | 0.7399            | 0.7038           | 0.7015         | 0.7518         |
| |TransUNet                   | 0.9314            | 0.8845           | 0.8908         | 0.9449         |
| |TransUNet+ADV               | 0.9208            | 0.8819           | 0.8817         | 0.9331         |
| |TransUNet+GroupDRO          | 0.9285            | 0.8768           | 0.8873         | 0.9426         |
| |**Ours (TransUNet)**        | 0.9262            | 0.8796           | 0.8850         | 0.9397         |
| **Rim**  |                   |                   |                  |                |                |
| |SAMed                       | 0.7483            | 0.7313           | 0.7215         | 0.7556         |
| |SAMed+ADV                   | 0.8063            | 0.7694           | 0.7686         | 0.8182         |
| |SAMed+GroupDRO              | 0.8078            | 0.7760           | 0.7696         | 0.8191         |
| |**Ours (SAMed)**            | 0.8043            | 0.7732           | 0.7704         | 0.8146         |
| |TransUNet                   | 0.9601            | 0.9326           | 0.9326         | 0.9688         |
| |TransUNet+ADV               | 0.9554            | 0.9245           | 0.9221         | 0.9656         |
| |TransUNet+GroupDRO          | 0.9596            | 0.9322           | 0.9307         | 0.9685         |
| |**Ours (TransUNet)**        | 0.9581            | 0.9306           | 0.9315         | 0.9666         |

Reviewer comment (Experiments):
nnUNet is still the state-of-the-art in many segmentation tasks. It would be great to evaluate it on your dataset.

Author response:
We have included nnUNet in our segmentation benchmarks, which are detailed in the supplementary material. The table provided below presents results focusing on racial disparities. From the table, it is evident that nnUNet demonstrates marginally better performance compared to SAMed and TransUNet. However, it still exhibits significant performance disparities across different racial groups. This suggests that algorithmic unfairness is a pervasive issue in our proposed cup-disc segmentation, regardless of the choice of segmentation architectures.


|       | Method | Overall ES-Dice↑ | Overall Dice↑ | Overall ES-IoU↑ | Overall IoU↑ | Asian Dice↑ | Black Dice↑ | White Dice↑ | Asian IoU↑ | Black IoU↑ | White IoU↑ |
|-------|--------|------------------|---------------|-----------------|--------------|-------------|-------------|-------------|------------|------------|------------|
| Cup   | nnUNet | 0.8625           | 0.8710        | 0.7704          | 0.7865       | 0.8675      | 0.8855      | 0.8697      | 0.7639     | 0.8035     | 0.7725     |
| Rim   | nnUNet | 0.8003           | 0.8335        | 0.6959          | 0.7231       | 0.7930      | 0.7682      | 0.8490      | 0.6854     | 0.6639     | 0.7397     |

Reviewer comment (Reproducibility):
The dataset was released as npz format. Could you please also release the original format?

Author response:
We will release the dataset with the original format in addition to the existing npz format of our database.

Reviewer comment (Reproducibility):
It would be great if you could release the trained models as well.

Author response:
The trained checkpoints of our models have been released through our Github repository.

Reviewer comment (Reproducibility):
Where do you plan to host this benchmark? CodaLab could be a good platform.

Author response:
We will co-host our benchmark/dataset using both Google Drive and CodaLab.

Reviewer comment (Presentation):
There is no visualization comparison between different methods.

Author response:
We have added a visualization comparison of segmentation results between our method and other competing approaches.

Reviewer comment (Writing):
In equation (1), a parenthesis is missing in the formula.

Author response:
We have addressed the formatting issue in the paper.

Reviewer comment (Experiments):
Why validation set is not constructed/used in selecting models in training?

Author response:
**Why validation set is not constructed/used in selecting models in training?**

Given that our model is selected based on the last epoch of training. The finetuning of foundation models like SAM could be computationally expensive for both training and inference. This makes computing the validation accuracy every few epochs and selecting the checkpoints based on the accuracy is infeasible.

To better facilitate future research, we have released an extra 500 images as the validation set in our codebase and dataset.

Reviewer comment (Evaluation):
It would be helpful to report Hausdorff distance and average surface distance along with Dice to better evaluate the methods.

Author response:
**Report Hausdorff distance and average surface distance along with Dice.**


Please see the table below for Hausdorff distance (HD95) and average surface distance (ASD) for race. We have included such metrics across all sensitive attributes in the supplementary material. From table below, we observed that the disparity between different demographic groups commonly existed regardless of its evaluation metrics, which further strengthened the significance of our proposed FairSeg dataset.  


|      | Method               | Overall HD95↓ | Asian HD95↓ | Black HD95↓ | White HD95↓ | Overall ASD↓ | Asian ASD↓ | Black ASD↓ | White ASD↓ |
|------|----------------------|---------------|-------------|-------------|-------------|--------------|------------|------------|------------|
| **Cup**  | SAMed                | 9.6231        | 11.0005     | 11.0142     | 9.1866      | 3.9650       | 4.6765     | 4.7743     | 3.7209     |
|      | SAMed+ADV            | 9.4594        | 11.0170     | 11.1193     | 8.9479      | 4.0123       | 4.8623     | 4.9236     | 3.7321     |
|      | SAMed+GroupDRO       | 9.4633        | 11.0891     | 11.0449     | 8.9603      | 3.9494       | 4.6462     | 4.8634     | 3.6855     |
|      | Ours (SAMed)         | 9.4494        | 10.8376     | 11.1637     | 8.9453      | 3.9288       | 4.7042     | 4.8210     | 3.6606     |
|      | TransUNet            | 4.7577        | 5.6983      | 5.5411      | 4.4937      | 2.0552       | 2.3692     | 2.4573     | 1.9382     |
|      | TransUNet+ADV        | 5.0157        | 5.9379      | 5.8292      | 4.7476      | 2.1319       | 2.4693     | 2.4978     | 2.0198     |
|      | TransUNet+GroupDRO   | 4.8195        | 5.6424      | 5.6753      | 4.5535      | 2.0615       | 2.3637     | 2.4952     | 1.9393     |
|      | Ours (TransUNet)     | 4.9603        | 5.8818      | 5.7398      | 4.6992      | 2.1441       | 2.4641     | 2.5445     | 2.0268     |
| **Rim**  | SAMed                | 9.9379        | 11.4859     | 11.5671     | 9.4337      | 3.9353       | 4.5013     | 4.5473     | 3.7476     |
|      | SAMed+ADV            | 8.8316        | 10.5364     | 10.2364     | 8.3562      | 3.3523       | 3.9946     | 3.8975     | 3.1700     |
|      | SAMed+GroupDRO       | 8.7609        | 10.4638     | 10.0592     | 8.3076      | 3.3473       | 3.8345     | 3.9499     | 3.1702     |
|      | Ours (SAMed)         | 8.7930        | 10.3167     | 10.3407     | 8.3082      | 3.4174       | 4.0053     | 4.0608     | 3.2208     |
|      | TransUNet            | 4.4404        | 5.4496      | 5.0893      | 4.1964      | 1.7534       | 1.9643     | 1.9941     | 1.6809     |
|      | TransUNet+ADV        | 4.6301        | 5.6065      | 5.4390      | 4.3569      | 1.8110       | 2.1085     | 2.0885     | 1.7213     |
|      | TransUNet+GroupDRO | 4.5268 | 5.4243 | 5.2290 | 4.2842 | 1.7498 | 1.9683 | 2.0017 | 1.6742 |
|      | Ours (TransUNet)   | 4.5311 | 5.5251 | 5.2808 | 4.2682 | 1.8321 | 2.0642 | 2.0771 | 1.7564 |

Reviewer comment (Evaluation):
The details of how standard deviation is computed need to be elaborated. Is it computed across the mean of for each group?

Author response:
**The details of how standard deviation is computed need to be elaborated. Is it computed across the mean of for each group?**

The computational cost of multiple runs with large segmentation foundation models is high. Previous studies  [3, 4]  typically evaluate these models only once, without standard deviation, rather than conducting multiple runs of evaluations.

Reviewer comment (Experiments):
How is the training/testing split performed? Is it just randomly sampled without considering sensitive attributes at patient level?

Author response:
**How is the training/testing split performed? Is it just randomly sampled without considering sensitive attributes at patient level?**

The training and testing split are just randomly sampled, and our sample distribution of different sensitive attributes reflects the real-world clinical patient distribution.

Reviewer comment (Experiments):
It would be helpful to discuss the importance of registration in preprocessing using NiftyReg.

Author response:
**It would be helpful to discuss the importance of registration in preprocessing using NiftyReg.**

As mentioned above and in Section 4/Figure 1, to obtain a large-scale fundus dataset with high-quality pixel-wise annotation, we need to use the OCT machine to generate the OCT fundus images and their corresponding cup-disc mask. However, OCT machines are fairly expensive and less prevalent in primary care, therefore, we propose to migrate those annotations from 3D OCT to 2D SLO fundus for potentially broader impact in early-stage glaucoma screening in the primary care domains. In order to transfer the annotations from 3D OCT fundus images to 2D SLO fundus images, we need to register the image to align the two imaging modalities by comparing the characteristic features between the two fundus imaging modalities of the same patient.  The computed alignment matrix is then applied to the disc-cup masks of the OCT fundus images, aligning them to the SLO fundus images.