Paper: Experimental Design for Multi-Channel Imaging via Task-Driven Feature Selection (ICLR-2024)

Reviewer comment (Presentation):
Also you repeat some of the results from the beginning of section 4 afterwards in different paragraphs, which makes it hard to follow. I would reiterate here.

Author response:
Thank you for this feedback.  The aim of the paragraph at the start of section 4 is to give a short overview of the experiments and results and how they fit together. We believe this is important signposting to help the reader navigate the extensive results section and understand how they fit together to demonstrate TADRED's overall benefit.  Then we go into detail for each experiment in each paragraph.  We're unsure what you are suggesting we reiterate  and where, but we are keen to make this large section as clear as possible - any specific advice gratefully received!

Reviewer comment (Theory):
To me it is still not so clear the difference between feature selection and experimental design optimisation. In paper you mention the difference about features being irrelevant and channels being repetitive. How do you ensure or balance this in your work? Is there a specific loss you use?

Author response:
In general the problems of experiment design and feature selection differ in that the former is usually continuous (identifying a combination of points in a space of possible measurement settings) -- see first paragraph and equation 1 whereas the latter is inherently discrete (selecting a subset of a predefined superset) -- see paragraphs around equation 2.  The novelty of the paradigm we propose is to equate the two tasks  As we explain in paragraph 6 of the introduction, the key difference in the problems once equated in this way lies in the structure of the problem: often in classical feature selection problems most features are uninformative and the challenge is to identify a small fraction that carry important information.  In the experiment design problem, most features/channels are informative, as it is usually easy to identify image channels with no information content a priori.  The challenge is to identify a combination of channels that are maximally complementary.  This difference demands a different search strategy, which we expand on in `Supervised Feature Selection' paragraph of the related work.  TADRED does not require a particular loss, similar to the baselines, we use mean-squared-error (MSE).

Reviewer comment (Experiments):
**Validation of score network**: I think a nice validation that the score network is actually working as intended would be to set random score values at the beginning of optimization of TADRED for each channel (and keep them fixed). This way, the task network is optimized for the same number of steps and in the same manner, i.e. with gradually less channels. Maybe this gradual aspect is actually making the reconstruction network better and it's why TADRED is performing better. I list this as a weakness and not as a question, because to me the experiments are lacking of proof that the score network is working as intended and not just a smart preprocessor or anything else.

Author response:
Thank you for the question.  We have now tested a modified training procedure that works in the 'same manner' whilst the scores are random. In settings of table 5 results (MSE) are more than 10 \% worse, even more than 'w/o iterative subsampling' - we are repeating over more randomizations before adding any statement to the paper.  Thus, as you suggest, although the 'gradual aspect' you mention improves performance (in fact line 2 in the ablation study in table 5 already demonstrates this with `less gradual' scenario without iterative subsampling decreases performance), the learning of the scoring network is working as intended and further improves results. The objective is (see related work) for the 'scoring and subsampling procedure enables efficient identification of subsets of complementarily informative channels jointly with training a high-performing network for the task' -- which is showed through large improvements over the baselines in multiple tasks.

Reviewer comment (Experiments):
In the experiment section, although quite a number of baselines are used for comparison, including Fisher-matrix, the baselines are not clearly explained. Some details are included in the appendix, but it may be better to at least explain one of them in detail.

Author response:
Thank you for the feedback.

The baselines are quite complicated.  For example FIRDL's pseudocode in [93]-appendix-D runs to 1.5 pages and SSEFS [56]-algs-1,2,3,4 is even longer.  In fact we believe the relative simplicity of our algorithm is a nice advantage of our method.  Thus, rather than explain in great detail, we have tried to give the essence of each method in related work section 2 and refer to the corresponding papers for full descriptions.  Note Wojtas \& Chen (2020) and Lee et al. (2022) -- and others use the same approach when discussing prior work / baselines in the respective papers.  We do give substantial detail in Appendix A and while we don't think it is useful to repeat the full detail of any particular method if the reviewer feels there is necessary information missing we would be very happy to add it.  

Please note we discussed the baselines and how they relate to our approach in the related work section 2 which is quite expansive (greater than 1 page). More specifically the Fisher Information Baseline (used in table 1 top, table 7 top) in 'Approaches in Experimental Design'; and the supervised feature selection baselines (FIRDL, SSEFS) along with the PROSUB Blumberg et al. (2022) in the three paragraphs of `Supervised Feature Selection'.  Finally, the other reviewers liked the discussion of prior work (see general response).

Reviewer comment (Writing):
The paper could do a better job explaining the proposed method. For instance, algorithm 2 would benefit from additional comments. I have little intuition for what each step is trying to do.

Author response:
Thank you for highlighting this.  The intention was to keep the intuition to the main text leaving the algorithm succinct and uncluttered.  We have now added comments to the algorithm to make the mapping between steps of the algorithm and blocks of the main text clear, which we hope resolves this issue.

Reviewer comment (Presentation):
The figures and table captions are generally uninformative, which makes the results very difficult to follow.

Author response:
**The figures table captions are generally uninformative which makes the results very difficult to follow**

Reviewer comment (Presentation):
In figure 1, "Model Free" is uninformative.

Author response:
We agree and have removed `model free' from the figure.

Reviewer comment (Presentation):
Table 1 is very hard to parse. It's essentially two tables, with the top table having a different format than the bottom one.

Author response:
Thank you for the suggestion.  As the top table only has a single comparison, we replaced it with a sentence in the main text (paragraph `TADRED Outperforms Classical Experimental Design and Baselines in Model Parameter Estimation').

Reviewer comment (Presentation):
Figure 2 is covered with arrows, but the caption doesn't state what they're pointing to.

Author response:
Thank you for spotting this.  We have adjusted the caption accordingly `Right: Qualitative comparison where arrows highlight closer agreement from TADRED's design with the gold standard than those from the best performing baseline in each metric.'

Reviewer comment (Writing):
"Feature Fill" is not defined in the main text.

Author response:
Thank you for pointing this out.  We have explicitly added `feature fill' in the text under equation (3) page 5 where the corresponding symbol first appears.

Reviewer comment (Writing):
The sentence, "e.g. the standard design for VERDICT model Panagiotaki et al. (2015a) (used as a baseline in table 1) is computed by optimizing the Fisher-matrix for one specific combination of parameter values, despite aiming to highlight contrast in those parameters throughout the entire prostate" in related work is provided without the context of prostate cancer.

Author response:
We modified the main text and adaded `with primary application in prostate cancer detection and classification'.

Reviewer comment (Writing):
The statement "Rather than learning the mask mt end-to-end e.g. using a sparsity term/prior, we modify elements of mt during our training procedure" could use further clarification.

Author response:
We are referring to the baseline SSEFS approach here, we have modified the text to include an explicit reference.  Please note in the `Supervised Feature Selection' paragraph in section 2 we describe the relationship of TADRED with the SSEFS baseline in detail.