Paper: Learning the Hidden Set Locally (ICLR-2024)

Reviewer comment (Theory):
There is a gap between the length of the constructed protocols and ones that are known to exist. This is claimed to be very small, which I do not entirely agree with as it contains terms in n. I would definitely consider closing this gap as well as the gaps between lower and upper bounds open questions worth highlighting.

Author response:
Indeed, our statement was a bit ambiguous. We have clarified in the new version of the paper what we understand by the fact that the gap is small: the ratio between the two is only a polylogarithmic function of input parameters, where the degrees of polylogarithm are small. We will clearly emphasize that obtaining asymptotically optimal constructions (i.e., with ratio being a constant that does not depend on input parameters) and further improving constant factors are challenging open directions.

Reviewer comment (Presentation):
It seems that the upper bound results could be unified into one main result and a corollary and the same holds for the lower bounds.

Author response:
Thanks for this observation.  
Indeed, one could obtain a $(n,k)$-LocS from $(n,k,\ell)$-LocAS, constructed in Section 4, by setting $\ell=1$. However, the result would have larger polylogarithmic factor, and thus, would be asymptotically less efficient than the $(n,k)$-LocS obtained in direct construction in Section 3. The actual benefit of the construction in Section 3 depends on parameter $k$, for instance, if $k$ is at least some root of $n$, i.e., $k\sim n^c$ for some $c\in (0,1)$, the direct construction of $(n,k)$-LocS gives polylogarithmic factor of $\log^2 n$, as in Theorem 3.1, while obtaining $(n,k)$-LocS from $(n,k,1)$-LocAS construction in Section 4 would have polylogarithmic factor of $\log^3\log\log n$, as in Theorem 4.1 for $\ell=1$.  

The main technical difference between the two constructions is that, in order to assure avoidance property, we use more prime numbers in the construction in Section 4 comparing to the one in Section 3. In order to be able to select sufficient number of different prime numbers, we are bounded by the distributions of prime numbers -- each imposing roughly a logarithmic overhead. Possibly, one could improve the $(n,k,\ell)$-LocAS construction by taking *certain distribution of triplets* of prime numbers to induce smaller polylogarithmic overhead than the *product of three distributions* of prime numbers (used in our construction) -- this is, however, a challenging question in number theory, left for future study.  

Another benefit of having Section 3 is that the direct construction of $(n,k)$-LocS is slightly less complex than the construction of $(n,k,\ell)$-LocAS in Section 4 - therefore, it gradually introduces the readers first to obtaining the locality and next, adding avoidance property.  

We will add this whole discussion to Section 5, changing its title to "Further Discussion and Extensions".

Reviewer comment (Writing):
Judging from the last sentence I would expect a more in-depth discussion of how the algorithms could be extended to more feedback functions. Actually this is later only done in some detail for one more type of feedback functions. I would suggest weakening this sentence in the abstract (or omitting it).

Author response:
The Reviewer is right - we will try to be more specific by writing that we discuss only some extensions (in the Section 5), while others could be subject to future work.

Reviewer comment (Writing):
Page 1: "before the hidden set" rather than "prior the hidden set".

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 1: "using as small *a* number".

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 2: "ask *the* question of efficient".

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 2: "Intuitively *the* presence".

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 2: "each consecutive queries" is confusing because normally I would think this means consecutive pairs.

Author response:
We will re-write this sentence to clarify that in adaptive GT, algorithm can design an $i$-th query based on the feedback received for the preceding $i-1$ queries.  
A non-adaptive GT has to construct one universal sequence of queries and cannot change them during the learning process.

Reviewer comment (Novelty):
Page 2: I think it would be appropriate to give some bibliographic references for the encoding/decoding methods being a motivation for non-adaptive protocolls.

Author response:
We will provide a few citations, including a recent paper by Kowalski \& Pajak (NeurIPS 2022) which clearly separates the query construction algorithm (corresponding to the encoding algorithm) from the set reconstruction algorithm (corresponding to the decoding algorithm).

Reviewer comment (Writing):
Page 3: the caligraphic N notation for the universe is not really used and could be omitted.

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 3: Maybe for completeness it would be good to say that queries are subsets of the universe.

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Novelty):
Page 3 (and in other places): I think a source from 2018 is not particularly recent.

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Writing):
Page 9: "*The* adaptive version".

Author response:
We agree with all the suggestions and will update the paper accordingly.

Reviewer comment (Theory):
Is it true that the result for (n,k)-LocS could have been presented as a corollary of (n,k,l)-LocS? If so, why did you decide against it?

Author response:
Thanks for this observation.  
Indeed, one could obtain a $(n,k)$-LocS from $(n,k,\ell)$-LocAS, constructed in Section 4, by setting $\ell=1$. However, the result would have larger polylogarithmic factor, and thus, would be asymptotically less efficient than the $(n,k)$-LocS obtained in direct construction in Section 3. The actual benefit of the construction in Section 3 depends on parameter $k$, for instance, if $k$ is at least some root of $n$, i.e., $k\sim n^c$ for some $c\in (0,1)$, the direct construction of $(n,k)$-LocS gives polylogarithmic factor of $\log^2 n$, as in Theorem 3.1, while obtaining $(n,k)$-LocS from $(n,k,1)$-LocAS construction in Section 4 would have polylogarithmic factor of $\log^3\log\log n$, as in Theorem 4.1 for $\ell=1$.  

The main technical difference between the two constructions is that, in order to assure avoidance property, we use more prime numbers in the construction in Section 4 comparing to the one in Section 3. In order to be able to select sufficient number of different prime numbers, we are bounded by the distributions of prime numbers -- each imposing roughly a logarithmic overhead. Possibly, one could improve the $(n,k,\ell)$-LocAS construction by taking *certain distribution of triplets* of prime numbers to induce smaller polylogarithmic overhead than the *product of three distributions* of prime numbers (used in our construction) -- this is, however, a challenging question in number theory, left for future study.  

Another benefit of having Section 3 is that the direct construction of $(n,k)$-LocS is slightly less complex than the construction of $(n,k,\ell)$-LocAS in Section 4 - therefore, it gradually introduces the readers first to obtaining the locality and next, adding avoidance property.  

We will add this whole discussion to Section 5, changing its title to "Further Discussion and Extensions".

Reviewer comment (Writing):
In general, I had a hard time following the writing style of the paper. The sentences are sometimes unnecessarily complicated and/or too long, which makes it harder to understand the ideas presented in the paper. My overall comment is that the paper needs a significant amount of rewriting and polishing.

Author response:
Thank you for all the spotted write-up problems - we will take them into account when preparing an updated version.

Reviewer comment (Writing):
The improper citation notation is one of my major concerns about this paper. The existing citations are not inside parentheses which interrupts the flow of the text for the readers.

Author response:
We apologize for this inconvenience, but we followed the ICLR 2024 guidance as in the instruction and sample files: https://iclr.cc/Conferences/2024/CallForPapers and https://github.com/ICLR/Master-Template/raw/master/iclr2024.zip . We are, however, perfectly fine with changing the style, if needed.

Reviewer comment (Writing):
This sentence in the introduction needs citations: "Originally GT was applied for identifying infected individuals in large populations using pooled tests..."

Author response:
We will add an explicit citation (to the original Dorfman's paper).

Reviewer comment (Presentation):
LocAS is mentioned in the footnote of page 3 before it has been introduced in the main body of the text.

Author response:
Thank you for spotting this issue, we will fix it by referring to the the definition of the LocAS provided on the next page.

Reviewer comment (Writing):
The beeping feedback mention on Section 5.2 needs citations.

Author response:
We will add more citations to the beeping model, including the seminal work: Cornejo \& Kuhn. Deploying Wireless Networks with Beeps. In: DISC 2010.

Reviewer comment (Writing):
Note: I am aware that almost all of the remarks I make under weaknesses related to language and grammar concerns. I am also ware that some of them are easily fixable. I do not want to discriminate against anybody based on their linguistic abilities. My main problem here is that, when the frequency of such mistakes increase, it makes it harder to focus on the actual contents of the paper. Plus, it signals me that the sufficient effort to express the novelty of the paper clearly and efficiently has not been made. I strongly believe that expressing our ideas clearly is a skill we all should aim to advance ourselves as researchers.

Author response:
Thank you for all the spotted write-up problems - we will take them into account when preparing an updated version.

Reviewer comment (Writing):
What does "Our algorithms could be .. applied to codes." mean? Do you refer to a computer program for empirical implementation? If yes, why not provide it with the paper? Or does it mean that it can be applied to something like error correcting codes? If this is the case, explicit mentioning would be better. (I understand this is being explained on Section 5.3 at the very end but maybe a very brief explanation towards the beginning would have helped the reader better.)

Author response:
Thank you for pointing this out - throughout the paper, by "code" we mean a coding/decoding function used in information theory, such as error-correcting codes, compression codes, cryptographic codes, etc. We will clarify it in the beginning.

Reviewer comment (Novelty):
Are there another applications where learning the hidden set would be useful other than the group testing setting?

Author response:
In Section 1 we give references to applications in:  

- machine learning, such as: simplifying multi-label classifiers, eg. Ubaru et al. (NeurIPS 2020), approximating the nearest neighbor, eg. Engels et al. (NeurIPS 2021), or accelerating forward pass of a deep neural network, eg. Liang \& Zou (ISIT 2021);  

- stream processing, e.g., extracting the most frequent elements, see Cormode et al. (VLDB 2003), Cormode \& Muthukrishnan (ACM ToDS 2005), Cormode \& Hadjieleftheriou (VLDB 2008), Yu et al. (VLDB 2004), Kowalski \& Pajak (IJCAI 2022);  

- coding, eg. Kautz \& Singleton (IEEE ToIT 1964); Porat \& Rothschild (IEEE ToIT 2011); Cheraghchi \& Ribeiro (ISIT 2019);  

- network communication, eg. Clementi et al. (SODA 2001), Kowalski \& Pelc (PODC 2003), Jurdzinski et al. (PODC 2018).  

A part of the cited book by Du \& Hwang \& Hwang (2000) is devoted to applications of GT. See also examples in Kowalski \& Pajak (NeurIPS 2022).  

Specific applications close to our exact settings are discussed in more detail in sub-section "Motivation for this work.", including various variants of the problem.  

See also Section 5.2 ("Local GT as Codes"), where we discuss applications to coding theory.  

There is also a rich literature on GT applications, which we did not include in our work because it studies slightly different GT settings. Eg., average case instead of worst case performance over some probabilistic distribution of the sets of infected items, random choice of tests with applications in information theory (eg. Alaoui et al. Decoding from Pooled Data: Phase Transitions of Message Passing. ISIT 2017) and of interest for ML/NN community (eg. Scarlett \& Cevher. Phase Transitions in the Pooled Data Problem. NeurIPS 2017). We will add some of these examples to the paper in order to strengthen the motivation part of our work even further.

Reviewer comment (Writing):
What does "whichever are relevant" in the following sentence mean?: We say that Q is constructible in polynomial time if there exists a polynomial-time algorithm, that given parameters n, k, ℓ (whichever are relevant) outputs an appropriate sequence of queries satisfying the requirements.

Author response:
We meant that in the construction of LocS, we are given parameters $n,k$ as input, and the constructing algorithms should work in time polynomial in $n,k$. Whereas the constructions of LocAS, having three input parameters $n,k,\ell$, should work in time polynomial with respect to these three parameters. We will clarify it in the updated version.

Reviewer comment (Writing):
What are existential upper bounds?

Author response:
On p. 7 and in other similar places, we actually meant "existential result". By this, or more specifically "existential upper bound", we mean that the result, e.g., existence of some type of structure, is formally proven using mathematical arguments, but the proof does not provide an efficient (i.e., working in time polynomial in the size of the input) algorithm for building such a structure. We will provide detail explanation in the paper.