Paper: BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics (ICLR-2024)

Reviewer comment (Evaluation):
Many evaluation datasets are proposed. Does each of these datasets emphasize something (e.g. Label shift influence, long-tail distribution)? If not, they lack of research potential to be set as a part of benchmark dataset. A good example for constructing evaluation set would be ImageNet-C[1].

Author response:
Each of the datasets maps to one of the tasks we describe in Section 4.3, each of which aligns to one or more challenges that we measure by evaluating on that dataset. While the alignment is not necessarily one-to-one, the datasets are diversified across geographical locations and species distributions, which helps establish generalizability in the benchmark results. We also note that the "Disentangling Label and Covariate Shift" subsection in Section 4.3 does present ablations that directly disentangles the various learning challenges presented by BIRB. The methodology we present there can be followed by users of the benchmark to study the effect of those challenges in isolation.

"Integration" benchmarks like BIRB are important to deep learning researchers for the reasons outlined in our response to Weakness 1. Our benchmark is intended to drive fundamental research in a way that is directly beneficial to ecological researchers and conservationists, whose modelling challenges do not neatly decompose into different research subfields. Transfer learning, domain adaptation/generalization, and class-imbalanced learning applications all have the opportunity to demonstrate effectiveness on BIRB, and we believe that the fact that such advances can directly translate to real-world impact will be encouraging to many researchers in those fields.

Reviewer comment (Experiments):
Throughout the paper, no example dataset sample is provided to give intuition of how the dataset is constructed.

Author response:
To clarify, are you referring to an example as it goes through the data processing pipeline, or do you mean more explicitly the modality/format of an example? We'd be glad to provide a more explicit description and/or additional modality to illustrate this more clearly--please let us know which component you were referring to.

Reviewer comment (Experiments):
Is it a reasonable practice to use existent public datasets to construct one's own dataset? Are the credits properly given?

Author response:
There is precedent in constructing "datasets of datasets" benchmarks; refer for instance to "A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark" or "Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples". We agree that proper credit attribution is important in those cases, which we do through citations in Table 1. We will clarify in the text that we are making use of data collected as part of previous works to help avoid any ambiguity, and we are happy to incorporate any further suggestion on how to make that clear. 

We also note that the datasets we employ are designed for ecologically themed research and are not usable off-the-shelf in machine learning applications. They do not prescribe any specific evaluation procedure; the evaluation protocol we propose is our own contribution. The datasets were also prepared by a variety of different groups, which required significant effort on our part to achieve data consistency across datasets. Preprocessing details (for reproducibility) are outlined in Appendix 1, which we summarize here:

- Resolving significant differences in the bird taxonomies used by various datasets
- Correctly and consistently aligning labels in various input formats
- Extracting fixed length slices from file-level labels/annotations using peak-finding (Xeno-Canto queries)
- Converting timeboxed annotations in soundscape data into fixed length labeled slices

Reviewer comment (Writing):
What does BIRB stand for? Apologies in advance if I missed it in the paper.

Author response:
Thank you for pointing this out. We will clarify by expanding the acronym (Benchmark for Information Retrieval in Bioacoustics) the first time it is used in the main text. We will also add a footnote to an online article (https://www.audubon.org/news/when-bird-birb-extremely-important-guide) to provide context on the origin of the word.

Reviewer comment (Novelty):
While the paper primarily focuses on introducing the benchmark, it could benefit from more innovative approaches or methodologies for retrieval tasks. I think it's important to inspire researchers with novel ideas for addressing the challenges in bioacoustics, beyond providing a baseline model.

Author response:
> [The benchmark] could benefit from more innovative approaches or methodologies for retrieval tasks. I think it's important to inspire researchers with novel ideas for addressing the challenges in bioacoustics, beyond providing a baseline model.

The ICLR 2024 call for papers explicitly mentions datasets and benchmarks in the list of relevant topics. As such, our contributions in designing an evaluation protocol which closely aligns with the real-world needs of ecological researchers and conservationists and demonstrating that it represents a significant and multi-faceted challenge stand on their own. In the evaluation framework we present, we encapsulate naturally occurring challenges, align these with important ML problem areas, and disentangle the influences of the underlying challenges into a comprehensive suite of tasks. Furthermore, in our initial empirical investigation and evaluation, we characterize the extent to which each provides challenges for the models we benchmark. Lastly, we provide inspiration to researchers by outlining concrete follow-up directions by enumerating our findings and open problems that stem from these. 

## Rebuttal 3 (ID: A57UCiQ4nj)
Thank you for responding. We disagree with the claim that our submission should present innovative or significant approaches on top of the initial baselines we contribute in this work (in addition to the framework). The [ICLR 2024 Call for Papers](https://iclr.cc/Conferences/2024/CallForPapers#:~:text=datasets%20and%20benchmarks) explicitly lists datasets and benchmarks as part of the relevant topics. A reviewer insisting that our submission should also present novel technical contributions on top our our extensive benchmarking contributions (from your review: "a novel and comprehensive benchmark designed for evaluating model generalization in the field of bioacoustics.") imposes an unfair double burden on us. For your reference, here is a (non-exhaustive) list of benchmark papers ICLR 2023 that demonstrate precedence in accepting work on benchmarking contributions alone:

- [Dr. Spider](https://openreview.net/forum?id=Wc5bmZZU9cy): *Top-5% paper*, Section 4 lists 7 approaches, all prior works, which are evaluated on their benchmark.
- [DaxBench](https://openreview.net/forum?id=1NAzMofMnWl): *Top-5% paper*, Sections 4.2 through 4.4 list 6 approaches in total, all prior works, which are evaluated on their benchmark.
- [A framework for benchmarking class-out-of-distribution detection and its application to ImageNet](https://openreview.net/forum?id=Iuubb9W6Jtk). *Top-25% paper*, benchmarks 525 ImageNet-1k classifiers available from popular repositories.
- [STREET](https://openreview.net/forum?id=1C_kSW1-k0): *Top-25% paper*, all approaches listed in Section 3 either adopt the methodology of or make direct use of prior works.
- [MEDFAIR](https://openreview.net/forum?id=6ve2CkeQe5S): *Top-25% paper*, Section 3.2 lists 11 algorithms, all prior works, used for evaluation in the benchmark.

We hope this clarifies for you that the benchmarking contributions you noted as part of the submission's strengths do meet the bar for acceptance.

Reviewer comment (Experiments):
The paper could be enhanced by a more extensive comparative analysis of different approaches or models for the tasks presented in the benchmark. Currently, it provides results for a set of baseline models but does not explore alternative methodologies.

Author response:
> The paper could be enhanced by a more extensive comparative analysis of different approaches or models for the tasks presented in the benchmark.

Within the extensive comparative analysis that we've presented in this work, are there any specific approaches or models that you think would be important to include in providing this first set of comprehensive evaluations, results, and analyses?

## Rebuttal 2 (ID: kY7Em27PUP)
Dear Reviewer N4MM: we would value your input on the approach(es) you think are important to compare against within the presentation and baseline evaluation of the BIRB benchmark. 

It’s worth acknowledging that extending beyond the baseline setting we present in this work is itself a viable standalone research problem and contribution. Adopting a “more complex” approach requires either applying it as an “off-the-shelf” approach and accepting that it may perform quite poorly on some of these tasks (which isn’t necessarily conclusive of the approaches applicability for this setting or tasks), or requires substantial innovation to address the complex challenges included in this benchmark, i.e. audio domain (not image), substantial class imbalance, multiple sources of distribution shift. 

Beyond that point, we believe we addressed your other questions and concerns; if you agree, would you be willing to update your score?

Reviewer comment (Evaluation):
This work provides detailed results but could improve the interpretation of these results. For instance, it mentions that there's a significant difference in performance between deep models trained on XC upstream data and models pre-trained on AudioSet, but it doesn't delve into the reasons behind this discrepancy or suggest potential solutions.

Author response:
> In regards to Weaknesses 3 and Question 1:

A meaningful insight from our work reveals the importance of using an appropriate learned representation in this framework when addressing bioacoustics retrieval. In particular, the model trained on XC upstream is more relevant and applicable to this task, providing a rich feature space that translates well to downstream tasks, much more so than the AudioSet dataset which includes many unrelated or irrelevant acoustics data.
For domain adaptation, our conclusion is that we benefit from training on data that is more directly related than data that is less so. More explicitly, bird vocalizations provide a more useful representation for detecting other bird vocalizations than the broad and less relevant classes included in AudioSet.

Reviewer comment (Evaluation):
The paper mentions performance differences between models pre-trained on AudioSet and those trained on XC upstream data. Could the authors offer more insights into why this divergence occurs and what it might imply for domain adaptation in bioacoustics?

Author response:
> In regards to Weaknesses 3 and Question 1:

A meaningful insight from our work reveals the importance of using an appropriate learned representation in this framework when addressing bioacoustics retrieval. In particular, the model trained on XC upstream is more relevant and applicable to this task, providing a rich feature space that translates well to downstream tasks, much more so than the AudioSet dataset which includes many unrelated or irrelevant acoustics data.
For domain adaptation, our conclusion is that we benefit from training on data that is more directly related than data that is less so. More explicitly, bird vocalizations provide a more useful representation for detecting other bird vocalizations than the broad and less relevant classes included in AudioSet.

Reviewer comment (Experiments):
Are there any specific experiments, ablations, or investigations that the authors plan to conduct in the future with the benchmark, beyond the preliminary baseline models presented in this paper?

Author response:
> Are there any specific experiments, ablations, or investigations that the authors plan to conduct in the future with the benchmark, beyond the preliminary baseline models presented in this paper?

As we touched on in response to your concerns in Weaknesses #1, our thorough investigation has provided meaningful open problems (in the context of foundational ML research and bioacoustics-related fields) and future work within this setting, as we have outlined in Section 6: Conclusion. Our group continues to explore problems within this space and are particularly interested in building self-supervised approaches that leverage large quantities of unlabeled and in-domain data.

Reviewer comment (Evaluation):
The evaluation benchmark retrieves relevant samples from a downstream dataset given a query, and it allows reusing class embeddings for queries learned during training. I'm not sure to get this point. If the label of a query is known, then what is the point of retrieving samples from a dataset? We already know the answer.

Author response:
Thank you for your feedback.

You are correct that we evaluate the learned representation's ability to retrieve relevant audio samples given a query. We simulate a setting in which a practitioner wants to retrieve all audio clips in which a species of interest is heard in a downstream dataset of audio clips. We repeat the experiment for multiple species, collecting the ROC AUC metric each time, and then aggregating the ROC AUCs across species.

Let us assume that the practitioner wants to retrieve audio clips for a particular bird species. They need something to compare the downstream dataset's audio clips against to rank them and hopefully surface the most relevant at the top. Usually this takes the form of one or a few example audio clips in which the bird species is heard. For BIRB we have a portion of Xeno-Canto that is held out for that purpose: it contains, for each species we evaluate on, **annotated** audio clips in which that species is heard. Those are the annotated examples we draw from to form the search query for the species of interest, and they are disjoint from both the training data *and* the downstream datasets.

If the particular bird species the practitioner is searching for in the downstream dataset was part of the training data, they have another option: the audio representation was trained to solve a multilabel binary classification problem and it has learned one output weight vector per species in the upstream dataset. Rather than drawing examples from the held-out portion of Xeno-Canto to form a search query, we can use the learned weight vector as our basis for comparison to rank downstream audio clips.

In summary: the labels for the downstream audio clips are not known, they are predicted (as you rightfully pointed out); what *is* known is the labels for a *disjoint* set of audio recordings that are drawn from to form the search query. If you are familiar with few-shot learning nomenclature, this is analogous to the way in which a test episode's support set is labeled but its query set is unlabeled (although to avoid any possible confusion, we point out that the "query" in the context of retrieval is a different concept from the "query set" in the context of few-shot learning: "search query" would map to "support set", and "downstream dataset whose audio clips are to be ranked" would map to "query set").

We hope this clarifies your question and remain available to clarify further if needed.

Reviewer comment (Experiments):
Regardless, it would be nice to report results where none of the class embeddings from training time is used when evaluating the models. Because, one of the generalization aspects posed by the datasets is domain-shift, and it is not clear how to measure domain-shift when query embeddings come from training dataset (rather then being embedded at test time using downstream dataset images).

Author response:
Dear Reviewer SRto: we believe we addressed your concerns regarding the soundness of the evaluation protocol which you identified as your main concerns. Do you have any follow-up questions or concerns? If not, would you be willing to reconsider your score? 

Regarding your comment `It would be nice to report results where none of the class embeddings from training time is used`, for the setting of the Artificially Rare task, we did compare using learned weights from the output layer of the pretrained model against using class embeddings as the 'class representatives'. If that is what you had been referring to, then we would like to direct your attention to these results, shown in Figure 2 and the discussion in the 'Artificially Rare Species' paragraph in Section 4.3. 

In summary, we compared using exemplars vs. the learned representation–from our results, each model we evaluated benefits from taking advantage of the learned representation when the species was available during embedding model training, providing improved performance over using the exemplars as the query. However, depending on the application, it’s not always possible to rely on a learned representation being available (such as for regions or species not reflected in the embedding model training data).

Reviewer comment (Evaluation):
Also, when measuring label shift, what is the semantic relation (due to their granularity) between seen and unseen bird classes?

Author response:
Is your question in regards to a specific setting that we evaluate, as in Section 4.3 Generalization Results?
Overall, the consideration of semantic relation is irrelevant/independent when measuring label shift. It is, however, relevant when considering performance on low resource classes (those with minimal representation at training), as in the case of the Artificially Rare task and matters when considering generalization to such classes. Additionally,
1. The ablation study includes all classes observed ("seen") at training time and investigates the impacts of label shift.
2. There is a component of label shift in the Artificially Rare task, and all bird classes within that collection are included at training time.
3. In the "Performance in mixed conditions" task, there is a combination of bird classes included at training time and those which are not present due to overlap with the Heldout Region sets.

Reviewer comment (Experiments):
Is it possible to evaluate domain-shift while fixing label-shift? For instance, seen and unseen classes being equal, whereas recordings being focal vs passive.

Author response:
Absolutely! Refer to Figure 3: the "evaluation (label-shifted)" dataset is drawn from the same XC dataset as the training set, and therefore is not subject to domain shift, while the "Pennsylvania, USA" dataset is subject to domain shift (focal to passive). The "evaluation (label-shifted)" dataset's marginal class distribution is constructed specifically so as to match the marginal class distribution of the Pennsylvania dataset.
Comparing the two allows us to assess the residual effect of domain shift while controlling for label shift.
Note that it's not feasible to instead subsample the Pennsylvania dataset so that its marginal class distribution matches that of XC for two reasons: 1) some classes have a zero marginal probability in the Pennsylvania dataset, and 2) the Pennsylvania dataset is considerably smaller than XC, and evaluation noise would be unacceptably large if we were to subsample it.

Reviewer comment (Writing):
Given that this is a benchmark paper, and that not everybody is super familiar with the domain (bioacoustics), I would expect a more direct and clear explanation of the task being solved, the types of input given to the network, etc. In that sense the paper is not easy to understand.

Author response:
Thank you for this feedback, we agree that we want to provide a helpful "table setting" for folks who aren't familiar with bioacoustics and provided overview in the Introduction and more in-depth explanation into this setting in the Appendix A.1 Why Avian Bioacoustics and A.2 Related Work. While we do describe in the main paper the base model training and inputs, along with the task description and breakdown thoroughly in the main paper, we can imagine that there would be value in having a high-level summary of all the components in either a figure, table, or brief highlight. Do you think that would help assure the reader has a clear sense of the elements of this framework? We would like to update the main paper to provide a bit more clarity for the reader.