bacpipe 1.2.0
Use bacpipe to streamline the process of generating embeddings and analysing your PAM datasets.
Requires Python: >=3.11

# Welcome to **bacpipe** (**B**io**A**coustic **C**ollection **Pipe**line)

[![Documentation Status](https://readthedocs.org/projects/bacpipe/badge/?version=latest)](https://bacpipe.readthedocs.io/en/latest/?badge=latest)

![](src/bacpipe_logo.png)
image by Nicole Allison

**bacpipe** makes using deep learning models for bioacoustics easy!
Using **bacpipe** you can generate embeddings, classification predictions and clusters. All you need is your audio data and to customize the settings.

And the best part is, it comes with a GUI for you to explore the results.

**bacpipe** also ties in nicely with [acodet](https://github.com/vskode/acodet), allowing you to generate heatmaps of species activity from your datasets based on predictions of deep learning models. But keep in mind predictions are only as good as the model's you are using - they can seem confident but still be very wrong - **bacpipe**'s aim is to make model evaluations easier and let us improve them.

**bacpipe** is also available on pip: `pip install bacpipe`

```python
import bacpipe

# This will execute the whole pipeline
# if nothing is specified it will generate embeddings on
# a set of audio test data using the models birdnet and perch

bacpipe.play()
```
A more detailed description of the API can be found under [API](#api). Full documentation can be found at [https://bacpipe.readthedocs.io](https://bacpipe.readthedocs.io).

There is a [video tutorial](https://www.youtube.com/watch?v=kw713jF5ts8) available on youtube to install and run bacpipe.

## 📚 Table of Contents

- [How it works](#how-it-works)
- [API](#api)
    - [Use bacpipe immediately on the integrated test data](#use-bacpipe-immediately-on-the-integrated-test-data)
    - [Modify configurations and settings as attributs](#modify-configurations-and-settings-as-attributs)
    - [Modify audio source path, models, device](#modify-audio-source-path-models-device)
    - [Use bacpipe in an existing pipeline](#use-bacpipe-in-an-existing-pipeline)
    - [Produce embeddings for multiple models in your own pipeline](#produce-embeddings-for-multiple-models-in-your-own-pipeline)
- [Dashboard visualization](#dashboard-visualization)
- [Installation](#installation)
    - [Install prerequisites](#install-prerequisites)
    - [Clone the git repository](#clone-the-git-repository)
    - [Install the dependencies once the prerequisites are satisfied](#install-the-dependencies-once-the-prerequisites-are-satisfied)
    - [Model checkpoints are downloaded automatically](#model-checkpoints-are-downloaded-automatically)
    - [Test the installation was successful](#test-the-installation-was-successful)
- [Usage](#usage)
    - [Configurations and settings](#configurations-and-settings)
    - [Running the pipeline](#running-the-pipeline)
    - [Model selection](#model-selection)
    - [Dimensionality reduction](#dimensionality-reduction)
    - [Dashboard](#dashboard)
    - [Evaluation](#evaluation)
    - [Models with classifiers](#models-with-classifiers)
    - [Generated Files](#generated-files)
        - [Embedding Folders](#embedding-folders)
        - [Dimensionality reduced embedding folders](#dimensionality-reduced-embedding-folders)
        - [Evaluation folders](#evaluation-folders)
        - [Example result files structure](#example-result-files-structure)
- [Available models](#available-models)
- [Contribute](#contribute)
- [Known issues](#known-issues)
- [Citation](#citation)
- [Newsletter and Q&A sessions](#newsletter-and-qa-sessions)


---
<!-- <details> -->
<!-- <summary><b style="font-size: 2em;">How it works</b></summary> -->
# How it works



### This repository aims to streamline the generation and evaluation of embeddings using a large variety of bioacoustic models.

The below image shows a comparison of umap embeddings based on 15 different bioacoustic models. The models are being evaluated on a bird and frog dataset (more details in [this conference paper](https://arxiv.org/abs/2504.06710)). 

![](src/normal_overview.png)

**bacpipe** requires a dataset of audio files, runs them through a series of models, and generates embeddings. These embeddings can then be used visualized and evaluated for various tasks such as clustering or classification.

By default the embeddings will be generated for the models specified in the [config.yaml](bacpipe/config.yaml) file. 

Currently these bioacoustic models are supported (more details below):
```yaml

available_models : [
    "audiomae"
    "audioprotopnet"
    "avesecho_passt"
    "aves_especies"
    "beats"
    "birdaves_especies"
    "biolingual"
    "birdnet"
    "birdmae"
    "convnext_birdset"
    "hbdet"
    "insect66"
    "insect459"
    "mix2"
    "naturebeats"
    "perch_bird"
    "pervh_v2"
    "protoclr"
    "rcl_fs_bsed"
    "surfperch"
    "google_whale"
    "vggish"
  ]
```
Once the embeddings are generated, 2d reduced embeddings will be created using the dimensionality reduction model specified in the [config.yaml](bacpipe/config.yaml) file. 
And these dimensionality reduction models are supported:

```yaml
available_reduction_models: [
  "pca",
  "sparse_pca",
  "t_sne",
  "umap"
]
```

Furthermore, the embeddings can be evaluated using different metrics. The evaluation is done using the `evaluate.py` script, which takes the generated embeddings and computes various metrics such as clustering performance and classification performance. The evaluation results are saved in the `bacpipe/results` directory.

```yaml

available_evaluation_tasks: [
  "classification",
  "clustering"
]
```


The repository also includes a panel dashboard for visualizing the generated embeddings. To enable the dashboard, simply set the `dashboard` variable to `True` in the [settings.yaml](bacpipe/settings.yaml) file. The dashboard will automatically open in your browser (at `http://localhost:8050`) after running the `run_dashboard.py` script.

The pipeline is designed to be modular, so you can easily add or remove models as needed. The models are organized into pipelines, which are defined in the `bacpipe/embedding_generation_pipelines/feature_extractors` directory. If you want to add a different dimensionality reduction model, you do so by adding new pipeline to the `bacpipe/embedding_generation_pipelines/dimensionality_reduction` directory.

## Using annotations for evaluation

If you have annotations for your dataset, you can use them to evaluate the generated embeddings. The labels will be used to compute the clustering and classification performance of the embeddings. 

To use the annotations for evaluation, create a file called `annotations.csv` in the directory specified in the `audio_dir` variable in the [config.yaml](bacpipe/config.yaml) file. The file should contain the following columns:
```csv
audiofilename,start,end,label
```
Where `audiofilename` is the name of the audio file, `start` and `end` are the start and end times of the annotation in seconds, and `label` is the label of the annotation.

For reference see the [example annotations file](bacpipe/tests/test_data/annotations.csv).

If this file exists, the evaluation script will automatically use the annotations to compute the clustering and classification performance of the embeddings. The labels will also be used to color the points in the dashboard visualization showing the embeddings.

# API

**bacpipe** can be used as a package and installed from pip

`pip install bacpipe`

## Use bacpipe immediately on the integrated test data

```python
import bacpipe

# This will execute the whole pipeline
# if nothing is specified it will generate embeddings on
# a set of audio test data using the models birdnet and perch

bacpipe.play()
```

## Modify configurations and settings as attributs
To modify configurations and settings, you can simply access them as attributes. To see available settings and configs run the following commands

```python
bacpipe.config
bacpipe.settings
# you can also check the bacpipe/config.yaml and bacpipe/settings.yaml
# files here in the repository to see all the available settings and
# read their respective description
```

## Modify audio source path, models, device
If you're on a Windows machine, make sure to add a `r` before the path like `r'path\to\audio'` otherwise the path will likely cause problems due to the backslashes. 
```python
# to modify the audio data path for example, do
bacpipe.config.audio_dir = '/path/to/your/audio/dir'

# to modify the models you want to run, do
bacpipe.config.models = ['birdnet', 'birdmae', 'naturebeats']
# if you do not have the checkpoint yet, it will be automatically 
# downloaded and stored locally

bacpipe.settings.device = 'cuda' 
# bacpipe uses multithreading which speeds up model inference if 
# run on a machine supporting cuda

# then run with your settings.
# By default the save logs is True
bacpipe.play(save_logs=True)
# That way bacpipe will generate log files of the outputs and also save your
# config and settings files, which can be helpful in retrospect to remember
# all the settings you chose for a run. 

```

## Use bacpipe in an existing pipeline
If you just want to run models and get embeddings returned without saving them and don't want the dashboard and all of that, define an embedder object and pass it the model name and the settings you modified.

```python
em = bacpipe.Embedder('perch_bird', **vars(bacpipe.settings)) 
# the vars part is important!

audio_file = '/path/to/all/the/audio/file'
embeddings = em.get_embeddings_from_model(audio_file)

# if the model has a built in classifier, like birdnet, the classification
# score are also saved. You can check if there is a classifier included by 
# checking 
em.model.bool_classifier

# After the generating of embeddings above, you will then be able to access 
# the class predictions using
em.model.classifier_outputs
```

## Produce embeddings for multiple models in your own pipeline
If you want to produce embeddings for multiple models, bacpipe will always store them to keep your memory from overfilling. Still you can use the package to easily access the embeddings and all the metadata

```python
loader = bacpipe.model_specific_embedding_creation(
  **vars(bacpipe.config), **vars(bacpipe.settings)
  )
# this call will initiate the embedding generation process, it will check if embeddings
# already exist for the combination of each model and the dataset and if so it will
# be ready to load them. The loader keys will be the model name and the values will
# be the loader objects for each model. Each object contains all the information
# on the generated embeddings. To name access them:
loader['birdnet'].embedding_dict() 
# this will give you a dictionary with the keys corresponding to embedding files
# and the values corresponding to the embeddings as numpy arrays

loader['birdnet'].metadata_dict
# This will give you a dictionary overview of:
# - where the audio data came from,
# - where the embeddings were saved
# - all the audio files, 
# - the embedding size of the model, 
# - the audio file lengths,
# - the number of embeddings for each audio files
# - the sample rate
# - the number of samples per window
# - and the total length of the processed dataset in seconds
# Thic dictionary is also saved as a yaml file in the directory of the embeddings
```


# Dashboard visualization

### bacpipe includes a dashboard visualization by default allowing you to easily explore the generated embeddings

Once embeddings are generated, they can be easily visualized using a dashboard (built using `panel`) by simply setting the `dashboard` setting in the [config.yaml](bacpipe/config.yaml) file to `True`.

Below you can see a gif showing the basic usage of the dashboard.

![](src/bacpipe_demo.gif)

The dashboard has 3 main sections:
1. Single model
2. Two models
3. All models

In the single model section, you can select a model and visualize the embeddings generated by that model. The embeddings can be colored by :
- metadata extracted from the files (date and time information, and file and parent directory) 
- the labels specified in the `annotations.csv` file
- the cluster labels generated by the clustering algorithm (kmeans)

In the dashboard sidebar you can select the model, by which to label the embeddings, whether to remove noise, and the type of classification task to show the results for. 

The noise removal is done by removing the embeddings that do not correspond to annotated sections of the audio files. This is useful if you want to focus on the annotated sections of the audio files and disregard the rest of the data. 

The visualizations can be saved as png files by clicking the save button in the bottom right corner of the plot.

__Try it out__ and (__please__) feel free to give feedback and ask questions (or suggestions for improvements) - or in case something does not work raise issues.




---
# Installation

## Install prerequisites
<details>
<summary> Install python 3.11, uv and git <b>(click to expand)</b> </summary> 

### Install Python 3.11 and git on your local computer

For **Windows**:
- Download python 3.11: https://www.python.org/ftp/python/3.11.9/python-3.11.9-amd64.exe
- Download Git: https://github.com/git-for-windows/git/releases/download/v2.51.0.windows.1/Git-2.51.0-64-bit.exe

For **Linux**
- `sudo add-apt-repository ppa:deadsnakes/ppa`
- `sudo apt install python3.11`
- `sudo apt install git`

For **Mac**
- (install homebrew: `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" `
- install git: `brew install git`


### Create a virtual environment for this project

Virtual environments are very important. They ensure that specific libraries that are needed for one project don't get in the way of libraries you need for another project.

Create your virtual environment (all systems):
- `uv venv --python 3.11 env_acodet`

For **Windows**, Activate your environment:
- `source env_acodet/Scripts/activate`

For **Linux**/**Mac**, Activate your environment:
- `source env_acodet/bin/activate`

Install the project dependencies (all systems):
- `uv pip install -r pyproject.toml`


## Install `uv` (recommended) or `poetry`

It is recommended to use python 3.11 for this repository, as some of the models require it. 

For speed and stability it is recommended to use `uv`. To install `uv` use the following command (you can also do it all without `uv` then, just leave the `uv` part away, as all the commands are also `pip` commands):

`pip install uv` 

(for windows use `$HOME/AppData/Local/Programs/Python/Python311/python.exe -m pip install uv`)

If you prefer to use `poetry`, you can install it using: 

`pipx install poetry` 

### Create a virtual environment

`python3.11 -m uv venv .env_bacpipe`

(for windows use `$HOME/AppData/Local/Programs/Python/Python311/python.exe -m uv venv .env_bacpipe`)

(alternatively for `poetry` use `poetry env use 3.11`)

activate the environment

`source .env_bacpipe/bin/activate` (for windows use `source .env_bacpipe\Scripts\activate`)

</details>


## Clone the git repository

For **Windows**:
- move to a folder of your choice (choose wisely - something like `Documents` is always a good starting point), then right click and `Open Git Bash here`
- install **uv**: `"$HOME/AppData/Local/Programs/Python/Python311/python" -m pip install uv`

For **Linux** / **MAC**:
- open a terminal (console) in the folder of your choice
- install **uv**: `/usr/bin/python/Python311/python -m pip install uv`
- (if Mac users get an error locate python with `which python3` and use that path instead followed by `-m pip install uv`)

For all systems:

- run `git clone https://github.com/bioacoustic-ai/bacpipe`


## Install the dependencies once the prerequisites are satisfied.

`uv pip install -r pyproject.toml`

- this will automatically install requirements based on your os, so windows should also work fine. However, gpu support is not available on windows

For `poetry`:

`poetry lock`

`poetry install`

If you do not have admin rights and encounter a `permission denied` error when using `pip install`, use `python -m pip install ...` instead.


### Install gpu support for tensorflow

Because of the requirements of `torch==2.6` the cuda versions have to be installed corresponding to what pytorch supports. However, I have tested that you are able to install different cuda dependencies to also support tensorflow gpus once the environment is set up. Once you have installed the requirements. Install the following dependencies using 
`uv pip install -r requirements_tf_gpu.txt`.

```python
nvidia-cublas-cu12==12.5.2.13
nvidia-cuda-cccl-cu12==12.5.39.post1
nvidia-cuda-cupti-cu12==12.5.39
nvidia-cuda-cuxxfilt-cu12==12.5.39
nvidia-cuda-nvcc-cu12==12.5.40
nvidia-cuda-nvrtc-cu12==12.5.40
nvidia-cuda-opencl-cu12==12.5.39
nvidia-cuda-profiler-api-cu12==12.5.39
nvidia-cuda-runtime-cu12==12.5.39
nvidia-cuda-sanitizer-api-cu12==12.5.39
nvidia-cudnn-cu12==9.3.0.75
nvidia-cufft-cu12==11.2.3.18
nvidia-curand-cu12==10.3.6.39
nvidia-cusolver-cu12==11.6.2.40
nvidia-cusparse-cu12==12.4.1.24
nvidia-cusparselt-cu12==0.6.2
nvidia-nccl-cu12==2.21.5
nvidia-npp-cu12==12.3.0.116
nvidia-nvfatbin-cu12==12.5.39
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvjpeg-cu12==12.3.2.38
nvidia-nvml-dev-cu12==12.5.39
nvidia-nvtx-cu12==12.5.39
```

## Model checkpoints are downloaded automatically. 

Model checkpoints will be downloaded automatically. Once you run `bacpipe.play()`, it will automatically download models that were included but are not yet available locally. Models are downloaded from [this huggingface repo](https://huggingface.co/datasets/vskode/bacpipe_models/tree/main).

## Test the installation was successful

By doing so you will also ensure that the directory structure for the model checkpoints will be created. 

`pytest -v --disable-warnings bacpipe/tests/test_embedding_creation.py`

The tests could take a while, so to run a small test, you can also pass the model you would like to test:

`pytest -v --disable-warnings bacpipe/tests/test_embedding_creation.py --models=birdnet,perch`

(keep in mind you have to have the checkpoints locally for the models that require it)

In case of a permission denied error, run
`python -m pytest -v --disable-warnings bacpipe/tests/test_embedding_creation.py`

If everything passes then you've successfully installed bacpipe and can now proceed to use it.

# Usage

## Configurations and settings

To see the capabilities of bacpipe, go ahead and run the `run_pipeline.py` script. This will run the pipeline with the default settings and configurations on a small set of test data. 

### To use bacpipe on your own data, you will need to modify the configuration files.

The only two files that need to be modified are the [config.yaml](bacpipe/config.yaml) and [settings.yaml](bacpipe/settings.yaml) files. The [config.yaml](bacpipe/config.yaml) is used for the standard configurations: 
- path to audio files
- models to run
- dimensionality reduction model
- evaluation tasks
- whether to run the dashboard or not

The [settings.yaml](bacpipe/settings.yaml) file is used for more advanced configurations and does not need to be modified unless you have specific preferences. It includes settings such as to run on a `cpu` or a `cuda` (gpu) (by default `cpu`), the paths where results are saved, configurations for the evaluation tasks and more. 

Modify the [config.yaml](bacpipe/config.yaml) file in the root directory to specify the path to your `dataset`. Define what models to run by specifying the strings in the `models` list (copy and paste as needed, I usually just comment the model's I don't want to run). 

If you have already computed embeddings on the dataset specified in audio_data, and you want to do the dimensionality reduction and evaluation for the models you have already run, you can set the `already_computed` variable to `True`. This will only select the models that have already been computed.

In either case if you have already computed embeddings with a model, **bacpipe** will skip the model and use the **already computed** embeddings (if they are still located in the same directory). Even if `overwrite` is set to `True`, **bacpipe** will not overwrite the embeddings if they already exist. It will recompute clusterings and label generation.

## Running the pipeline

Once the configuration is complete, execute the run_pipeline.py file (make sure the environment is activated)
`python run_pipeline.py`

While the scripts are executed, directories will be created corresponding to the `main_results_dir` setting. Embeddings will be saved in `main_results_dir/YOUR_DATASET/embeddings` (see [here](results/test_data/embeddings/README.md) for more info) and if selected, reduced dimensionality embeddings will be saved in `main_results_dir/evaluation/dim_reduced_embeddings` (see [here](results/test_data/dim_reduced_embeddings/README.md) for more info).


## Model selection

Select the models you want to run in the [config.yaml](bacpipe/config.yaml) file. The models are specified in this ReadMe and in the [test_file](bacpipe/tests/test_embedding_creation.py). You can select the models you want to run by adding them to the `models` list in the [config.yaml](bacpipe/config.yaml) file.

## Dimensionality reduction

Different dimensionality reduction models can be selected in the [config.yaml](bacpipe/config.yaml) file. The available models are specified in the section [Dimensionality reduction models](#dimensionality-reduction-models). Insert the name of the selected model in the `dim_reduction_model` variable in the [config.yaml](bacpipe/config.yaml) file. The default is `umap`, but you can also select `pca`, `sparse_pca` or `t_sne`.


## Dashboard

The dashboard is a panel application that allows you to visualize the generated embeddings. To enable the dashboard, set the `dashboard` variable in the [config.yaml](bacpipe/config.yaml) file to `True`. The dashboard will automatically open in your browser (at `http://localhost:8050`) after running the `run_dashboard.py` script.

## Evaluation

You can use `bacpipe` to evaluate the generated embeddings using different metrics. To evaluate the embeddings, you need annotations for your dataset. The annotations should be in a file called `annotations.csv` in the directory specified in the `audio_dir` variable in the [config.yaml](bacpipe/config.yaml) file or the results directory of your dataset `main_results_dir/YOUR_DATASET`. The file should contain the following columns:
```csv
audiofilename,start,end,label:species
```

Where `audiofilename` is the name of the audio file, `start` and `end` are the start and end times of the annotation in seconds, and `label` is the label of the annotation.

Using the settings attribute `only_embed_annotations`, you can also decide to only generate embedding corresponding to your annotations. In that case bacpipe will take each annotation and create embeddings for each selected model from only those segments. If segments are shorter than the model input length, the segments will be padded. If they are longer, they will produce several embeddings.

`species` is a placeholder here and can be replaced with any label description. So if you have labelled call types, change it to `label:call_type`. But it's important that there are no spaces and that it contains `label:`. By doing this you will be able to visualize your data based on all of these label columns.

The labels can then be used to perform clustering and classification evaluation. This can be done only in regard to one label, so specify the main label column in the `label_column` variable in [settings.yaml](bacpipe/settings.yaml). This defaults to `species`. Only labels that exceed the `min_label_occurrences` value will be used. This is to make sure you have enough data to train linear classifiers and do meaningful evaluations. If you have enough labeled data, feel free to increase this. 

See the file [annotations.csv](bacpipe/tests/test_data/annotations.csv) for an example of how the annotations file should look like.

Once the annotations file is created, add either `classification` or `clustering` (or both) to the `evaluation_task` variable in the [config.yaml](bacpipe/config.yaml) file (use double quotes: "classification" or "clustering"). You can run the evaluation script using normal `python run_pipeline.py` command. The evaluation script will automatically use the annotations to compute the clustering and classification performance of the embeddings. The results will be saved in the `bacpipe/results/YOUR_DATASET/evaluation` directory.

If you selected classification, a linear classifier will be trained and saved in the classification subdirectory of the evaluation folder. This .pt file can be used to generate class predictions with a model that wasn't originally trained on these classes. A tutorial will be available shortly explaining this in more detail. The .pt file can be used in the repository [acodet](https://github.com/vskode/acodet) to generate class predictions with the combination of a feature extractor and the trained linear classifier.

## Models with classifiers

Models that already contain classification heads, are the following:
- Perch_v2
- BirdNET
- Perch_bird
- SurfPerch
- AudioProtoPNet
- ConvNeXT_birdset
- google_whale

With all of these models, you only need to set `run_pretrained_classifier` to True and then the model will save the classification outputs in the `classification/original_classifier_outputs` folder. Only predictions exceeding the `classifier_threshold` value will be saved. A csv file in the shape of the annotations.csv file is also saved corresponding to the class predictions. The dashboard will also contain an extra `label_by` option `default_classifier`.

## Generated Files

When processing `bacpipe` generates a number of files. It will firstly create the `main_results_dir` specified in [settings.yaml](bacpipe/settings.yaml) and within that it will create a folder named like the dataset selected in `audio_dir` in [config.yaml](bacpipe/config.yaml). Within this directory, bacpipe will create 3 directories: `embed_parent_dir`, `dim_reduc_parent_dir` and `evaluations_dir`. 

The `embed_parent_dir` will contain the model-specific folders with the timestamps when they were processed. The `dim_reduc_parent_dir` contains the model-specific and dimensionality reduction-specific folder again with a timestamp when they were processed. And the `evaluations_dir` contains folders for each model that has been processed. 

### Embedding folders

By default the naming conventions for the embedding directories is:

`_year-month-day_hour-minute___modelname_datasetname`

The `metadata.yaml` file that will be created in each of the embedding folders has the following structure:

Inside this directory you will find:
- `metadata.yml`, which contains metadata in the shape of a dictionary with the following keys:
    - audio_dir: _data_directory_
    - embed_dir: _path_to_embedding_files_
    - embedding_size: _dimension_of_embeddings_
    - files: _dictionary_with_lists_containing_per_file_information_
        - audio_files: _name_of_audio_files_
        - file_lengths (s): _file_length_in_seconds_
    - model_name: _name_of_model_
    - sample_rate (Hz): _sample_rate_
    - segment_length (samples): _length_of_input_segment_in_samples_
    - total_dataset_length (s): _total_length_of_dataset_in_seconds_
- either the embeddings files (ending on `.npy`) or directories corresponding to subdirectories in the dataset folder


It is important that the name of this directory remains unchanged, so that it can be found automatically. 


### Dimensionality reduced embedding folders

By default the naming conventions for the embedding directories is:

`_year-month-day_hour-minute___DimReductionModelName-DatasetName-ModelName`

Inside this directory you will find:
- `metadata.yml`, which contains metadata in the shape of a dictionary with the following keys (all of the metadata from the embeddings is also repeated):
    - files: _dictionary_with_lists_containing_per_file_information_
        - embedding_dimensions: _tuple_of_number_of_embeddings_by_embedding_size_
        - embedding_files: _name_of_embedding_files_
        - nr_embeds_per_file: _number_of_embeddings_per_file_
    - nr_embeds_total: _total_number_of_embeddings_
- one `.json` file containing the reduced embeddings of the entire dataset
- a plot, visiualizing the reduced embeddings in 2d

It is important that the name of this directory remains unchanged, so that it can be found automatically. 

### Evaluation folders

Within the `evaluations_dir` folder, you will find the following folders: `classification`, `clustering`, `distances`, `labels` and `plots`. These folders will be filled with results if the corresponding evaluation tasks are selected. `distanctes` is currently not supported and therefore only a place-holder. `label` will contain a `.npy` file containing the auto-generated labels from the metadata and if available ground_truth.

### Pretrained classifier annotations

If the model has a pretrained classifier and `run_pretrained_classifier` is `True`, `bacpipe` will create a `original_classifier_output` folder and a `default_classifier_annotations.csv` file, corresponding to annotations from the pretrained classifier. The folder will contain `.json` files for each audio file with the species and in which time bin they occurred with what certainty. The `default_classifier_annotations.csv` file contains annotations in the same style as the [annotations.csv](bacpipe/tests/test_data/annotations.csv) file.


### Example result files structure

Generated from running with the test_data (timestamps will change) with only directories shown.

This is the source audio data structure:
```.
└── audio
    ├── FewShot
    └── UrbanSoundscape
```

This is the resulting folder structure:

```
.
├── dim_reduced_embeddings
│   ├── 2025-09-09_03-08___umap-test_data-birdnet
│   └── 2025-09-09_03-09___umap-test_data-perch_bird
├── embeddings
│   ├── 2025-09-09_03-08___birdnet-test_data
│   │   └── audio
│   │       ├── FewShot
│   │       └── UrbanSoundscape
│   └── 2025-09-09_03-08___perch_bird-test_data
│       └── audio
│           ├── FewShot
│           └── UrbanSoundscape
└── evaluations
    ├── birdnet
    │   ├── classification
    │   │   └── original_classifier_outputs
    │   │       └── audio
    │   │           ├── FewShot
    │   │           └── UrbanSoundscape
    │   ├── clustering
    │   ├── distances
    │   ├── labels
    │   └── plots
    ├── overview
    └── perch_bird
        ├── classification
        │   └── original_classifier_outputs
        │       └── audio
        │           ├── FewShot
        │           └── UrbanSoundscape
        ├── clustering
        ├── distances
        ├── labels
        └── plots
```


# Available models

The models all have their model specific code to ensure inference runs smoothly. More info on the models and their pipelines can be found [here](bacpipe/pipelines/README.md).

Models currently include:

|   Name|   ref paper|   ref code|   sampling rate|   input length| embedding dimension |
|---|---|---|---|---|---|
|   AudioMAE    |   [paper](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b89d5e209990b19e33b418e14f323998-Abstract-Conference.html)   |   [code](https://github.com/facebookresearch/AudioMAE)    |   16 kHz|   10 s| 768 |
|   AudioProtoPNet   |   [paper](https://www.sciencedirect.com/science/article/pii/S1574954125000901)   |   [code](https://github.com/DBD-research-group/AudioProtoPNet)    |   32 kHz|   5 s| 1024 |
|   AvesEcho_PASST   |   [paper](https://arxiv.org/abs/2409.15383)   |   [code](https://gitlab.com/arise-biodiversity/DSI/algorithms/avesecho-v1)    |   32 kHz|   3 s| 768 |
|   AVES_ESpecies        |   [paper](https://arxiv.org/abs/2210.14493)   |   [code](https://github.com/earthspecies/aves)    |   16 kHz|   1 s| 768 |
| BEATs | [paper](https://arxiv.org/abs/2212.09058) | [code](https://github.com/microsoft/unilm/tree/master/beats) | 16 kHz | 10 s | 768 |
|   BioLingual  |   [paper](https://arxiv.org/abs/2308.04978)   |   [code](https://github.com/david-rx/biolingual)    |   48 kHz|   10 s| 512 |
|   BirdAVES_ESpecies    |   [paper](https://arxiv.org/abs/2210.14493)   |   [code](https://github.com/earthspecies/aves)    |   16 kHz|   1 s| 1024 |
|   BirdMAE    |   [paper](https://arxiv.org/abs/2504.12880)   |   [code](https://github.com/DBD-research-group/Bird-MAE)    |   32 kHz|   10 s| 1280 |
|   BirdNET     |   [paper](https://www.sciencedirect.com/science/article/pii/S1574954121000273)   |   [code](https://github.com/kahst/BirdNET-Analyzer)    |   48 kHz|   3 s| 1024 |
|   ConveNeXT_BirdSet   |   [paper](https://arxiv.org/abs/2504.12880)   |   [code](https://github.com/DBD-research-group/BirdSet)    |   32 kHz|   5 s| 1024 |
|   Google_Whale       |   paper   |   [code](https://www.kaggle.com/models/google/multispecies-whale/TensorFlow2/default/2)    |   24 kHz|   5 s| 1280 |
|   HumpbackNET |   [paper](https://pubs.aip.org/asa/jasa/article/155/3/2050/3271347)   |   [code](https://github.com/vskode/acodet)    |   2 kHz|   3.9124 s| 2048|
|   Insect66NET |   [paper](https://doi.org/10.1371/journal.pcbi.1011541)   |   [code](https://github.com/danstowell/insect_classifier_GDSC23_insecteffnet)    |   44.1 kHz|   5.5 s| 1280 |
|   Insect459NET |   [paper](https://arxiv.org/pdf/2503.15074)   |   pending    |   44.1 kHz|   5.5 s| 1280 |
|   Mix2        |   [paper](https://arxiv.org/abs/2403.09598)   |   [code](https://github.com/ilyassmoummad/Mix2/tree/main)    |   16 kHz|   3 s| 960 |
|   NatureBEATs        |   [paper](https://arxiv.org/abs/2411.07186)   |   [code](https://github.com/earthspecies/NatureLM-audio)    |   16 kHz|   10 s| 768 |
|   Perch_Bird       |   [paper](https://www.nature.com/articles/s41598-023-49989-z.epdf)   |   [code](https://github.com/google-research/perch)    |   32 kHz|   5 s| 1280 |
|   Perch_V2       |   [paper](https://arxiv.org/abs/2508.04665)   |   [code](https://github.com/google-research/perch_hoplite)    |   32 kHz|   5 s| 1536 |
|   ProtoCLR     |   [paper](https://arxiv.org/pdf/2409.08589)   |   [code](https://github.com/ilyassmoummad/ProtoCLR)    |   16 kHz|   6 s| 384 |
|   RCL_FS_BSED     |   [paper](https://arxiv.org/abs/2309.08971)   |   [code](https://github.com/ilyassmoummad/RCL_FS_BSED)    |   22.05 kHz|   0.2 s| 2048 |
|   SurfPerch       |   [paper](https://arxiv.org/abs/2404.16436)   |   [code](https://www.kaggle.com/models/google/surfperch)    |   32 kHz|   5 s| 1280 |
|   VGGish      |   [paper](https://ieeexplore.ieee.org/document/7952132)   |   [code](https://github.com/tensorflow/models/tree/master/research/audioset/vggish)    |   16 kHz|   0.96 s| 128 |

<details>
<summary>Click to see more details on the models</summary>


|   Name|   paper|   code|   training|   CNN/Trafo| architecture | checkpoint link |
|---|---|---|---|---|---|---|
|   [AudioMAE](#audiomae)    |   [paper](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b89d5e209990b19e33b418e14f323998-Abstract-Conference.html)   |   [code](https://github.com/facebookresearch/AudioMAE)    | ssl + ft|   trafo| ViT | [weights](https://drive.google.com/file/d/18EsFOyZYvBYHkJ7_n7JFFWbj6crz01gq/view)|
|   [AudioProtoPNet](#audioprotopnet)   |   [paper](https://www.sciencedirect.com/science/article/pii/S1574954125000901)   |   [code](https://github.com/DBD-research-group/AudioProtoPNet)    |  sup l |   CNN | ConvNext | included|
|   [AvesEcho_PaSST](#avesecho_passt)   |   [paper](https://arxiv.org/abs/2409.15383)   |   [code](https://gitlab.com/arise-biodiversity/DSI/algorithms/avesecho-v1)    |   sup l |   trafo | PaSST | [weights](https://gitlab.com/arise-biodiversity/DSI/algorithms/avesecho-v1/-/blob/main/checkpoints/best_model_passt.pt?ref_type=heads) |
|   [AVES_ESpecies](#aves_especies)        |   [paper](https://arxiv.org/abs/2210.14493)   |   [code](https://github.com/earthspecies/aves)    |   ssl|   trafo | HuBERT | [weights](https://storage.googleapis.com/esp-public-files/ported_aves/aves-base-all.torchaudio.pt)|
| [BEATs](#beats) | [paper](https://arxiv.org/abs/2212.09058) | [code](https://github.com/microsoft/unilm/tree/master/beats) | ssl | trafo | ViT | [weights](https://1drv.ms/u/s!AqeByhGUtINrgcpoZecQbiXeaUjN8A?e=DasbeC) |
|   [BioLingual](#biolingual)  |   [paper](https://arxiv.org/abs/2308.04978)   |   [code](https://github.com/david-rx/biolingual)    |   ssl|   trafo| CLAP | included |
|   [BirdAVES_ESpecies](#birdaves_especies)    |   [paper](https://arxiv.org/abs/2210.14493)   |   [code](https://github.com/earthspecies/aves)    |   ssl|   trafo | HuBERT | [weights](https://storage.googleapis.com/esp-public-files/birdaves/birdaves-biox-large.torchaudio.pt)|
|   [BirdMAE](#birdmae)    |   [paper](https://arxiv.org/abs/2504.12880)   |   [code](https://github.com/DBD-research-group/Bird-MAE)    |   ssl | trafo | ViT | included |
|   [BirdNET](#birdnet)     |   [paper](https://www.sciencedirect.com/science/article/pii/S1574954121000273)   |   [code](https://github.com/kahst/BirdNET-Analyzer)    |   sup l|   CNN | EffNetB0 | [weights](https://github.com/kahst/BirdNET-Analyzer/tree/main/birdnet_analyzer/checkpoints/V2.4/BirdNET_GLOBAL_6K_V2.4_Model)|
|   [ConvNeXT_BirdSet](#convnext_birdset)   |   [paper](https://arxiv.org/abs/2504.12880)   |   [code](https://github.com/DBD-research-group/BirdSet)    |  sup l |   CNN | ConvNext | included|
|   [Google_Whale](#google_whale)       |   paper   |   [code](https://www.kaggle.com/models/google/multispecies-whale/TensorFlow2/default/2)    |   sup l|   CNN| EffNetb0 | included|
|   [HumpbackNET](#humpbacknet) |   [paper](https://pubs.aip.org/asa/jasa/article/155/3/2050/3271347)   |   [code](https://github.com/vskode/acodet)    |   sup l |   CNN | ResNet50| [weights](https://github.com/vskode/acodet/blob/main/acodet/src/models/Humpback_20221130.zip)|
|   [Insect66NET](#insect66net) |   paper   |   [code](https://github.com/danstowell/insect_classifier_GDSC23_insecteffnet)    |   sup l|   CNN | EffNetv2s | [weights](https://gitlab.com/arise-biodiversity/DSI/algorithms/cricket-cicada-detector-capgemini/-/blob/main/src/model_traced.pt?ref_type=heads)|
|   [Insect459NET](#insect459net) |   paper   |   pending    |   sup l|   CNN | EffNetv2s | pending |
|   [Mix2](#mix2)        |   [paper](https://arxiv.org/abs/2403.09598)   |   [code](https://github.com/ilyassmoummad/Mix2/tree/main)    |   sup l|   CNN| MobNetv3 | release pending|
|   [NatureBEATs](#naturebeats)        |   [paper](https://arxiv.org/abs/2411.07186)   |   [code](https://github.com/earthspecies/NatureLM-audio)    | ssl | trafo | BEATs | [weights](https://drive.google.com/file/d/12BrWRbxJsuwZHOkzX8HEpGgSMy5VnwCp/view?usp=sharing) |
|   [Perch_Bird](#perch_bird)       |   [paper](https://www.nature.com/articles/s41598-023-49989-z.epdf)   |   [code](https://github.com/google-research/perch)    |   sup l|   CNN| EffNetb0 | included |
|   [Perch_V2](#perch_v2)       |   [paper](https://arxiv.org/abs/2508.04665)   |   [code](https://github.com/google-research/perch_hoplite)    |   sup l |   CNN | EfficientNetB3 | included |
|   [ProtoCLR](#protoclr)     |   [paper](https://arxiv.org/pdf/2409.08589)   |   [code](https://github.com/ilyassmoummad/ProtoCLR)    |   sup cl|   trafo| CvT-13 | [weights](https://huggingface.co/ilyassmoummad/ProtoCLR)|
|   [RCL_FS_BSED](#rcl_fs_bsed)     |   [paper](https://arxiv.org/abs/2309.08971)   |   [code](https://github.com/ilyassmoummad/RCL_FS_BSED)