speechx 0.1.0
PyTorch-based audio toolkit
License: MIT
Requires Python: >=3.6

<div align="center">
<img src="docs/source/_static/images/speechx_light.png" width="50%">

**The PyTorch-based audio toolkit for researchers.**

[![PyPI Status](https://badge.fury.io/py/speechx.svg)](https://badge.fury.io/py/speechx)
[![Build Status](https://github.com/Speech-AI/SpeechX/workflows/CI/badge.svg)](https://github.com/Speech-AI/SpeechX/actions?query=workflow%3ACI+branch%3Adevelop+event%3Apush)
[![codecov][codecov-badge]][codecov]
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Documentation Status](https://img.shields.io/badge/docs-0.3.3-blue)](https://speech-ai.github.io/SpeechX/index.html)
[![Latest Docs Status](https://github.com/Speech-AI/SpeechX/workflows/Latest%20docs/badge.svg)](https://Speech-AI.github.io/SpeechX/)


[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Speech-AI/SpeechX/pulls)
[![Python Versions](https://img.shields.io/pypi/pyversions/speechx.svg)](https://pypi.org/project/speechx/)
[![PyPI Status](https://pepy.tech/badge/speechx)](https://pepy.tech/project/speechx)
[![Slack][slack-badge]][slack-invite]

</div>

--------------------------------------------------------------------------------


SpeechX is a Pytorch-based audio toolkit
that enables fast experimentation on common datasets.
It comes with a source code that supports a large range
of datasets and architectures, and a set of
 recipes to reproduce some important papers.


### Join the SpeechX Community
Please, if you have found a bug, [open an issue][issue],
if you solved it, [open a pull request][pr]!
Same goes for new features, tell us what you want or help us building it!
Don't hesitate to [join the slack][slack-invite]
and ask questions / suggest new features there as well!
SpeechX is intended to be a __community-based project__
so hop on and help us!
## Contents
- [Installation](#installation)
- [Quick start](#quick-start)
- [Tutorials](#tutorials)
- [Running a recipe](#running-a-recipe)
- [Available recipes](#available-recipes)
- [Supported datasets](#supported-datasets)
- [Pretrained models](#pretrained-models)
- [Calls for contributions](#contributing)
- [Citing us](#citing)

## Installation
([↑up to contents](#contents))
In order to install SpeechX, clone the repo and install it using
pip or python :
```bash
git clone https://github.com/Speech-AI/SpeechX
cd SpeechX
# Install with pip in editable mode
pip install -e .
# Or, install with python in dev mode
pip install -e .[dev]
```
SpeechX is also on PyPI, you can install the latest release
with
```bash
pip install speechx
```

## Quick start
([↑up to contents](#contents))
If you want to see more concrete examples, please go to [Tutorials](#tutorials)

#### Download and use the models directly
 ```python
 from speechx import TransMask
 model = TransMask.from_pretrained(
   pretrained_model_conf_or_path='deciding/TransMask_Libri2Mix_sepclean'
 )
 model.separate(f, force_overwrite=True, output_dir='results')
 ```
 check what pretrained models are available
 ```python
 from speechx import show_available_models
 show_available_models()
 # Out: 'deciding/TransMask_Libri2Mix_sepclean',
 # 'Cosentino/ConvTasNet_LibriMix_sepclean' ...
 ```
 check what networks are available
 ```python
 from speechx import show_available_networks
 show_available_networks()()
 # Out: TransMask
 # ConvTasNet ...
 ```
#### Datasets
 ```python
 from torch.utils.data import DataLoader
 from speechx.data import LibriMix
 train_set = LibriMix(
   csv_dir='Libri2Mix/',
   task='sep_clean', # enh_single, enh_both, sep_clean, sep_noisy
   sample_rate=16000,
   n_src=2,
   segment=3, # seconds
 )
 train_loader = DataLoader(
   train_set,
   shuffle=True,
   batch_size=6,
   num_workers=8,
   drop_last=True,
 )
 ```
 download Mini LibriMix for fast test
 ```python
 from torch.utils.data import DataLoader
 from speechx.data import LibriMix
train_set, val_set = LibriMix.mini_from_download(task='sep_clean')
train_loader = DataLoader(train_set, batch_size=4, drop_last=True)
val_loader = DataLoader(val_set, batch_size=4, drop_last=True)
 ```
 check what datasets do we have
 ```python
 from speechx.data import show_availables
 show_availables()
 # Out: AVSpeechDataset
 # LibriMix ...
 ```

#### Convert wav to 2D representation and convert the result back to wav
```python
import torch
from speechx.filterbanks import make_enc_dec
encoder, decoder = make_enc_dec('stft', n_filters=256,
                                kernel_size=128, stride=64)
# Waveform-like (number of sources, 1, number of frames)
# p.s. (number of sources, number of frames) is also ok
wav = torch.randn(2, 1, 16000)
# Time-frequency representation
tf_rep = encoder(wav)
# Back to time domain
wav_back = decoder(tf_rep)
```
 check what filter banks are available
```python
from speechx.filterbanks import show_availables
show_availables()
# Out: FreeFB ...

# Alias defined
free = FreeFB
analytic_free = AnalyticFreeFB
param_sinc = ParamSincFB
stft = STFTFB
multiphase_gammatone = mpgtf = MultiphaseGammatoneFB
```
#### Use the different metrics & losses
 most widely used speech separation PIT SI-SNR loss
 ```python
 from speechx.losses import PITLossWrapper, pairwise_neg_sisdr
 loss_func = PITLossWrapper(pairwise_neg_sisdr, pit_from='pw_mtx')
 ```
 evaluation of speech separation using all kinds of metrics
 ```python
 import numpy as np
 from speechx.metrics import get_metrics
 mix = np.random.randn(1, 16000)
 clean = np.random.randn(2, 16000)
 est = np.random.randn(2, 16000)
 metrics_dict = get_metrics(mix, clean, est, sample_rate=8000,
                                          metrics_list='all')
# all : ["si_sdr", "sdr", "sir", "sar", "stoi", "pesq"]
 ```
 check what metrics are available
 ```python
 from speechx import show_available_metrics
 show_available_metrics()
 # Out: si_sdr ...
 ```
#### Use the acoustic modules
We use the similar architecture as `pytoch`, and put all useful modules in a flattern design into the `speechx.nn` package.
```python
from speechx.nn import AcousticTransformerLayer
# This is a specially designed transformer encoder layer for Speech Tasks
trans_layer=AcousticTransformerLayer(256, n_heads=8, dim_ff=512)
```
 To check all modules, please go to our docs
[Latest Docs](https://Speech-AI.github.io/SpeechX/)
## Tutorials
([↑up to contents](#contents))
Here is a list of notebooks showing example usage of SpeechX's features.
- [Getting started with SpeechX](./notebooks/01_SpeechXGettingStarted.ipynb)
- [Filterbank API](./notebooks/02_Filterbank.ipynb)
- [Permutation invariant training wrapper `PITLossWrapper`](./notebooks/03_PITLossWrapper.ipynb)
- [Process large wav files](./notebooks/04_ProcessLargeAudioFiles.ipynb)


## Running a recipe
([↑up to contents](#contents))
Running the recipes requires additional packages in most cases,
we recommend running :
```bash
# from SpeechX/
pip install -e .[dev]
```
Then choose the recipe you want to run and run it!
```bash
cd egs/SpeechSeparation/librimix/TransMask
. ./run.sh
```
More information in [egs/README.md](./egs).

## Available recipes
([↑up to contents](#contents))
* [x] [ConvTasnet](./egs/SpeechSeparation/librimix/ConvTasNet) ([Luo et al.](https://arxiv.org/abs/1809.07454))
* [x] [DualPathRNN](./egs/SpeechSeparation/librimix/DPRNN) ([Luo et al.](https://arxiv.org/abs/1910.06379))
* [x] [TransMask](./egs/SpeechSeparation/librimix/TransMask)([Zhang et al.](https://arxiv.org/abs/2102.09978))
* [ ] Wavesplit (coming) ([Zeghidour et al.](https://arxiv.org/abs/2002.08933))

## Supported datasets
([↑up to contents](#contents))
* [x] [WSJ0-2mix](./egs/asteroid/wsj0-mix) / WSJ03mix ([Hershey et al.](https://arxiv.org/abs/1508.04306))
* [x] [WHAM](./egs/asteroid/wham) ([Wichern et al.](https://arxiv.org/abs/1907.01160))
* [x] [WHAMR](./egs/asteroid/whamr) ([Maciejewski et al.](https://arxiv.org/abs/1910.10279))
* [x] [LibriMix](./egs/SpeechSeparation/librimix) ([Cosentino et al.](https://arxiv.org/abs/2005.11262))
* [x] [Microsoft DNS Challenge](./egs/asteroid/dns_challenge) ([Chandan et al.](https://arxiv.org/abs/2001.08662))
* [x] [SMS_WSJ](./egs/asteroid/sms_wsj) ([Drude et al.](https://arxiv.org/abs/1910.13934))
* [x] [MUSDB18](./egs/asteroid/musdb18) ([Raffi et al.](https://hal.inria.fr/hal-02190845))
* [x] [FUSS](./egs/asteroid/fuss) ([Wisdom et al.](https://zenodo.org/record/3694384#.XmUAM-lw3g4))
* [x] [AVSpeech](./egs/asteroid/avspeech) ([Ephrat et al.](https://arxiv.org/abs/1804.03619))
* [x] [Kinect-WSJ](./egs/asteroid/kinect-wdj) ([Sivasankaran et al.](https://github.com/sunits/Reverberated_WSJ_2MIX))

## Pretrained models
([↑up to contents](#contents))
See [here](./docs/source/readmes/pretrained_models.md)

## Contributing
([↑up to contents](#contents))
We are always looking to expand our coverage of the source separation
and speech enhancement research, the following is a list of
things we're missing.
You want to contribute? This is a great place to start!
* Wavesplit ([Zeghidour and Grangier](https://arxiv.org/abs/2002.08933))
* Parametrized filterbanks from [Tukuljac et al.](https://openreview.net/forum?id=HyewT1BKvr)


Don't forget to read our [contributing guidelines](./CONTRIBUTING.md).

You can also open an issue or make a PR to add something we missed in this list.

## TensorBoard visualization
The default logger is TensorBoard in all the recipes. From the recipe folder,
you can run the following to visualize the logs of all your runs. You can
also compare different systems on the same dataset by running a similar command
from the dataset directiories.
```bash
# Launch tensorboard (default port is 6006)
tensorboard --logdir exp/ --port tf_port
```
If your launching tensorboard remotely, you should open an ssh tunnel
```bash
# Open port-forwarding connection. Add -Nf option not to open remote.
ssh -L local_port:localhost:tf_port user@ip
```
Then open `http://localhost:local_port/`. If both ports are the same, you can
click on the tensorboard URL given on the remote, it's just more practical.


## Guiding principles
([↑up to contents](#contents))
* __Modularity.__ Building blocks are thought and designed to be seamlessly
plugged together. Filterbanks, encoders, maskers, decoders and losses are
all common building blocks that can be combined in a
flexible way to create new systems.
* __Extensibility.__ Extending SpeechX with new features is simple.
Add a new filterbank, separator architecture, dataset or even recipe very
easily.
* __Reproducibility.__ Recipes provide an easy way to reproduce
results with data preparation, system design, training and evaluation in a
single script. This is an essential tool for the community!

## Citing SpeechX
([↑up to contents](#contents))
Currently, we don't have a specific paper for SpeechX, since it is far from our
 expected status (we want to include much more light-weight speech taks like music
 separation, speech extraction, speech emotion recognition and more). However, we
 would appreciate it if anyone find this repo useful. Temporarily, you can cite this
 paper
```BibTex
@article{zhang2021transmask,
  title={TransMask: A Compact and Fast Speech Separation Model Based on Transformer},
  author={Zhang, Zining and He, Bingsheng and Zhang, Zhenjie},
  journal={arXiv preprint arXiv:2102.09978},
  year={2021}
}
```

[comment]: <> (Badge)
[codecov-badge]: https://codecov.io/gh/Speech-AI/SpeechX/branch/master/graph/badge.svg
[codecov]: https://codecov.io/gh/Speech-AI/SpeechX
[slack-badge]: https://img.shields.io/badge/slack-chat-green.svg?logo=slack
[slack-invite]: https://join.slack.com/t/speechxgroup/shared_invite/zt-pjdxwjvo-Lygc5dlD1zSpyA2SwR1Pyg

[comment]: <> (Others)
[issue]: https://github.com/Speech-AI/SpeechX/issues/new
[pr]: https://github.com/Speech-AI/SpeechX/compare