Repository: mir-evaluation/mir_eval
Issue: bss_eval_sources gets slower with more CPUs
Root cause: External Dependency Breakage

### What ? 
`bss_eval_sources` gets slower as the number of available CPUs grow. We  got up to 4x slower computation with 32 cores compared to only one core (thx @sunits).

### Expected behaviour
If not speeding up with number of available CPU cores, at least not slowing down. 

### How to reproduce :
- Create a file `sdr_speed_test.py` containing 
```python
import numpy as np
from mir_eval.separation import bss_eval_sources

def time_sdr():
    ref = np.random.randn(2, 64000)
    est = np.random.randn(2, 64000)
    sdrs = bss_eval_sources(ref, est, compute_permutation=True)
```
- Time the execution with all available cores : 
```
python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()" 
```
- Time the execution with a single core:
```
taskset --cpu-list 0 python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()"
```
The difference is even greater if you have a large number of cores on your machine but I could see the difference on my 8 core laptop. This was replicated on three different machines. 

### Additional info
Out `np.__config__.show()` looks like this : 
```
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
```

did you try to disable BLAS for testing?

How could you be that fast? ;-) 
Trying now

Is there an easier way to do that than instaling numpy from source? 

> Is there an easier way to do that than instaling numpy from source?

I guess the easiest is to just force the number of threads to be 1

```bash
export MKL_NUM_THREADS=1
export OMP_NUM_THREADS=1
```

> I guess the easiest is to just force the number of threads to be 1
> 
> ```shell
> export MKL_NUM_THREADS=1
> export OMP_NUM_THREADS=1
> ```

Thanks. 
It's now around 8 times slower than with one core (with BLAS). And two times slower than with all the cores and BLAS.

@craffel Sorry for pinging you but, have you seen this? 

I definitely have never seen this. Do all of your numpy/scipy calls exhibit similar behavior? I would try to narrow down where in the code it's getting Nx slower, and see if there is an issue with one of those particular functions. `mir_eval` is just a pure-Python client library so it's unlikely to be a bug with our code (but who knows).

Good point. I don't have these problems with numpy in general. 
I'm checking it right now and I have the same behavior with `_project`, and I still find the same execution speed difference in the [`solve` lines](https://github.com/craffel/mir_eval/blob/19ed4dfbc193f9de48e56fc34198556fd3de3766/mir_eval/separation.py#L714) so it indeed sounds like a numpy problem. 

I copy here the script I used to time it (copy if sdr_speed_test.py) 
```python
import numpy as np


def time_sdr():
    # If I run this command and enter it with ipdb
    # I get the all the variables' shapes needed for
    # the _project function.
    # ref = np.random.randn(2, 64000)
    # est = np.random.randn(2, 64000)
    # sdrs = bss_eval_sources(ref, est, compute_permutation=True)
    flen, nsrc = 512, 1
    G = np.random.randn(flen, flen)
    D = np.random.randn(flen)
    nsampl, nsrc = 64000, 1
    reference_sources = np.random.randn(1, nsampl + flen - 1)

    # One or the other, more CPU usage in list comp
    # C = solve(G, D, flen, nsrc)
    [solve(G, D, flen, nsrc) for _ in range(5)]


def solve(G, D, flen, nsrc):
    try:
        C = np.linalg.solve(G, D).reshape(flen, nsrc, order='F')
    except np.linalg.linalg.LinAlgError:
        C = np.linalg.lstsq(G, D)[0].reshape(flen, nsrc, order='F')
    return C
```

And same two commands to time it, one CPU 
```
taskset --cpu-list 0 python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()"
```
With all CPUs
```
python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()"
```

Weird, I have never heard of `np.linalg.solve` being slower with more CPUs. I'm not sure it's within `mir_eval` scope to debug this.

No, it's not. 
Are you able to reproduce the timing though? 
I'll open an issue there soon.

Here is what I get on a Google Cloud VM, FWIW:
```
> taskset --cpu-list 0 python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()"
10 loops, best of 3: 51.3 msec per loop
>python -m timeit -s "from sdr_speed_test import time_sdr" "time_sdr()"
10 loops, best of 3: 74.1 msec per loop
```

Thanks for trying. 
NumPy sends me to OpenBLAS, I'll close this.