update spectral clustering method
State: closed | Pull request: yes

This PR :
* Adds new NMSE spectral clustering method
* Updates Docs to include dataset creation for any dataset, with special mentions for AMI and CH109. 
* Adds ability to diarize for unknown number of speakers/session with very good accuracies 
* Replace SpectralCluster from requirements with pyamg (math computing repo) 

Signed-off-by: nithinraok <nithinrao.koluguri@gmail.com>

This pull request **introduces 3 alerts** when merging b7b73650384ffbf4109fdaac69a3aa31e4ecbfe9 into 81df4358cd3afb3750bbc87981d57f54244f354c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6034fac3632b1bb543e7793c3c154b7958b232b4)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This pull request **introduces 3 alerts** when merging 78c63c7311d85d7b75e7fe71c13f4f21640cbcbf into 84c4d102b8c78b75c067f73247166d04a8161d0a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a238e472e4b1d7e6fe7af1838f10a3ed9d9528c8)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This pull request **introduces 3 alerts** when merging d4db6ff5bcd4002b2253004de4510efe582cc39d into 917356570239ac196241335632caa7c2a0225397 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-eede0e547ff995089fd0bf2be23ea9bf23ec1f45)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This pull request **introduces 3 alerts** when merging e088574bd168fbfc43b090704e93f70ce13df6b9 into 917356570239ac196241335632caa7c2a0225397 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8dc7d8393253c15cbcff62f2c4b570a7ad9685b2)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This pull request **introduces 3 alerts** when merging 37e7a1b55d6016fb1c086fb39d94d36a889683cf into 917356570239ac196241335632caa7c2a0225397 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-04e157cab92d850ecffa025495c1d612fb73a649)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This pull request **introduces 3 alerts** when merging b8009df687ab1b96893136cc4964ad3dd0a2e6ed into 917356570239ac196241335632caa7c2a0225397 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5f0a902c75d4e2072a652654b02847f297280c6b)

**new alerts:**

* 2 for Except block handles &#39;BaseException&#39;
* 1 for Variable defined multiple times

This change adds `pyamg` to `requirements_asr.txt` and breaks Nemo installation in a conda environment without gcc.
Is it added by accident?

@nithinraok  import guard this and warn / error out if the user doesn't have pyamg so that we can remove it from the requirements list 

Thanks @DXist for the question. May I know your environment setup? Like Mac/Linux and conda or with pip. Because first gcc is required for conda installation. https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#compiler-packages 


@nithinraok actually I use [miniforge](https://github.com/conda-forge/miniforge) and build inference and train Docker images using the [Dockerfile](http://ix.io/3mJ3). I get minimal images without build requirements. `base` target image is more compact then the image based on nvcr.io Pytorch.

I could install pyamg from conda-forge but I haven't found any usages of pyamg in the code and decided to comment in this issue.

pyamg is used as a backend, while computing embedding clusterers using sklearn. So its only useful for speaker diarization tasks. 
In NeMo we place speech classification, speaker dirarization, and of course speech recognition as part of asr collection, and hence the requirement. 

Thank you for clarification!

@nithinraok is it possible to have both spectral and COS clustering to pass as arguments to the diarizer. I have noticed that with the recent update to COS the DER has gone up from 0 to 0.4 for the an4.wav example file used in the demo notebooks