Data loader for seq of label model
State: closed | Pull request: yes

Add feature loader to accept external features stored in pickle file.
Add feature to seq_label data loader
Add function to convert seq_label to relative labels.
Add test and test data.


Input manifest file looks like: 
```
{"feature_filepath": "tests/.data/asr/feat/AMI-00IB4001-000.p", "seq_label": "zero one two two one two one two two one two two three one two two two zero two one one two two one one two one one one one...."}
{"feature_filepath": "tests/.data/asr/feat/AMI-00IB4001-001.p", "seq_label": "0 1 0 2 1 1 0 1 1 1 1........"}
```
Output looks like:
Tensor(external feature), Tensor(feature length) ,Tensor(label), Tensor(length of seq list) 

```
(tensor([[-0.6812,  0.5182, -0.3697,  ..., -0.4531, -0.8935, -1.6601],                                                               
        [-0.2840,  0.8350,  0.9644,  ...,  0.0409,  0.0546, -0.6683],                                                                
.....), tensor(50), tensor([0., 1., 2., 2., 1., 2., 1., 2., 2., 1., 2
., 2., 3., 1., 2., 2., 2., 0.,
        2., 1., 1., 2., 2., 1., 1., 2., 1., 1., 1., 1., 0., 0., 0., 2., 0., 2.,
        2., 2., 1., .......]), tensor(50))
```

This pull request **introduces 2 alerts** when merging a4efaa132eae55f2663972c1d8f368b23419aff6 into c2d9559e0cf36dfeb88c1848e8360ae9a6b08776 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2714799267083096cd0a3b6f4f534df19d9bcabb)

**new alerts:**

* 2 for Unused import

is it draft or ready to review?
Are there any feature models that uses these loaders, if isn't better to have it merge along with it ?!

> is it draft or ready to review?
> Are there any feature models that uses these loaders, if isn't better to have it merge along with it ?!
 
@nithinraok  ready to review. Well I thought about it. Tried to split huge PR into small things. And the feature loader should be less correlated with models. 

General questions on overall feature loader:
instead of seq label inputted through manifest should we support reading from file?
any helper function to get seq_label from rttm file?
does each row in manifest correspond to each session or part of session? if its part of session can that be handled automatically for seq_label inputs (grouth truth) 

> General questions on overall feature loader:
> instead of seq label inputted through manifest should we support reading from file?

Not for now. It will make data augmentation much complicated.
> any helper function to get seq_label from rttm file?

No right now the emb and seq_label is generated offline. That is **not related to loader but data preparation imo** and shall include in the following PRs. 


> does each row in manifest correspond to each session or part of session? if its part of session can that be handled automatically for seq_label inputs (grouth truth)


Could be each or part. Not quite understand this "if its part of session can that be handled automatically for seq_label inputs (grouth truth)". If I understand your question correctly, theoretically yes but as in above question, the augmentation will be difficult. This will be handled in neural clustering in clustering diarizer.


This pull request **introduces 2 alerts** when merging 76265fa76f528d54098dbab4423701d950ad6770 into 6d53452eb1487102c0d5ae657815ec2aecc12076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-329bc8e6c7700dfe43401c574fc0d3d7b214b4b2)

**new alerts:**

* 1 for Unused import
* 1 for Illegal raise

> Add the docstring, LGTM otherwise

@titu1994 Thanks! Added. will merge once the checks finished.