Title: Hind_Encorp all done
Author: rahul-art (CONTRIBUTOR)
State: closed
Created: 2020-12-15T01:36:02Z


Comments:
ALL TEST PASSED locally  @yjernite 

@rahul-art  kindly run the following from the datasets folder 

```
make style 
flake8 datasets

```


@skyprince999 I did that before it says all done 


I did that again it gives the same output all done and then  I synchronized my changes with this branch 

@lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
`**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`




I cloned the branch and it seems to work fine at my end. try to clear the cache - 

```
rm -rf  /home/ubuntu/.cache/huggingface/datasets/
rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
```
But the dataset has only one record. Is that correct? 
![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)


> @lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
> `**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`

You can ignore this error by adding `ignore_verifications=True` to `load_dataset`.

This error is raised because you're loading a dataset that you've already loaded once in the past. Therefore the library does some verifications to make sure it's generated the same way. 

However since you've done changes in the dataset script you should ignore these verifications.

You can regenerate the dataset_infos.json with
```
datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications
```


> I cloned the branch and it seems to work fine at my end. try to clear the cache -
> 
> ```
> rm -rf  /home/ubuntu/.cache/huggingface/datasets/
> rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
> ```
> 
> But the dataset has only one record. Is that correct?
> ![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)

Yes the current parsing is wrong, I've already given @rahul-art some suggestions and it looks like it works way better now (num_examples=273885).

Thanks for fixing the parsing @rahul-art !
Feel free to commit and push your changes once it's ready :) 

i ran the command you provided datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications 
but now its giving this error @lhoestq 

FileNotFoundError: Couldn't find file locally at ./datasets/hindi_encorp/hindi_encorp.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/./datasets/hindi_encorp/hindi_encorp.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/hindi_encorp/hindi_encorp.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.


whoops I meant `hind_encorp` instead of `hindi_encorp` sorry

@lhoestq  all changes have done successfully in this PR #1584

Ok thanks ! closing this one in favor of #1584