Title: add BNL newspapers 
Author: davanstrien (MEMBER)
State: closed
Created: 2021-12-07T15:43:21Z

This pull request adds the BNL's [processed newspaper collections](https://data.bnl.lu/data/historical-newspapers/) as a dataset. This is partly done to support BigScience see: https://github.com/bigscience-workshop/data_tooling/issues/192. 

The Datacard is more sparse than I would like but I plan to make a separate pull request to try and make this more complete at a later date. 

I had to manually add the `dummy_data` but I believe I've done this correctly (the tests pass locally).

Comments:

> Also, maybe calling the dataset as "bnl_historical_newspapers" and setting "processed" as one configuration name?

This sounds like a good idea but my only question around this is how easy it would be to use the same approach for processing the other newspaper collections [https://data.bnl.lu/data/historical-newspapers/](). 

For example, the "BIG DATA PACK" is `257GB` of ALTO XML. This format is slightly more annoying to process because the metadata and text are contained in different files but the bigger issue might be that processing this XML using the Python XML libraries will probably be quite slow? I had thought for those larger datasets it might be more appropriate to use the Beam datasets? I don't have any experience using Beam so I'm not sure what that would involve and there is a reason to not include it in a dataset script alongside a non Beam dataset? 

If there isn't an issue with potentially later adding other datasets (which may require Beam) into the same script I'll add one config for the processed version now which leaves open the option for later adding the other datasets. If this makes sense I'll also change the name as you suggest. 

There is another dataset that could be a good candidate for inclusion here is the "Monograph Text pack" which is also processed into a simpler XML format however as the name suggests this isn't newspapers so might be confusing to include under a 'newspapers' script. One option would be to put everything under a `BNL` collection but it might be better to keep the monographs separate if they are added as a dataset so a single script doesn't end up including too much variety of content types? 





> My initial idea was to contribute the script also as "community" datasets (instead of canonical), i.e. in this case, pushing the script to the repo [huggingface.co/datasets/bigscience-catalogue-data/bnl_historical_newspapers](https://huggingface.co/datasets/bigscience-catalogue-data/bnl_historical_newspapers)

Sorry to respond to this late - happy for this to go in the community datasets. I think it would be nice to include in the canonical datasets at some point but since there is less urgency with this I could try and first work on improving the Datacard before doing that (i.e. make this a draft PR) - let me know if you think that makes more sense? 




> My initial idea was to contribute the script also as "community" datasets (instead of canonical), i.e. in this case, pushing the script to the repo https://huggingface.co/datasets/bigscience-catalogue-data/bnl_historical_newspapers
> One of the advantages is that no dummy data is required, so the addition can be made faster
> On the other hand, one disadvantage is that contributions cannot be made through PRs
> Therefore, we should use the Issue page for discussions, reviews, decisions,...

Sure we can use the issues to discuss/review community datasets. Maybe let's have an issue template for that ?
For this dataset in particular I'll let @albertvillanova decide whether it's best as community dataset or not. IMO both are fine :)

> I had thought for those larger datasets it might be more appropriate to use the Beam datasets? I don't have any experience using Beam so I'm not sure what that would involve and there is a reason to not include it in a dataset script alongside a non Beam dataset?

Beam is nice to process a dataset once and for all and store the resulting processed data on the Hugging Face Hub or elsewhere. However for big datasets it must run on a distributed processing runtime like Google DataFlow, which is often inconvenient for many users. We've been using it though for datasets like Wikipedia and sharing the processed data in a GCS bucket.

So feel free to use the tools you like to process the datasets, but in the end I think we just need to host the processed data in a convenient format on the Hugging Face Hub to share it with the community. The processing script you used can also be shared with the community for reproducibility and documentation. But maybe @albertvillanova already has something in mind

> > My initial idea was to contribute the script also as "community" datasets (instead of canonical), i.e. in this case, pushing the script to the repo [huggingface.co/datasets/bigscience-catalogue-data/bnl_historical_newspapers](https://huggingface.co/datasets/bigscience-catalogue-data/bnl_historical_newspapers)
> > One of the advantages is that no dummy data is required, so the addition can be made faster
> > On the other hand, one disadvantage is that contributions cannot be made through PRs
> > Therefore, we should use the Issue page for discussions, reviews, decisions,...
> 
> Sure we can use the issues to discuss/review community datasets. Maybe let's have an issue template for that ? For this dataset in particular I'll let @albertvillanova decide whether it's best as community dataset or not. IMO both are fine :)

Thanks, I'll hold off and let @albertvillanova decide best place for this. 

> > I had thought for those larger datasets it might be more appropriate to use the Beam datasets? I don't have any experience using Beam so I'm not sure what that would involve and there is a reason to not include it in a dataset script alongside a non Beam dataset?
> 
> Beam is nice to process a dataset once and for all and store the resulting processed data on the Hugging Face Hub or elsewhere. However for big datasets it must run on a distributed processing runtime like Google DataFlow, which is often inconvenient for many users. We've been using it though for datasets like Wikipedia and sharing the processed data in a GCS bucket.
> 
> So feel free to use the tools you like to process the datasets, but in the end I think we just need to host the processed data in a convenient format on the Hugging Face Hub to share it with the community. The processing script you used can also be shared with the community for reproducibility and documentation. But maybe @albertvillanova already has something in mind

That's useful, my own 2 cents are that it would make sense to do as @albertvillanova suggested and:-

- rename the dataset to 'bnl_newspapers' 
- make the 'processed dataset' a config 

I won't try and include all the other datasets now but this leaves open the option of adding those later. The actual ALTO processing should be okay to do but I think it makes sense to do this as a one-off process and make the plain text + some associated metadata available elsewere so the dataset script can be kept simple and the processing doesn't get done multiple times. 

@albertvillanova if that sounds okay I'll update pull request to include those changes. 


@albertvillanova I've now created a config (currently with only one option) and renamed the dataset. This should keep the option to add other configs based on different bnl newspapers in the future. 


@mariosasko thanks for those suggestions 

I just merged `master` into your branch to fix the CI :)

@albertvillanova do you have additional comments ? Otherwise I think this PR is ready to merge :)

> @davanstrien you did an awsome job!!! Thanks a lot!
> 
> Just some very minor comments (mainly about the README documentation), and we merge this to master!

Thanks! Hopefully all addressed now.  Thanks again for all the support with this pull request!