rachel521/github-issues-simple huggingface dataset:
Add numeric fused head dataset
Opened by ghomasHudson (closed)

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela 

Comments: > Thanks for adding this @ghomasHudson!
> I added some comments for some of the fields.
> 
> Also, I'm not sure about this since I haven't used the library yet, but maybe it's worth adding the identification and resolution as two separate datasets?

Thanks for replying @yanaiela - I hope this will make your dataset more accessible to a wider audience - I've added the changes to the model card you suggested.

In terms of the identification and resolution tasks, I've currently added them as separate `splits` in huggingface/datasets so you can load identification like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "identification")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "start_index": 11, "end_index": 12, "label": 0}
```
And resolution like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "resolution")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "head": ["AGE"], "anchors_indices": [12], ...}
```

I hope so too, thanks!

Re the splits, that makes sense to me.
custom load dataset to hub
Opened by andysingal (closed)

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub

Comments: This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

> This is an issue for the [Datasets repo](https://github.com/huggingface/datasets).

Thanks @sgugger , I guess I will wait for them to address the issue . Looking forward to hearing from them 

You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

> You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.

how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot ! 

> > You can use `.push_to_hub("<username>/<repo>")` to push a `Dataset` to the Hub.
> 
> how about subset? like `.load_dataset("<username>/<repo>", "<subset>")`, how can I upload multi-dataset in one repo? thanks a lot !

I solved it by upgrading `Datasets` version to 2.15.0
Fix license/citation information of squadshifts dataset card
Opened by albertvillanova (closed)

This PR fixes the license/citation information of squadshifts dataset card, once the dataset owners have responded to our request for information:
- https://github.com/modestyachts/squadshifts-website/issues/1

Additionally, we have updated the mention in their website to our `datasets` library (they were referring old name `nlp`):
- https://github.com/modestyachts/squadshifts-website/pull/2#event-7500953009

Comments: _The documentation is not available anymore as the PR was closed or merged._
Add spanish billion words corpus
Opened by mariagrandury (closed)

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.

Comments: Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits. 
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Yes thank you !

The new PR is #1476 :)
Perform minor refactoring: use config
Opened by albertvillanova (closed)

Perform minor refactoring related to `config`.

Comments: @lhoestq is there a problem in the master branch? I got a segmentation fault...
```
tests/test_table.py::test_concatenation_table_cast[in_memory] Fatal Python error: Segmentation fault
```

Oh wow. Let me re-run the CI just to make sure

Hmm interesting, the segfault is still there. I'm investigating this issue on my windows machine

Feel free to merge master into this branch to fix the CI :)
Lossy json serialization - deserialization of dataset info 
Opened by anuragprat1k (open)

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead. 

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
    
test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}, 
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```


### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3

Comments: Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.
datasets slicing with seed 
Opened by ghost (closed)

Hi
I need to slice a dataset with random seed, I looked into documentation here https://huggingface.co/docs/datasets/splits.html 
I could not find a seed option, could you assist me please how I can get a slice for different seeds?
thank you.
@lhoestq  

Comments: Hi :) 
The slicing API from https://huggingface.co/docs/datasets/splits.html doesn't shuffle the data.
You can shuffle and then take a subset of your dataset with
```python
# shuffle and take the first 100 examples
dataset = dataset.shuffle(seed=42).select(range(100))
```

You can find more information about shuffling and selecting rows in the documentation: https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows

thank you so much

On Mon, Jan 18, 2021 at 3:17 PM Quentin Lhoest <notifications@github.com>
wrote:

> Hi :)
> The slicing API doesn't shuffle the data.
> You can shuffle and then take a subset of your dataset with
>
> # shuffle and take the first 100 examplesdataset = dataset.shuffle(seed=42).select(range(100))
>
> You can find more information about shuffling and selecting rows in the
> documentation:
> https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1747#issuecomment-762278134>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM5D5MDPLJGI4IG3UADS2Q7GPANCNFSM4WHLOZJQ>
> .
>
Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0
Opened by JGrel (open)

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

Comments: Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results


Hi, sorry for no response from my side. I will try to do it today.

Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)

Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
Add hashes to cnn_dailymail
Opened by jbragg (closed)

The URL hashes are helpful for comparing results from other sources.

Comments: Looks you to me :)

Could you also update the json file that goes with the dataset script by doing 
```
nlp-cli test ./datasets/cnn_dailymail --save_infos --all_configs
```
It will update the features metadata and the size of the dataset with your changes.

@lhoestq I ran that command.

Thanks for the helpful repository!
Support soft links for load_datasets imagefolder
Opened by billytcl (open)

### Feature request

Load_dataset from a folder of images doesn't seem to support soft links. It would be nice if it did, especially during methods development where image folders are being curated.

### Motivation

Images are coming from a complex variety of sources and we'd like to be able to soft link directly from the originating folders as opposed to copying. Having a copy of the file ensures that there may be issues with image versioning as well as having double the amount of required disk space.

### Your contribution

N/A
The current implementation has a potential bug in the sort method
Opened by wangyuxinwhy (closed)

### Describe the bug

In the sort method，here's a piece of code

```python
# column_names: Union[str, Sequence_[str]]

# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    column_names = [column_names]
```

I get an error when I pass in a tuple based on the column_names type annotation, it will raise an errror.As in the example below, while the type annotation implies that a tuple can be passed.

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

Of course, after I modified the tuple into a list, everything worked fine

Change the code to the following so there will be no problem

```python
# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    if isinstance(column_names, str):
        column_names = [column_names]
    else:
        column_names = list(column_names)
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

### Expected behavior

Passing tuple into column_names should be equivalent to passing list

### Environment info

- `datasets` version: 2.13.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2

Comments: Thanks for reporting, @wangyuxinwhy.
Make inspect.get_dataset_config_names always return a non-empty list of configs
Opened by severo (closed)

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161


Comments: Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?

Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)
Deprecate canonical Multilingual Librispeech
Opened by polinaeterna (closed)

Deprecate canonical Multilingual Librispeech in favor of [the community one](https://huggingface.co/datasets/facebook/multilingual_librispeech) which supports streaming.

However, there is a problem regarding new ASR template schema: since it's changed, I guess all community datasets that use this template do not work with new version of the library, including MLS. Should we somehow notify users about that or is it possible to change this line ourselves? For MLS specifically, I cannot change the code directly as I'm not the member of the Facebook org. 

Hm, and the code should be change after the release, no?

Comments: _The documentation is not available anymore as the PR was closed or merged._

Yes, as discussed in #4006 we should update facebook/multilingual_librispeech indeed before we do a release. @anton-l could you help taking care of updating facebook/multilingual_librispeech ? We need to update the task template
```python
task_templates=[AutomaticSpeechRecognition(audio_column="audio", transcription_column="text")],
```
and write that `datasets>=2.1` is necessary to load it in the dataset card.

Once the change is done we can merge this PR and do the release I think

@polinaeterna @lhoestq 
Updated the script and the dataset card: https://huggingface.co/datasets/facebook/multilingual_librispeech 

@anton-l @lhoestq now previewer doesn't work for this datasets as it cannot recognize new `audio_column` argument:
![image](https://user-images.githubusercontent.com/16348744/161233533-3170760b-5141-4525-9592-6675669c223a.png)

I'm not an expert in previewer things, where should I look into the corresponding code?

Yes, there are several datasets with the same error, eg https://github.com/huggingface/datasets-preview-backend/issues/188. I'm not sure what I should do to fix this? Upgrade datasets to master?


@anton-l ended up removing the task template in facebook/multilingual_librispeech to make it work for the current version of `datasets` and fix the viewer :) thanks !

@lhoestq can we merge now? ^^
Fix docs typos
Opened by mariosasko (closed)

This PR fixes few typos in the docs and the error in the code snippet in the set_format section in docs/source/torch_tensorflow.rst. `torch.utils.data.Dataloader` expects padded batches so it throws an error due to not being able to stack the unpadded tensors. If we follow the Quick tour from the docs where they add the `truncation=True, padding='max_length'` arguments to the tokenizer before passing data to Dataloader, we can easily fix the issue.
ConnectionError: Couldn't reach dataset_infos.json 
Opened by yulingao (closed)

### Describe the bug

I'm trying to load codeparrot/codeparrot-clean-train, but get the following error:

ConnectionError: Couldn't reach https://huggingface.co/datasets/codeparrot/codeparrot-clean-train/resolve/main/dataset_infos.json (ConnectionError(ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))))



### Steps to reproduce the bug

train_data = load_dataset('codeparrot/codeparrot-clean-train', split='train')


### Expected behavior

download the dataset

### Environment info

centos7

Comments: Unfortunately, I can't reproduce the error. What does the following code return for you?
```python
import requests
from huggingface_hub import hf_hub_url
r = requests.get(hf_hub_url("codeparrot/codeparrot-clean-train", "dataset_infos.json", repo_type="dataset"))
```

Also, can you provide more info about your network (region, proxies, etc.)?
Add `features` param to `IterableDataset.map`
Opened by alvarobartt (closed)

## Description

As suggested by @lhoestq in #3888, we should be adding the param `features` to `IterableDataset.map` so that the features can be preserved (not turned into `None` as that's the default behavior) whenever the user passes those as param, so as to be consistent with `Dataset.map`, as it provides the `features` param so that those are not inferred by default, but specified by the user, and later validated by `ArrowWriter`.

This is internally handled already by the functions relying on `IterableDataset.map` such as `rename_column`, `rename_columns`, and `remove_columns` as described in #5287.

## Usage Example

```python
from datasets import load_dataset, Features

ds = load_dataset("rotten_tomatoes", split="validation", streaming=True)
print(ds.info.features)
ds = ds.map(
    lambda x: {"target": x["label"]},
    features=Features(
        {"target": ds.info.features["label"], "label": ds.info.features["label"], "text": ds.info.features["text"]}
    ),
)
print(ds.info.features)
```

Comments: _The documentation is not available anymore as the PR was closed or merged._
Trying to get in touch regarding a security issue
Opened by JamieSlome (closed)

Hey there!

I'd like to report a security issue but cannot find contact instructions on your repository.

If not a hassle, might you kindly add a `SECURITY.md` file with an email, or another contact method? GitHub [recommends](https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository) this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Comments: Hi @JamieSlome,

Thanks for reaching out. Yes, you are right: I'm opening a PR to add the `SECURITY.md` file and a contact method.

In the meantime, please feel free to report the security issue to: feedback@huggingface.co
None Type error for swda datasets
Opened by hannan72 (closed)

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Comments: Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
[WMT] Add all wmt
Opened by patrickvonplaten (closed)

This PR adds all wmt datasets scripts. At the moment the script is **not** functional for the language pairs "cs-en", "ru-en", "hi-en" because apparently it takes up to a week to get the manual data for these datasets: see http://ufal.mff.cuni.cz/czeng. 

The datasets are fully functional though for the "big" language pairs "de-en" and "fr-en". 

Overall I think the scripts are very messy and might need a big refactoring at some point.

For now I think there are good to merge (most dataset configs can be used). I will add "cs", "ru" and "hi" when the manual data is available.