t20062443/AT1ds huggingface dataset:
ameet-1997 (NONE) commented on 2021-01-13 20:00:15:
That's a great idea! Thank you so much!

When I try that solution, I get the following error when I try to concatenate `datasets` and `modified_dataset`. I have also attached the output I get when I print out those two variables. Am I missing something?

Code:
``` python
combined_dataset = concatenate_datasets([datasets, modified_dataset])
```

Error:
```
AttributeError: 'DatasetDict' object has no attribute 'features'
```

Output:
```
(Pdb) datasets
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
(Pdb) modified_dataset
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
```

The error is stemming from the fact that the attribute `datasets.features` does not exist. Would it not be possible to use `concatenate_datasets` in such a case? Is there an alternate solution?
thomwolf (MEMBER) commented on 2021-01-13 21:25:05:
You should do `combined_dataset = concatenate_datasets([datasets['train'], modified_dataset['train']])`

Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.
dwadden (CONTRIBUTOR) commented on 2021-01-14 01:41:44:


> > I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?
> 
> Sure ! Also feel free to ping us for reviews or if we can help :)
> 
OK! We're organizing a [shared task](https://sdproc.org/2021/sharedtasks.html#sciver) based on the dataset, and I made some updates and changed the download URL - so the current code points to a dead URL. I'll update appropriately once the task is finalized and make a PR.

> > It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?
> 
> What makes you think that ?
> Afaik there's no scifact on our google storage

You're right, I had the data cached on my machine somewhere.
albertvillanova (MEMBER) commented on 2021-01-14 09:34:45:
Hi @lhoestq, I am trying to understand better the logic...

Why do we have a `dynamic_module_path` besides the modules cache path?
```python
DYNAMIC_MODULES_PATH = os.path.join(HF_MODULES_CACHE, "datasets_modules")
```
Moreover, 2 subdirectories (for datasets and for metrics) were created inside it:
```python
DATASETS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "datasets")
METRICS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "metrics")
```
lhoestq (MEMBER) commented on 2021-01-14 09:41:30:
Hi :) 
The modules cache path is the path added to `sys.path`.
Therefore inside we need to have a folder that is going to be a package: `datasets_modules`.
This package will contain dynamic modules, i.e. datasets and metrics modules added on-the-fly.
Then we have two sub-modules `datasets_modules.datasets` and `datasets_modules.metrics`.

Maybe we can make things more explicit in the code with some comments explaining the structure, and maybe better variable naming as well..

Also I wanted to say that I started to work on offline loading of modules in #1726 and actually it lead to do similar changes to what you did to control the path where modules are stored.
lhoestq (MEMBER) commented on 2021-01-14 09:52:40:
> Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.

My opinion is that users should always know in advance what type of objects they're going to get. Otherwise the development workflow on their side is going to be pretty chaotic with sometimes unexpected behaviors.
For instance is `split=` is not specified it's currently always returning a DatasetDict. And if `split="train"` is given for example it's always returning a Dataset.
ricardorei (NONE) commented on 2021-01-14 10:34:35:
I also thought a bit about the fact that "sources" can't be added to the batch.. but changing that would require a lot more changes. And I agree that the idea of adding them as part of the references is not ideal. Conceptually they are not references.

I would keep it like this for now.. And in the future, work on a more consistent batch interface.
albertvillanova (MEMBER) commented on 2021-01-14 11:29:25:
Hi @lhoestq, I see...

Indeed I was also creating a draft for test_load, to clarify the expected behavior... ;)

So, for the command line:
```sh
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
the `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?
lhoestq (MEMBER) commented on 2021-01-14 13:18:13:
> So, for the command line:
> 
> ```shell
> python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
> ```
> 
> the `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?

Yes the cache_dir is used to download files and also so save the dataset arrow files.
This is indeed different from the path for dynamic modules.

I suggested to have `dynamic_module_path` as a parameter but actually this is the parent directory `hf_modules_cache` that we would need (it's the one that is passed to `init_dynamic_modules ` that we need to add to `sys.path`).

Currently it's already possible to override it using the env variable `HF_MODULES_CACHE` but we can imagine having it as a parameter as well.

This way the user controls both the `cache_dir` and the `hf_modules_cache` which are the two places used by the library to read/write stuff.
lhoestq (MEMBER) commented on 2021-01-14 16:17:24:
Alright now `csv`, `json`, `text` and `pandas` are "packaged datasets", i.e. they're part of the `datasets` package, which makes them available in offline mode without any change in terms of API:
```python
from datasets import load_dataset

d = load_dataset("csv", data_files=["path/to/data.csv"])
```

Instead of loading the dataset script from the module cache, it's loaded from inside the `datasets` package.

I updated the test to still be able to fetch the dummy data files for those datasets from `datasets/{text|csv|pandas|json}/dummy` in the repo.
lhoestq (MEMBER) commented on 2021-01-15 13:59:50:
I updated the PR:
- I removed the enable/disable fingerprinting function
- if caching is disabled arrow files are written in a temporary directory that is deleted when session closes
- the warning that is showed when hashing a transform fails is only showed once
- I added the `set_caching_enabled` function to the docs and explained the caching mechanism and its relation with fingerprinting

I would love to have some feedback :)
yjernite (MEMBER) commented on 2021-01-15 15:03:18:
> > * So we already have a keyword argument for `dataset1 = dataset1.map(..., in_place=True)`?
> 
> There's no such `in_place` parameter in map, what do you mean exactly ?

Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:

```
# case 1 - keep both cache files (dataset1 and dataset2)
dataset2 = dataset1.map(...)
# case 2 - keep only the new cache file
dataset1 = dataset1.map(..., in_place=True)
```

Case 1 returns a new reference using the new cache file, case 2 returns the same reference
lhoestq (MEMBER) commented on 2021-01-15 15:25:08:
> Sorry, that wasn't clear at all. I was responding to your previous comment about case 1 / case 2. I don't think the behavior should depend on the command, but we could have:
> 
> ```
> # case 1 - keep both cache files (dataset1 and dataset2)
> dataset2 = dataset1.map(...)
> # case 2 - keep only the new cache file
> dataset1 = dataset1.map(..., in_place=True)
> ```
> 
> Case 1 returns a new reference using the new cache file, case 2 returns the same reference

Ok I see !
`in_place` is a parameter that is used in general to designate a transform so I would name that differently (maybe `overwrite` or something like that).
Not sure if it's possible to update an already existing arrow file that is memory-mapped, let me check real quick.
Also it's possible to call `dataset2.cleanup_cache_files()` to delete the other cache files if we create a new one after the transform. Or even to get the cache file with `dataset1.cache_files` and let the user remove them by hand.

EDIT: updating an arrow file in place is not part of the current API of pyarrow, so we would have to make new files.
blmoistawinde (NONE) commented on 2021-01-17 03:31:32:
I encountered a simliar issue: `datasets` converted my float numpy array to `torch.float64` tensors, while many pytorch operations require `torch.float32` inputs and it's very troublesome. 

I tried @lhoestq 's solution, but since it's mixed with the preprocess function, it's not very intuitive. 

I just want to share another possible simpler solution: directly cast the dtype of the processed dataset.

Now I want to change the type of `labels` in `train_dataset` from float64 to float32, I can do this.

```
from datasets import Value, Sequence, Features
feats = train_dataset.features.copy()
feats['labels'].feature = Value(dtype='float32')
feats = Features(feats)
train_dataset.cast_(feats)
```
versae (CONTRIBUTOR) commented on 2021-01-18 10:47:02:
Quick question, @lhoestq. In this specific dataset, two special types `GPE_LOC` and `GPE_ORG` can easily be altered depending on the task, choosing either the more general `GPE` tag or the more specific `LOC`/`ORG` tags, conflating them with the other annotations of the same type. However, I have not found an easy way to implement that. Using splits or configs does not seem appropriate.
lhoestq (MEMBER) commented on 2021-01-18 14:16:54:
Hi :) 
The slicing API from https://huggingface.co/docs/datasets/splits.html doesn't shuffle the data.
You can shuffle and then take a subset of your dataset with
```python
# shuffle and take the first 100 examples
dataset = dataset.shuffle(seed=42).select(range(100))
```

You can find more information about shuffling and selecting rows in the documentation: https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
ghost (NONE) commented on 2021-01-18 14:45:34:
thank you so much

On Mon, Jan 18, 2021 at 3:17 PM Quentin Lhoest <notifications@github.com>
wrote:

> Hi :)
> The slicing API doesn't shuffle the data.
> You can shuffle and then take a subset of your dataset with
>
> # shuffle and take the first 100 examplesdataset = dataset.shuffle(seed=42).select(range(100))
>
> You can find more information about shuffling and selecting rows in the
> documentation:
> https://huggingface.co/docs/datasets/processing.html#selecting-sorting-shuffling-splitting-rows
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1747#issuecomment-762278134>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM5D5MDPLJGI4IG3UADS2Q7GPANCNFSM4WHLOZJQ>
> .
>
lhoestq (MEMBER) commented on 2021-01-18 17:58:16:
Currently it's only possible to define the features for the two columns `references` and `predictions`.
The data for these columns can then be passed to `metric.add_batch` and `metric.compute`.
Instead of defining more columns `text`, `offset_mapping` and `ground` you must include them in either references and predictions.

For example 
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence({
        "references_ids": datasets.Value("int32"),
        "offset_mapping": datasets.Value("int32"),
        'text': datasets.Value('string'),
        "ground": datasets.Value("int32")
    }),
})
```

Another option would be to simply have the two features like 
```python
features = datasets.Features({
    'predictions':datasets.Sequence(datasets.Value("int32")),
    "references": datasets.Sequence(datasets.Value("int32")),
})
```
and keep `offset_mapping`, `text` and `ground` as as parameters for the computation (i.e. kwargs when calling `metric.compute`).


What is the metric you would like to implement ?

I'm asking since we consider allowing additional fields as requested in the `Comet` metric (see PR and discussion [here](https://github.com/huggingface/datasets/pull/1577)) and I'd like to know if it's something that can be interesting for users.

What do you think ?
versae (CONTRIBUTOR) commented on 2021-01-19 15:38:14:
About the `GPE_LOC` and `GPE_ORG`. The original NorNE paper in which they published the dataset, does an evaluation on three different NER tag sets, one considering `GPE_LOC` and `GPE_ORG` as they are, another changing them to be just `GPE`, and another one by changing it to become `LOC` and `ORG`. The called these sets, `norne-full`, `norne-7`, and `norne-9`. What I would like is to provide a way for the user of this dataset to get `norne-7` and `norne-9` without having to duplicate the code.
lhoestq (MEMBER) commented on 2021-01-19 15:56:15:
Ok I see !
I guess you can have three configurations  `norne-full`, `norne-7` and `norne-9`.
Each config can have different feature types. You can simply check for the `self.config.name` in the `_info(self)` method and pick the right ClassLabel names accordingly. And then in `_generate_examples` as well you can check for `self.config.name` to know how to process the labels to yield either GPE_LOC/GPE_ORG, GPE or LOC/ORG
philschmid (CONTRIBUTOR) commented on 2021-01-19 16:20:08:
I created the documentation for `FileSystem Integration for cloud storage` with loading and saving datasets to/from a filesystem with an example of using `datasets.filesystem.S3Filesystem`. I added a note on the `Saving a processed dataset on disk and reload` saying that it is also possible to use other filesystems and cloud storages such as S3 with a link to the newly created documentation page from me. 
I Attach a screenshot of it here. 
![screencapture-localhost-5500-docs-build-html-filesystems-html-2021-01-19-17_16_10](https://user-images.githubusercontent.com/32632186/105062131-8d6a5c80-5a7a-11eb-90b0-f6128b758605.png)
versae (CONTRIBUTOR) commented on 2021-01-19 18:14:42:
But I'm already using the configurations for the different language
varieties. So you propose having something like `bokmaal`, `bokmaal-7`,
etc? Would there be a different way? If not, I'd be fine the corpus as it
is until we come up with a solution. Thanks in any case.

--
Sent using a cell-phone, so sorry for the typos and wrong auto-corrections.

On Tue, Jan 19, 2021, 4:56 PM Quentin Lhoest <notifications@github.com>
wrote:

> Ok I see !
> I guess you can have three configurations norne-full, norne-7 and norne-9.
> Each config can have different feature types. You can simply check for the
> self.config.name in the _info(self) method and pick the right ClassLabel
> names accordingly. And then in _generate_examples as well you can check
> for self.config.name to know how to process the labels to yield either
> GPE_LOC/GPE_ORG, GPE or LOC/ORG
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/1720#issuecomment-762936612>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABKLYOWNDBD76WZPJHFCWLS2WTTHANCNFSM4V6GSUQA>
> .
>
lhoestq (MEMBER) commented on 2021-01-20 13:37:21:
The first option about having configurations like `bokmaal-7`, `bokmaal-9` etc. would definitely work.

A second option would be to add a parameter `ner_tags_set` to `NorneConfig` and then one could load them with
```python
bokmaal_full = load_dataset("norne", "bokmaal", ner_tags_set="norne-full")
```
for example.

What do you think ?
Tomas0413 (NONE) commented on 2021-01-20 16:47:57:
Hello,

I ran https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh

And received this error:
```
100%|██████████| 407/407 [21:37<00:00,  3.44s/it]Traceback (most recent call last):
  File "run_ner.py", line 445, in <module>
    main()
  File "run_ner.py", line 398, in main
    results = trainer.evaluate()
  File "/data/2021/transformers/src/transformers/trainer.py", line 1470, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/data/2021/transformers/src/transformers/trainer.py", line 1622, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "run_ner.py", line 345, in compute_metrics
    results = metric.compute(predictions=true_predictions, references=true_labels)
  File "/usr/local/lib/python3.6/dist-packages/datasets/metric.py", line 398, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py", line 97, in _compute
    report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

I'm still trying multiple things to see if I can work around this, but I thought it might be useful to mention it here.

```
Name: transformers
Version: 4.3.0.dev0

Name: datasets
Version: 1.2.1
```
gchhablani (CONTRIBUTOR) commented on 2021-01-20 18:00:59:
Hi @lhoestq,

I am doing text segmentation and the metric is effectively dice score on character offsets. So I need to pass the actual spans and I want to be able to get the spans based on predictions using offset_mapping.

Including them in references seems like a good idea. I'll try it out and get back to you. If there's a better way to write a metric function for the same, please let me know.
lhoestq (MEMBER) commented on 2021-01-21 14:00:56:
By default the datasets loaded with `load_dataset` live on disk.
It's possible to load them in memory by using some transforms like `.map(..., keep_in_memory=True)`.

Small correction to @thomwolf 's comment above: currently we don't have the `keep_in_memory` parameter for `load_dataset` AFAIK but it would be nice to add it indeed :)
lhoestq (MEMBER) commented on 2021-01-21 17:19:08:
The wikipedia articles are processed using the `mwparserfromhell` library. Even if it works well in most cases, such issues can happen unfortunately. You can find the repo here: https://github.com/earwig/mwparserfromhell

There also exist other datasets based on wikipedia that were processed differently (and are often cleaner) such as `wiki40b`.
mariosasko (COLLABORATOR) commented on 2021-01-22 02:25:46:
Instead of:
```python
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
```
It should be:
```python
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32)
```

`batch_sampler` accepts a Sampler object or an Iterable, so you get an error.
EvanZ (NONE) commented on 2021-01-22 15:39:25:
@mariosasko I thought that would fix it, but now I'm getting a different error:

```
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py:851: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.tensor(x, **format_kwargs)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-3af1d82bf93a> in <module>()
      1 dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_size=32)
----> 2 next(iter(dataloader))

5 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: stack expects each tensor to be equal size, but got [7] at entry 0 and [10] at entry 1
```

Any thoughts what this means?I Do I need padding?
shuaihuaiyi (NONE) commented on 2021-01-23 10:26:22:
More information: `run_mlm.py` will raise same error when `data_args.line_by_line==True`

https://github.com/huggingface/transformers/blob/9152f16023b59d262b51573714b40325c8e49370/examples/language-modeling/run_mlm.py#L300
lhoestq (MEMBER) commented on 2021-01-24 15:16:54:
Hi ! Currently we don't have a way to `zip` datasets but we plan to add this soon :)
For now you'll need to use `map` to add the fields from one dataset to the other. See the comment here for more info : https://github.com/huggingface/datasets/issues/853#issuecomment-727872188
lhoestq (MEMBER) commented on 2021-01-25 10:23:57:
Multiprocessing in python require all the functions to be picklable. More specifically, functions need to be picklable with `dill`.

However objects like `typing.Union[str, NoneType]` are not picklable in python <3.7.
Can you try to update your python version to python>=3.7 ?
lhoestq (MEMBER) commented on 2021-01-25 10:29:05:
Note that you can pass any argument you would pass to `pandas.read_csv` as kwargs to `load_dataset`. For example you can do
```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=data_files, sep="\t")
```

for example to use a tab separator.

You can see the full list of arguments here: https://github.com/huggingface/datasets/blob/master/src/datasets/packaged_modules/csv/csv.py

(I've not found the list in the documentation though, we definitely must add them !)
lhoestq (MEMBER) commented on 2021-01-25 10:39:17:
Hi ! To avoid collisions you can specify a `experiment_id` when instantiating your metric using `load_metric`. It will replace "default_experiment" with the experiment id that you provide in the arrow filename. 

Also when two `experiment_id` collide we're supposed to detect it using our locking mechanism. Not sure why it didn't work in your case. Could you share some code that reproduces the issue ? This would help us investigate.
lamthuy (NONE) commented on 2021-01-25 11:25:08:
Thank you for your response. I fixed the issue by set "keep_in_memory=True" when load_metric. 
I cannot share the entire source code but below is the wrapper I wrote:

```python
class Evaluation:
    def __init__(self, metric='sacrebleu'):
        # self.metric = load_metric(metric, keep_in_memory=True)
        self.metric = load_metric(metric)

    def add(self, predictions, references):
        self.metric.add_batch(predictions=predictions, references=references)

    def compute(self):
        return self.metric.compute()['score']
```

Then call the given wrapper as follows:

```python
eval = Evaluation(metric='sacrebleu')
for query, candidates,  labels in tqdm(dataset):
    predictions = net.generate(query)
    references = [[s] for s in labels]
    eval.add(predictions, references)
    if n % 100 == 0:
        bleu += eval.compute()
        eval = Evaluation(metric='sacrebleu')