rachel521/github-issues-simple huggingface dataset:
Dataset Viewer issue for bigscience-biomedical/biosses
Opened by galtay (closed)

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) . 
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes

Comments: Possibly not related to the dataset viewer in itself. cc @huggingface/datasets.

In particular, I think that the import of bigbiohub is not working here: https://huggingface.co/datasets/bigscience-biomedical/biosses/blob/main/biosses.py#L29 (requires a relative path?)

```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names('bigscience-biomedical/biosses')
Downloading builder script: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.00k/8.00k [00:00<00:00, 7.47MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 289, in get_dataset_config_names
    dataset_module = dataset_module_factory(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1247, in dataset_module_factory
    raise e1 from None
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1220, in dataset_module_factory
    return HubDatasetModuleFactoryWithScript(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 931, in get_module
    local_imports = _download_additional_modules(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 215, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use bigscience-biomedical/biosses, you need to install the following dependency: bigbiohub.
Please install it using 'pip install bigbiohub' for instance'
```

Opened a PR here to (hopefully) fix the dataset script: https://huggingface.co/datasets/bigscience-biomedical/biosses/discussions/1/files

thanks for taking a look @severo . agree this isn't related to dataset viewer (sorry just clicked on the auto issue creator). also thanks @lhoestq , I see the format to use for relative imports. was a bit confused b/c it seems to be working here 

https://huggingface.co/datasets/bigscience-biomedical/scitail/blob/main/scitail.py#L31

I'll try this PR a see what happens. 

closing as I think the issue is relative imports and attempting to read json files directly in the repo (thanks again @lhoestq )
Adding ANERcorp-CAMeLLab dataset
Opened by vitalyshalumov (open)


Comments: Adding ANERcorp dataset

## Adding a Dataset
- **Name:** *ANERcorp-CAMeLLab*
- **Description:** *Since its creation in 2008, the ANERcorp dataset (Benajiba & Rosso, 2008) has been a standard reference used by Arabic named entity recognition researchers around the world. However, over time, this dataset was copied over from user to user, modified slightly here and there, and split in many different configurations that made it hard to compare fairly across papers and systems.

In 2020, a group of researchers from CAMeL Lab (Habash, Alhafni and Oudah), and Mind Lab (Antoun and Baly) met with the creator of the corpus, Yassine Benajiba, to consult with him and collectively agree on an exact split, and accepted minor corrections from the original dataset. Bashar Alhafni from CAMeL Lab working with Nizar Habash implemented the decisions provided in this release.*

- **Paper:** *(a) Benajiba, Yassine, Paolo Rosso, and José Miguel Benedí Ruiz. "Anersys: An Arabic named entity recognition system based on maximum entropy." In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 143-153. Springer, Berlin, Heidelberg, 2007.

(b)Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. "CAMeL Tools: An Open Source Python Toolkit, for Arabic Natural Language Processing." In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, 2020.*
- **Data:** *https://camel.abudhabi.nyu.edu/anercorp/*
- **Motivation:** This is the standard dataset for evaluating NER performance in Arabic*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl
Opened by JuanCarlosMartinezSevilla (closed)

### Describe the bug

Hi everyone, I uploaded my dataset https://huggingface.co/datasets/PRAIG/SMB a few months ago while I was waiting for a conference acceptance response. Without modifying anything in the dataset repository now the Dataset viewer is not rendering the metadata.jsonl annotations, neither it is being downloaded when using load_dataset. Can you please help? Thank you in advance.

### Steps to reproduce the bug

from datasets import load_dataset
ds = load_dataset("PRAIG/SMB")
ds = ds["train"]

### Expected behavior

 It is expected to have all the metadata available in the jsonl file. Fields like: "score_id", "original_width", "original_height", "regions"... among others.

### Environment info

datasets==3.6.0, python 3.13.3 (but he problem is already in the huggingface dataset page)

Comments: Maybe its been a recent update, but i can manage to load the metadata.jsonl separately from the images with:

```
metadata = load_dataset("PRAIG/SMB", split="train", data_files=["*.jsonl"])
images = load_dataset("PRAIG/SMB", split="train")
```
Do you know it this is an expected behaviour? This makes my dataset viewer to only load the images without the labeling of metadata.jsonl.

Thanks

Hi ! this is because we now expect the metadata file to be inside the directory named after the split "train" (this way each split can have its own metadata and can be loaded independently)

You can fix that by configuring it explicitly in the dataset's README.md header:

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - "train/**/*.png"
    - "metadata.jsonl"
```

(or by moving the metadata.jsonl in train/ but in this case you also have to modify the content of the JSONL to fix the relative paths to the images)

Thank you very much, dataset viewer is already working as expected!!
Added computer vision tasks
Opened by merveenoyan (closed)

Added various image processing/computer vision tasks.

Comments: Looks great, thanks ! If the 3d ones are really rare we can remove them for now.

And I can see that `object-detection` and `semantic-segmentation` are both task categories (top-level) and task ids (bottom-level). Maybe there's a way to group them and have less granularity for the task categories. For example `speech-processing` is a high level task category. What do you think ?

We can still update the list of tasks later if needed when we have more vision datasets


@lhoestq  @osanseviero I used the categories (there were main ones and subcategories) in the paperswithcode, I got rid of some of them that could be too granular. I can put it there if you'd like (I'll wait for your reply before committing it again)

We can ignore the ones that are too granular IMO. What we did for audio tasks is to have them all under "audio-processing". Maybe we can do the same here for now until we have more comprehensive tasks/applications ?

Following the discussion in (private) https://github.com/huggingface/moon-landing/issues/2020, what do you think of aligning the top level tasks list with the model tasks taxonomy ?

* Image Classification
* Object Detection
* Image Segmentation
* Text-to-Image
* Image-to-Text


I moved it to [a branch](https://github.com/huggingface/datasets/pull/3800) for ease.
Ensure ConcatenationTable.cast uses target_schema metadata
Opened by dtuit (closed)

Currently, `ConcatenationTable.cast` does not use target_schema metadata when casting subtables. This causes an issue when using cast_column and the underlying table is a ConcatenationTable.

Code example of where issue arrises:
```
from datasets import Dataset, Image

column1 = [0, 1]
image_paths = ['/images/image1.jpg', '/images/image2.jpg']

ds = Dataset.from_dict({"column1": column1})
ds = ds.add_column("image", image_paths)
ds.cast_column("image", Image()) # Fails here
```
Output
```
...
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```


Comments: Hi @lhoestq, Thanks for the detailed comment. I've tested the suggested approach and can confirm it works for the testcase outlined above! The PR is updated with the changes.

_The documentation is not available anymore as the PR was closed or merged._
Cache problem in the `load_dataset` method for local compressed file(s)
Opened by SaulLu (open)

## Describe the bug

Cache problem in the `load_dataset` method: when modifying a compressed file in a local folder `load_dataset` doesn't detect the change and load the previous version.

## Steps to reproduce the bug

To test it directly, I have prepared a [Google Colaboratory notebook](https://colab.research.google.com/drive/11Em_Amoc-aPGhSBIkSHU2AvEh24nVayy?usp=sharing) that shows this behavior. 

For this example, I have created a toy dataset at: https://huggingface.co/datasets/SaulLu/toy_struc_dataset

This dataset is composed of two versions: 
- v1 on commit `a6beb46` which has a single example `{'id': 1, 'value': {'tag': 'a', 'value': 1}}` in file `train.jsonl.gz`
- v2 on commit `e7935f4` (`main` head) which has a single example `{'attr': 1, 'id': 1, 'value': 'a'}` in file `train.jsonl.gz`

With a terminal, we can start to get the v1 version of the dataset
```bash
git lfs install
git clone https://huggingface.co/datasets/SaulLu/toy_struc_dataset
cd toy_struc_dataset
git checkout a6beb46
```
Then we can load it with python and look at the content:
```python
from datasets import load_dataset

path = "/content/toy_struc_dataset"
dataset = load_dataset(path, data_files={"train": "*.jsonl.gz"})
print(dataset["train"][0])
```
Output
```
{'id': 1, 'value': {'tag': 'a', 'value': 1}} # This is the example in v1
```

With a terminal, we can now start to get the v1 version of the dataset
```bash
git checkout main
```
Then we can load it with python and look at the content:
```python
from datasets import load_dataset

path = "/content/toy_struc_dataset"
dataset = load_dataset(path, data_files={"train": "*.jsonl.gz"})
print(dataset["train"][0])
```
Output
```
{'id': 1, 'value': {'tag': 'a', 'value': 1}} # This is the example in v1 (not v2)
```
## Expected results
The last output should have been 
```
{"id":1, "value": "a", "attr": 1} # This is the example in v2
```

## Ideas

As discussed offline with Quentin, if the cache hash was ever sensitive to changes in a compressed file we would probably not have the problem anymore.

This situation leads me to suggest 2 other features:
- to also have an `load_from_cache_file` argument in the "load_dataset" method
- to reorganize the cache so that we can delete the caches related to a dataset (cf issue #ToBeFilledSoon)

And thanks again for this great library :hugs: 

## Environment info
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


Comments: The problem is still present. 
One solution would be to add the `download_mode="force_redownload"` argument to load_dataset. 
However, doing so may lead to a `DatasetGenerationError: An error occurred while generating the dataset`. To mitigate, just do:
`rm -r ~/.cache/huggingface/datasets/*`
Errors loading dataset using fs = a gcsfs.GCSFileSystem
Opened by dconatha (open)

## Describe the bug

Cannot load dataset using a `gcsfs.GCSFileSystem`.  I'm not sure if this should be a bug in `gcsfs` or here...

Basically what seems to be happening is that since datasets saves datasets as folders and folders aren't "real objects" in gcs, gcsfs raises a 404 error.  There are workarounds if you use gcsfs directly to download the file, but as is I can't get `load_from_disk` to work.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# load some dataset
dataset = load_dataset("squad", split="train")

# save it to gcs
import gcsfs
fs = gcsfs.GCSFileSystem(project="my-gs-project")
dataset.save_to_disk("gs://my-bucket/squad", fs=fs)

# try to load it from gcs
from datasets import load_from_disk
dataset2 = load_from_disk("my-bucket/squad", fs=fs)
```


## Expected results
`dataset2` would be a copy of `dataset` but loaded from my bucket.

## Actual results
Long traceback but essentially it's a 404 error from gcsfs saying the object `my-bucket/squad` doesn't exist when this is called:
https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L977

This is because there is no actual object called `my-bucket/squad`, there are objects called `my-bucket/squad/dataset.arrow`, etc.

Note that *this* works fine, since it's explicitly saying "download all the objects with this prefix":
```python
            fs.download(src_dataset_path + "/*", dataset_path.as_posix(), recursive=True)
```
For example, I can do a workaround this way:
```python
import tempfile

with tempfile.TemporaryDirectory() as temppath:
    fs.download("gs://my-bucket/squad/*", temppath)
    dataset2 = load_from_disk(temppath)
```

It's unclear to me if it's `gcsfs`'s responsibility to say "hey that's folder not a file, I should try to get objects inside of it not the object itself", or if that's `datasets`'s responsibility... I'm leaning towards the latter since you're never loading a dataset from one file using this function/method, only a dataset folder?

Another minor thing that should maybe should be rolled into this bug...

https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L968

These fail if you pass in a `gs://` path, e.g.

```python
dataset2 = load_from_disk("gs://my-bucket/squad", fs=fs)
```
Because at this point, `dataset_info_path` is `gs:/my-bucket/squad/dataset_info.json`, gcsfs throws a:
```
Invalid bucket name: 'gs:'
```
error



## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: macOS Big Sur 11.6
- Python version: 3.7.12
- PyArrow version: 5.0.0
Some links to canonical datasets in the docs are outdated
Opened by polinaeterna (closed)

As we don't have canonical datasets in the github repo anymore, some old links to them doesn't work. I don't know how many of them are there, I found link to SuperGlue here: https://huggingface.co/docs/datasets/dataset_script#multiple-configurations, probably there are more of them. These links should be replaced by links to the corresponding datasets on the Hub. 

Comments: Thanks for catching this, I can go through the docs and replace the links to their corresponding datasets on the Hub!
Graceful Error Handling for cast_column("image", Image(decode=True)) in Hugging Face Datasets
Opened by ganiket19 (open)

### Feature request

Currently, when using dataset.cast_column("image", Image(decode=True)), the pipeline throws an error and halts if any image in the dataset is invalid or corrupted (e.g., truncated files, incorrect formats, unreachable URLs). This behavior disrupts large-scale processing where a few faulty samples are common.
reference : https://discuss.huggingface.co/t/handle-errors-when-loading-images-404-corrupted-etc/50318/5
                  https://discuss.huggingface.co/t/handling-non-existing-url-in-image-dataset-while-cast-column/69185
                  
Proposed Feature

Introduce a mechanism (e.g., a continue_on_error=True flag or global error handling mode) in Image(decode=True) that:

Skips invalid images and sets them as None, or

Logs the error but allows the rest of the dataset to be processed without interruption.


Example Usage

from datasets import load_dataset, Image

dataset = load_dataset("my_dataset")
dataset = dataset.cast_column("image", Image(decode=True, continue_on_error=True))

Benefits

Ensures robust large-scale image dataset processing.

Improves developer productivity by avoiding custom retry/error-handling code.

Aligns with best practices in dataset preprocessing pipelines that tolerate minor data corruption.


Potential Implementation Options

Internally wrap the decoding in a try/except block.

Return None or a placeholder on failure.

Optionally allow custom error callbacks or logging.

### Motivation

Robustness: Large-scale image datasets often contain a small fraction of corrupt files or unreachable URLs. Halting on the first error forces users to write custom workarounds or preprocess externally.

Simplicity: A built-in flag removes boilerplate try/except logic around every decode step.

Performance: Skipping invalid samples inline is more efficient than a two-pass approach (filter then decode).

### Your contribution


1. API Change
Extend datasets.features.Image(decode=True) to accept continue_on_error: bool = False.

2. Behavior
If continue_on_error=False (default), maintain current behavior: any decode error raises an exception.
If continue_on_error=True, wrap decode logic in try/except:
On success: store the decoded image.
On failure: log a warning (e.g., via logging.warning) and set the field to None (or a sentinel value).


3. Optional Enhancements
Allow a callback hook:
Image(decode=True, continue_on_error=True, on_error=lambda idx, url, exc: ...)
Emit metrics or counts of skipped images.

Comments: Hi! This is now handled in PR #7638

Thank you for implementing the suggestion it would be great help in our use case.
First draft of the docs for TF + Datasets
Opened by Rocketknight1 (closed)

I might cc a few of the other TF people to take a look when this is closer to being finished, but it's still a draft for now.

Comments: _The documentation is not available anymore as the PR was closed or merged._

Some links are still missing I think :)

This is probably quite close to being ready, so cc some TF people @gante @amyeroberts @merveenoyan just so they see it! No need for a full review, but if you have any comments or suggestions feel free.

Thanks ! We plan to make a new release later today for `to_tf_dataset` FYI, so I think we can merge it soon and include this documentation in the new release
[Tests] run local tests as default
Opened by patrickvonplaten (closed)

This PR also enables local tests by default

I think it's safer for now to enable both local and aws tests for every commit. The problem currently is that when we do a PR to add a dataset, the dataset is not yet on AWS on therefore not tested on the PR itself. Thus the PR will always be green even if the datasets are not correct. This PR aims at fixing this.

## Suggestion on how to commit to the repo from now on:
Now since the repo is "online", I think we should adopt a couple of best practices:
1) - No direct committing to the repo anymore. Every change should be opened in a PR and be well documented so that we can find it later
2) - Every PR has to be reviewed by at least x people (I guess @thomwolf you should decide here) because we now have to be much more careful when doing changes to the API for backward compatibility, etc...


Comments: You are right and I think those are usual best practice :) I'm 100% fine with this^^

Merging this for now to unblock other PRs.
Allow load_dataset to take a working dir for intermediate data
Opened by lu-wang-dl (open)

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.

Comments: Hi ! AFAIK a dataset must be present on a local disk to be able to efficiently memory map the datasets Arrow files. What makes you think that it is possible to load from a cloud storage and have good performance ?

Anyway it's already possible to download_and_prepare a dataset as Arrow files in a cloud storage with:
```python
builder = load_dataset_builder(..., cache_dir="/temp/dir")
builder.download_and_prepare("/cloud_dir")
```

but then 
```python
ds = builder.as_dataset()
```
would fail if "/cloud_dir" is not a local directory.

In my use case, I am trying to mount the S3 bucket as local system with S3FS-FUSE / [goofys](https://github.com/kahing/goofys). I want to use S3 to save the download data and save checkpoint for training for persistent. Setting the s3 location as cache directory is not fast enough. That is why I want to set a work directory for temp data for memory map and only save the final result to s3 cache. 

You can try setting `HF_DATASETS_DOWNLOADED_DATASETS_PATH` and `HF_DATASETS_EXTRACTED_DATASETS_PATH` to S3, and `HF_DATASETS_CACHE` to your local disk.

This way all your downloaded and extracted data are on your mounted S3, but the datasets Arrow files are on your local disk

If we hope to also persist the Arrow files on the mounted S3 but work with the efficiency of local disk, is there any recommended way to do this, other than copying the Arrow files from local disk to S3?
Datasets Unknown Keyword Argument Error - task_templates
Opened by varungupta31 (closed)

### Describe the bug

Issue

```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```

Gives error

```
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
```

A simple downgrade to lower `datasets v 2.21.0` solves it.




### Steps to reproduce the bug

1. `pip install datsets`
2.
```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```



### Expected behavior

Should load the dataset correctly.

### Environment info

- Datasets version  `3.0.0`
- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.35.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes


Comments: Thanks, for reporting.

We have been fixing most Hub datasets to remove the deprecated (and now non-supported) task templates, but we missed the "facebook/winoground".

It is fixed now: https://huggingface.co/datasets/facebook/winoground/discussions/8



Hello @albertvillanova 

I got the same error while loading this dataset: https://huggingface.co/datasets/alaleye/aloresb...

How can I fix it ? 
Thanks

I am getting the same error on the below code, any fix to this ?

```
from datasets import load_dataset

minds = load_dataset("PolyAI/minds14", name="en-AU", split="train")
minds
```