Title: `Sequence(Features(...))` causes PyArrow cast error in `load_dataset` despite correct schema.
Author: AHS-uni (NONE)
State: open
Created: 2025-05-29T22:53:36Z

### Description

When loading a dataset with a field declared as a list of structs using `Sequence(Features(...))`, `load_dataset` incorrectly infers the field as a plain `struct<...>` instead of a `list<struct<...>>`. This leads to the following error:

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<id: string, data: string>> to struct using function cast_struct
```

This occurs even when the `features` schema is explicitly provided and the dataset format supports nested structures natively (e.g., JSON, JSONL).

---

###  Minimal Reproduction

[Colab Link.](https://colab.research.google.com/drive/1FZPQy6TP3jVd4B3mYKyfQaWNuOAvljUq?usp=sharing)

#### Dataset

```python
data = [
    {
        "list": [
            {"id": "example1", "data": "text"},
        ]
    },
]
```

#### Schema

```python
from datasets import Features, Sequence, Value

item = Features({
    "id":   Value("string"),
    "data": Value("string"),
})

features = Features({
    "list": Sequence(item),
})
```

---

###  Tested File Formats

The same schema was tested across different formats:

| Format    | Method                      | Result              |
| --------- | --------------------------- | ------------------- |
| JSONL     | `load_dataset("json", ...)` |  Arrow cast error  |
| JSON      | `load_dataset("json", ...)` |  Arrow cast error  |
| In-memory | `Dataset.from_list(...)`    | Works as expected |

The issue seems not to be in the schema or the data, but in how `load_dataset()` handles the `Sequence(Features(...))` pattern when parsing from files (specifically JSON and JSONL).

---

### Expected Behavior

If `features` is explicitly defined as:

```python
Features({"list": Sequence(Features({...}))})
```

Then the data should load correctly across all backends — including from JSON and JSONL — without any Arrow casting errors. This works correctly when loading from memory via `Dataset.from_list`.

---

###  Environment

* `datasets`: 3.6.0
* `pyarrow`: 20.0.0
* Python: 3.12.10
* OS: Ubuntu 24.04.2 LTS
* Notebook: \[Colab test notebook available]

---

Comments:
Hi @lhoestq 

Could you help confirm whether this qualifies as a bug?

It looks like the issue stems from how `Sequence(Features(...))` is interpreted as a plain struct during schema inference, which leads to a mismatch when casting with PyArrow (especially with nested structs inside lists). From the description, this seems like an inconsistency with expected behavior.

If confirmed, I’d be happy to take a shot at investigating and potentially submitting a fix.

Also looping in @AHS-uni  — could you kindly share a minimal JSONL example that reproduces this?

Thanks!

Hello @Flink-ddd 

I updated the minimal example and included both JSON and JSONL minimal examples in the Colab notebook. 

Here is the minimal  JSON file for convenience (can't upload JSONL files).

[mini.json](https://github.com/user-attachments/files/20535145/mini.json)

I've also found a number of issues which describe a similar problem:

[7569](https://github.com/huggingface/datasets/issues/7569) (Open)
[7137](https://github.com/huggingface/datasets/issues/7137) (Open)
[7501](https://github.com/huggingface/datasets/issues/7501) (Closed)
[2434](https://github.com/huggingface/datasets/issues/2434) (Closed)

The closed issues don't really address the problem (IMO). [7501](https://github.com/huggingface/datasets/issues/7501) provides a workaround (using a Python list instead of `Sequence`), but it seem precarious. 

Hi ! `Sequence({...})` corresponds to a struct of lists ([docs](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/main_classes#datasets.Features)). This come from Tensorflow Datasets.

If you want to use a list of structs, you should use `[{...}]`, e.g.

```python
item = {
    "id":   Value("string"),
    "data": Value("string"),
}

features = Features({
    "list": [item],
})
```