Title: Sequence of array not supported for most dtype
Author: qgallouedec (MEMBER)
State: closed
Created: 2023-06-08T18:18:07Z

### Describe the bug

Create a dataset composed of sequence of array fails for most dtypes (see code below).

### Steps to reproduce the bug

```python
from datasets import Sequence, Array2D, Features, Dataset

import numpy as np

for dtype in [
    "bool",  # ok
    "int8",  # failed
    "int16",  # failed
    "int32",  # failed
    "int64",  # ok
    "uint8",  # failed
    "uint16",  # failed
    "uint32",  # failed
    "uint64",  # failed
    "float16",  # failed
    "float32",  # failed
    "float64",  # ok
]:
    features = Features({"foo": Sequence(Array2D(dtype=dtype, shape=(2, 2)))})
    sequence = [
        [[1.0, 2.0], [3.0, 4.0]],
        [[5.0, 6.0], [7.0, 8.0]],
    ]
    array = np.array(sequence, dtype=dtype)
    try:
        dataset = Dataset.from_dict({"foo": [array]}, features=features)
    except Exception as e:
        print(f"Failed for dtype={dtype}")
```

Traceback for `dtype="int8"`:

```
Traceback (most recent call last):
  File "/home/qgallouedec/datasets/a.py", line 29, in <module>
    raise e
  File "/home/qgallouedec/datasets/a.py", line 26, in <module>
    dataset = Dataset.from_dict({"foo": [array]}, features=features)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 899, in from_dict
    pa_table = InMemoryTable.from_pydict(mapping=mapping)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 799, in from_pydict
    return cls(pa.Table.from_pydict(*args, **kwargs))
  File "pyarrow/table.pxi", line 3725, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 5254, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 350, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 236, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/arrow_writer.py", line 204, in __arrow_array__
    out = cast_array_to_feature(out, type, allow_number_to_str=not self.trying_type)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 2091, in cast_array_to_feature
    casted_values = _c(array.values, feature.feature)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 2139, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 1833, in wrapper
    return func(array, *args, **kwargs)
  File "/home/qgallouedec/env/lib/python3.10/site-packages/datasets/table.py", line 1967, in array_cast
    return pa_type.wrap_array(array)
  File "pyarrow/types.pxi", line 879, in pyarrow.lib.BaseExtensionType.wrap_array
TypeError: Incompatible storage type for extension<arrow.py_extension_type<Array2DExtensionType>>: expected list<item: list<item: int8>>, got list<item: list<item: int64>>
```

### Expected behavior

Not to fail.

### Environment info


- Python 3.10.6
- datasets: master branch
- Numpy: 1.23.4

Comments:
Related, `float16` is the only dtype not supported by `Array2D` (probably by every `ArrayND`):

```python
from datasets import Array2D, Features, Dataset

import numpy as np

for dtype in [
    "bool",  # ok
    "int8",  # ok
    "int16",  # ok
    "int32",  # ok
    "int64",  # ok
    "uint8",  # ok
    "uint16",  # ok
    "uint32",  # ok
    "uint64",  # ok
    "float16",  # failed
    "float32",  # ok
    "float64",  # ok
]:
    features = Features({"foo": Array2D(dtype=dtype, shape=(3, 4))})
    array = np.zeros((3, 4), dtype=dtype)
    try:
        dataset = Dataset.from_dict({"foo": [array]}, features=features)
    except Exception as e:
        print(f"Failed for dtype={dtype}")
```

Here's something I can't explain:

When an array is encoded in the `from_dict` method, the numpy array is converted to a list (thus losing the original dtype, which is transfromed to the nearest builtin Python type)

https://github.com/huggingface/datasets/blob/6ee61e6e695b1df9f232d47faf3a5e2b30b33737/src/datasets/features/features.py#L524-L525

However, later on, this same data is written to memory, and it seems authorized that the data is an array (or in this case, a list of arrays). 

https://github.com/huggingface/datasets/blob/6ee61e6e695b1df9f232d47faf3a5e2b30b33737/src/datasets/arrow_writer.py#L185-L186

So the question is: why convert it to a Python list? This seems to be quite expensive both in terms of write time (all data is copied) and memory (e.g., an int8 is converted to an int64).

Finally, if I try to remove this step, it solves all the previous problems, and it seems to me that it doesn't break anything (the CI passes without problem).

Arrow only support 1d numpy arrays, so we convert multidim arrays to lists of 1s arrays (and keep the dtype).

Though you noticed that it's concerting to lists and lose the dtype. If it's the case then it's a bug.

Ok the conversion to list shouldn't be there indeed ! Could you open a PR to remove it ?