Title: [Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.
Author: JasonCZH4 (NONE)
State: closed
Created: 2024-12-19T14:31:27Z

### Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.


### Steps to reproduce the bug

# First
I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict `{text:'aaa'}`.
Their path are `/data/train.json`,  `/data/val.json`,  `/data/test.json`
I load dataset with `data_files` argument:
```py
files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})
```
However, If I load dataset with `data_dir` argument:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Two results are not the same. Their behaviors are not equal, even if the statement [here](https://github.com/huggingface/datasets/blob/d0c152a979d91cc34b605c0298aebc650ab7dd27/src/datasets/load.py#L1790) said that their behaviors are equal. 


# Second
If some filename include 'test' while others do not, `load_dataset` only return `test` dataset and others files are **abandoned**.
Given two files named `test.json` and `1.json`
Each one has a simple dict `{text:'aaa'}`.
I load the dataset using:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
Only `test` is returned, `1.json` is missing:
```py
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Things do not change even I manually set `split='train'`

 



### Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will `Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration`, I hope I can manually decide whether to do so. Sometimes users may accidentally put a `test` string in the filename but they just want a single `train` dataset. If the number of files in `data_dir` is huge, it's not easy to find out what cause the second situation metioned above.

### Environment info

datasets==3.2.0
Ubuntu18.84

Comments:
Hi ! `data_files` with a list is equivalent to `data_files={"train": data_files}` with a train test only.

When no split are specified, they are inferred based on file names, and files with no apparent split are ignored

Thanks for your reply!
`files with no apparent split are ignored`. Is there a option that I can choose to ignored it or not as I mention aboved? Thanks!

To include all the files, the best way is to pass `data_files` yourself. There is no option to disable split detection at the moment

Thanks! I hope you guys can consider adding this option in the future. :)