Dataset.to_json consumes excessive memory, appears to not be a streaming operation
Opened by ycq0125 (open)

### Describe the bug

When exporting a Dataset object to a JSON Lines file using the .to_json(lines=True) method, the process consumes a very large amount of memory. The memory usage is proportional to the size of the entire Dataset object being saved, rather than being a low, constant memory operation.

This behavior is unexpected, as the JSONL format is line-oriented and ideally suited for streaming writes. This issue can easily lead to Out-of-Memory (OOM) errors when exporting large datasets, especially in memory-constrained environments like Docker containers.

<img width="1343" height="329" alt="Image" src="https://github.com/user-attachments/assets/518b4263-ad12-422d-9672-28ffe97240ce" />

### Steps to reproduce the bug

```
import os
from datasets import load_dataset, Dataset
from loguru import logger

# A public dataset to test with
REPO_ID = "adam89/TinyStoriesChinese"
SUBSET = "default"
SPLIT = "train"
NUM_ROWS_TO_LOAD = 10  # Use a reasonably large number to see the memory spike


def run_test():
    """Loads data into memory and then saves it, triggering the memory issue."""
    logger.info("Step 1: Loading data into an in-memory Dataset object...")

    # Create an in-memory Dataset object from a stream
    # This simulates having a processed dataset ready to be saved
    iterable_dataset = load_dataset(REPO_ID, name=SUBSET, split=SPLIT, streaming=True)
    limited_stream = iterable_dataset.take(NUM_ROWS_TO_LOAD)
    in_memory_dataset = Dataset.from_generator(limited_stream.__iter__)

    logger.info(f"Dataset with {len(in_memory_dataset)} rows created in memory.")

    output_path = "./test_output.jsonl"
    logger.info(f"Step 2: Saving the dataset to {output_path} using .to_json()...")
    logger.info("Please monitor memory usage during this step.")

    # This is the step that causes the massive memory allocation
    in_memory_dataset.to_json(output_path, force_ascii=False)

    logger.info("Save operation complete.")
    os.remove(output_path)


if __name__ == "__main__":
    # To see the memory usage clearly, run this script with a memory profiler:
    # python -m memray run your_script_name.py
    # python -m memray tree xxx.bin
    run_test()

```

### Expected behavior

I would expect the .to_json(lines=True) method to be a memory-efficient, streaming operation. The memory usage should remain low and relatively constant, as data is converted and written to the file line-by-line or in small batches. The memory footprint should not be proportional to the total number of rows in the in_memory_dataset.

### Environment info

datasets version:3.6.0
Python version:3.9.18
os:macOS 15.3.1 (arm64)

Comments: Hi ! to_json is memory efficient and writes the data by batch:

https://github.com/huggingface/datasets/blob/d9861d86be222884dabbd534a2db770c70c9b558/src/datasets/io/json.py#L153-L159

What memory are you mesuring ? If you are mesuring RSS, it is likely that it counts the memory mapped data of the dataset. Memory mapped data are loaded as physical memory when accessed and are automatically discarded when your OS needs more memory, and therefore doesn't OOM.