miyamonz (CONTRIBUTOR) commented on 2021-02-01 06:35:32:
Thanks for your reply! 
I understood.

I tried again with installing apache-beam, add ` beam_runner="DirectRunner"` and an anther `mwparserfromhell` is also required so I installed it.
but, it also failed. It exited 1 without error message.

```py
import datasets
# BTW, 20200501.ja doesn't exist at wikipedia, so I specified date argument
wiki = datasets.load_dataset("wikipedia", language="ja", date="20210120", cache_dir="./datasets", beam_runner="DirectRunner")
print(wiki)
```
and its log is below
```
Using custom data configuration 20210120.ja
Downloading and preparing dataset wikipedia/20210120.ja-date=20210120,language=ja (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to ./datasets/wikipedia/20210120.ja-date=20210120,language=ja/0.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63...
Killed
```

I also tried on another machine because it may caused by insufficient resources.
```
$ python main.py
Using custom data configuration 20210120.ja
Downloading and preparing dataset wikipedia/20210120.ja-date=20210120,language=ja (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to ./datasets/wikipedia/20210120.ja-date=20210120,language=ja/0.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63...

Traceback (most recent call last):
  File "main.py", line 3, in <module>
    wiki = datasets.load_dataset("wikipedia", language="ja", date="20210120", cache_dir="./datasets", beam_runner="DirectRunner")
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/datasets/builder.py", line 1069, in _download_and_prepare
    pipeline_results = pipeline.run()
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/pipeline.py", line 561, in run
    return self.runner.run_pipeline(self, self._options)
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/direct/direct_runner.py", line 126, in run_pipeline
    return runner.run_pipeline(pipeline, options)
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 182, in run_pipeline
    self._latest_run_result = self.run_via_runner_api(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 193, in run_via_runner_api
    return self.run_stages(stage_context, stages)
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 358, in run_stages
    stage_results = self._run_stage(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 549, in _run_stage
    last_result, deferred_inputs, fired_timers = self._run_bundle(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 595, in _run_bundle
    result, splits = bundle_manager.process_bundle(
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 888, in process_bundle
    self._send_input_to_worker(process_bundle_id, transform_id, elements)
  File "/home/miyamonz/.cache/pypoetry/virtualenvs/try-datasets-4t4JWXxu-py3.8/lib/python3.8/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 765, in _send_input_to_worker
    data_out.write(byte_stream)
  File "apache_beam/coders/stream.pyx", line 42, in apache_beam.coders.stream.OutputStream.write
  File "apache_beam/coders/stream.pyx", line 47, in apache_beam.coders.stream.OutputStream.write
  File "apache_beam/coders/stream.pyx", line 109, in apache_beam.coders.stream.OutputStream.extend
AssertionError: OutputStream realloc failed.
```