Repository: tensorflow/tfx

Issue #243: Support more stats_options through config.
Created: 2019-06-19T17:04:34Z
The current (0.13.0) implementation of the StatisticsGen component only uses the default stats_options: stats_options = tensorflow_data_validation.statistics.stats_options.StatsOptions()
There is a TODO statement saying that this should be configurable.
There should be a default to the current behavior.

Duplicate issue #505: Error in StatisticsGen using Dataflow — Shuffle key too large
Created: 2019-08-19T21:16:01Z
My TFX pipeline on kubeflow fails at the StatisticsGen component with the following error. One of my features contains long strings potentially over 2MB, which could be the cause of the error according to StackOverflow (https://stackoverflow.com/a/49101242).
TFX docker image used: tensorflow/tfx:0.14.0.dev20190819 from https://hub.docker.com/r/tensorflow/tfx/tags
Kubeflow version: v0.5.0 on GKE, deployed using http://deploy.kubeflow.cloud
TFX built from 5e01e6a
Traceback (most recent call last):
  File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 106, in <module>
    main()
  File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 102, in main
    executor.run()
  File "/opt/venv/lib/python3.5/site-packages/tfx/orchestration/kubeflow/executor_wrappers.py", line 123, in run
    self._exec_properties)
  File "/opt/venv/lib/python3.5/site-packages/tfx/components/statistics_gen/executor.py", line 89, in Do
    split, output_uri))
  File "/opt/venv/lib/python3.5/site-packages/apache_beam/pipeline.py", line 426, in __exit__
    self.run().wait_until_finish()
  File "/opt/venv/lib/python3.5/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1338, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 649, in do_work
    work_executor.execute()
  File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 178, in execute
    op.finish()
  File "apache_beam/runners/worker/operations.py", line 874, in apache_beam.runners.worker.operations.PGBKCVOperation.finish
  File "apache_beam/runners/worker/operations.py", line 876, in apache_beam.runners.worker.operations.PGBKCVOperation.finish
  File "apache_beam/runners/worker/operations.py", line 887, in apache_beam.runners.worker.operations.PGBKCVOperation.output_key
  File "apache_beam/runners/worker/operations.py", line 256, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 143, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "dataflow_worker/shuffle_operations.py", line 188, in dataflow_worker.shuffle_operations.ReifyTimestampAndWindowsOperation.process
  File "dataflow_worker/shuffle_operations.py", line 192, in dataflow_worker.shuffle_operations.ReifyTimestampAndWindowsOperation.process
  File "apache_beam/runners/worker/operations.py", line 256, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 143, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "dataflow_worker/shuffle_operations.py", line 151, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.process
  File "dataflow_worker/shuffle_operations.py", line 171, in dataflow_worker.shuffle_operations.ShuffleWriteOperation.process
  File "/usr/local/lib/python3.5/site-packages/dataflow_worker/shuffle.py", line 620, in Write
    self.writer.Write(self.stream.getvalue())
  File "third_party/windmill/shuffle/python/shuffle_client.pyx", line 176, in shuffle_client.PyShuffleWriter.Write
OSError: Shuffle write failed: b'INVALID_ARGUMENT: Shuffle key too large:1745410 > 1572864'

Closing comment (2019-08-27T15:13:41Z): Okay, cool. So we need to expose StatsOptions config, then, which I believe is tracked by #243.