Repository: tensorflow/tensor2tensor

Issue #266: Evaluation failed when training with multiple work gpus
Created: 2017-08-30T23:53:29Z
I set worker_gpu to 2 and use 2 gpus in the same node. training is completely fine. But evaluation fails with this error
"
tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 47) and num_split 2
[[Node: split_2 = Split[T=DT_INT32, num_split=2, _device="/job:localhost/replica:0/task:0/cpu:0"](split_2/split_dim, input_reader/ExpandDims_3/_1823)]]
[[Node: split_2/_1825 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1113_split_2", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:1"]]
Caused by op u'split_2', defined at:
File "/raid/skyw/venv/tensorflow-pip-py27/bin/t2t-trainer", line 5, in 
pkg_resources.run_script('tensor2tensor==1.2.1', 't2t-trainer')
"
It wasn't very clear what the error is. But single GPU training/evaluation is fine, the problem comes with 2 worker GPUs.

Duplicate issue #287: Evaluation fails with multiple GPUs using --worker-gpu | split evenly...
Created: 2017-09-08T08:39:26Z
When trying to use multiple gpus with --worker_gpu, evaluation fails under Tenserflow 1.3 & Tensor2Tensor 1.2.1.
This is my training code:
PROBLEM=translate_ende_wmt_characters
MODEL=transformer
HPARAMS=transformer_base

t2t-trainer \
  --data_dir=$DATA_DIR \
  --problems=$PROBLEM \
  --model=$MODEL \
  --worker_gpu=2 \
  --hparams_set=$HPARAMS \
  --output_dir=$TRAIN_DIR

The error that occurs is as follows:
    metrics=self.metrics, hooks=self.hooks, name=self.name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 296, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 546, in evaluate
    log_progress=log_progress)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 832, in _evaluate_model
    model_fn_results = self._get_eval_ops(features, labels, metrics)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1193, in _get_eval_ops
    features, labels, model_fn_lib.ModeKeys.EVAL)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1136, in _call_model_fn
    model_fn_results = self._model_fn(features, labels, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/model_builder.py", line 273, in model_fn
    len(my_hp.problems) - 1)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/input_fn_builder.py", line 200, in cond_on_index
    return fn(cur_idx)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/model_builder.py", line 242, in nth_model
    features, skip=(skipping_is_on and skip_this_one))
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 430, in model_fn
    sharded_features = self._shard_features(features)
  File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/t2t_model.py", line 411, in _shard_features
    0))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1232, in split
    split_dim=axis, num_split=num_or_size_splits, value=value, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 3490, in _split
    num_split=num_split, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 11) and num_split 2
         [[Node: split = Split[T=DT_INT32, num_split=2, _device="/job:localhost/replica:0/task:0/cpu:0"](split/split_dim, input_reader/ExpandDims_1/_1867)]]
         [[Node: Identity_109/_1613 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_493_Identity_109", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:1"]()]]

Closing comment (2017-09-08T08:44:05Z): Duplicate of #266