Repository: tensorflow/tensor2tensor

Issue #266: Evaluation failed when training with multiple work gpus
Created: 2017-08-30T23:53:29Z
I set worker_gpu to 2 and use 2 gpus in the same node. training is completely fine. But evaluation fails with this error
"
tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 47) and num_split 2
[[Node: split_2 = Split[T=DT_INT32, num_split=2, _device="/job:localhost/replica:0/task:0/cpu:0"](split_2/split_dim, input_reader/ExpandDims_3/_1823)]]
[[Node: split_2/_1825 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1113_split_2", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:1"]]
Caused by op u'split_2', defined at:
File "/raid/skyw/venv/tensorflow-pip-py27/bin/t2t-trainer", line 5, in 
pkg_resources.run_script('tensor2tensor==1.2.1', 't2t-trainer')
"
It wasn't very clear what the error is. But single GPU training/evaluation is fine, the problem comes with 2 worker GPUs.

Duplicate issue #339: V1.2.4 Crashed after starting evaluation during Training
Created: 2017-10-06T10:36:43Z
Software:
tensor2tensor (1.2.4)
tensorflow-gpu (1.3.0)
Hardware:
DGX-1
Problem:
Tensor2tensor crashed after it started the evolution process during the training.
Command:
t2t-trainer   --data_dir=~/t2t_data   --TMP_DIR=~/tmp/t2t_datagen   --problems=languagemodel_wiki_full32k-programming_desc2code_py-sentiment_imdb-summarize_cnn_dailymail32k-translate_encs_wmt32k-translate_encs_wmt32k_rev-translate_end
e_wmt32k-translate_ende_wmt32k_rev-translate_enfr_wmt32k-translate_enfr_wmt32k_rev   --model=multi_model   --hparams_set=multimodel_base   --output_dir=~/t2t_train/multimodel_base_all_1_4 --worker_gpu=8

Result:
WARNING:tensorflow:uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Creating experiment, storing model files in /home/ga53huq2/t2t_train/multimodel_base_all_1_4
INFO:tensorflow:datashard_devices: ['gpu:0', 'gpu:1', 'gpu:2', 'gpu:3', 'gpu:4', 'gpu:5', 'gpu:6', 'gpu:7']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:Using config: {'_model_dir': '/home/ga53huq2/t2t_train/multimodel_base_all_1_4', '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 20, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
allow_soft_placement: true
graph_options {
  optimizer_options {
  }
}
, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f80770a02d0>, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 0.95
}
, '_num_worker_replicas': 0, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_master': '', '_log_step_count_steps': 100}
WARNING:tensorflow:uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:269: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/languagemodel_wiki_full32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/programming_desc2code_py-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/sentiment_imdb-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/summarize_cnn_dailymail32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_encs_wmt32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_encs_wmt32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_ende_wmt32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_ende_wmt32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_enfr_wmt32k-train*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 2048, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240, 192, 
192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_enfr_wmt32k-train*
INFO:tensorflow:datashard_devices: ['gpu:0', 'gpu:1', 'gpu:2', 'gpu:3', 'gpu:4', 'gpu:5', 'gpu:6', 'gpu:7']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:This model_fn took 20.970 sec.
INFO:tensorflow:This model_fn took 16.576 sec.
INFO:tensorflow:This model_fn took 9.398 sec.
INFO:tensorflow:This model_fn took 16.171 sec.
INFO:tensorflow:This model_fn took 18.245 sec.
INFO:tensorflow:This model_fn took 17.129 sec.
INFO:tensorflow:This model_fn took 15.378 sec.
INFO:tensorflow:This model_fn took 17.810 sec.
INFO:tensorflow:This model_fn took 18.538 sec.
INFO:tensorflow:This model_fn took 15.213 sec.
INFO:tensorflow:Weight    body/dec_layer_0/attention/decoder_self_attention/output_transform_single/bias        shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/attention/decoder_self_attention/output_transform_single/kernel      shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_0/attention/decoder_self_attention/qkv_transform_single/bias           shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/dec_layer_0/attention/decoder_self_attention/qkv_transform_single/kernel         shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/kv_transform_single/bias                  shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/kv_transform_single/kernel                shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/q_transform_single/bias                   shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/attention/encdec_attention/q_transform_single/kernel                 shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_0/attention/layer_norm/layer_norm_bias                                 shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/attention/layer_norm/layer_norm_scale                                shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/ffn/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/dec_layer_0/ffn/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_0/ffn/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/ffn/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_0/ffn/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_0/ffn/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/attention/decoder_self_attention/output_transform_single/bias        shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/attention/decoder_self_attention/output_transform_single/kernel      shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_1/attention/decoder_self_attention/qkv_transform_single/bias           shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/dec_layer_1/attention/decoder_self_attention/qkv_transform_single/kernel         shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/kv_transform_single/bias                  shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/kv_transform_single/kernel                shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/q_transform_single/bias                   shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/attention/encdec_attention/q_transform_single/kernel                 shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_1/attention/layer_norm/layer_norm_bias                                 shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/attention/layer_norm/layer_norm_scale                                shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/ffn/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/dec_layer_1/ffn/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_1/ffn/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/ffn/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_1/ffn/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_1/ffn/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/attention/decoder_self_attention/output_transform_single/bias        shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/attention/decoder_self_attention/output_transform_single/kernel      shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_2/attention/decoder_self_attention/qkv_transform_single/bias           shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/dec_layer_2/attention/decoder_self_attention/qkv_transform_single/kernel         shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/kv_transform_single/bias                  shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/kv_transform_single/kernel                shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/q_transform_single/bias                   shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/attention/encdec_attention/q_transform_single/kernel                 shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_2/attention/layer_norm/layer_norm_bias                                 shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/attention/layer_norm/layer_norm_scale                                shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/ffn/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/ffn/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/noisy_top_k_gating/w_gate                                    shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/noisy_top_k_gating/w_noise                                   shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_0/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_0/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_1/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_1/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_10/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_10/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_11/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_11/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_12/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_12/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_13/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_13/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_14/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_14/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_15/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_15/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_16/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_16/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_17/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_17/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_18/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_18/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_19/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_19/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_2/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_2/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_20/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_20/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_21/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_21/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_22/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_22/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_23/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_23/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_24/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_24/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_25/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_25/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_26/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_26/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_27/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_27/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_28/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_28/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_29/w_0                                              shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_29/w_1                                              shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_3/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_3/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_4/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_4/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_5/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_5/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_6/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_6/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_7/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_7/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_8/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_8/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_9/w_0                                               shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_2/ffn/moe/parallel_9/w_1                                               shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/dec_layer_3/attention/decoder_self_attention/output_transform_single/bias        shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/attention/decoder_self_attention/output_transform_single/kernel      shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_3/attention/decoder_self_attention/qkv_transform_single/bias           shape    (1536,)                size    1536
INFO:tensorflow:Weight    body/dec_layer_3/attention/decoder_self_attention/qkv_transform_single/kernel         shape    (1, 1, 512, 1536)      size    786432
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/kv_transform_single/bias                  shape    (1024,)                size    1024
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/kv_transform_single/kernel                shape    (1, 1, 512, 1024)      size    524288
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/output_transform_single/bias              shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/output_transform_single/kernel            shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/q_transform_single/bias                   shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/attention/encdec_attention/q_transform_single/kernel                 shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/dec_layer_3/attention/layer_norm/layer_norm_bias                                 shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/attention/layer_norm/layer_norm_scale                                shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/ffn/conv_hidden_relu/conv1_single/bias                               shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/dec_layer_3/ffn/conv_hidden_relu/conv1_single/kernel                             shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_3/ffn/conv_hidden_relu/conv2_single/bias                               shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/ffn/conv_hidden_relu/conv2_single/kernel                             shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/dec_layer_3/ffn/layer_norm/layer_norm_bias                                       shape    (512,)                 size    512
INFO:tensorflow:Weight    body/dec_layer_3/ffn/layer_norm/layer_norm_scale                                      shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_0_single/depthwise_kernel         shape    (3, 1, 512, 1)         size    1536
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_0_single/pointwise_kernel         shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_1_single/depthwise_kernel         shape    (3, 1, 2048, 1)        size    6144
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_1_single/pointwise_kernel         shape    (1, 1, 2048, 2048)     size    4194304
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_norm_0/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_norm_0/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_norm_1/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual1/conv_block_norm_1/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_0_single/depthwise_kernel         shape    (15, 1, 2048, 1)       size    30720
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_0_single/pointwise_kernel         shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_1_single/depthwise_kernel         shape    (15, 1, 512, 1)        size    7680
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_1_single/pointwise_kernel         shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_norm_0/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_norm_0/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_norm_1/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/conv_res_step/residual2/conv_block_norm_1/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/layer_norm/layer_norm_bias                                           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/layer_norm/layer_norm_scale                                          shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_0/moe/noisy_top_k_gating/w_gate                                        shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_0/moe/noisy_top_k_gating/w_noise                                       shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_0/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_0/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_1/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_1/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_10/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_10/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_11/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_11/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_12/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_12/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_13/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_13/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_14/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_14/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_15/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_15/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_16/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_16/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_17/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_17/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_18/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_18/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_19/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_19/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_2/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_2/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_20/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_20/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_21/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_21/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_22/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_22/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_23/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_23/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_24/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_24/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_25/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_25/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_26/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_26/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_27/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_27/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_28/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_28/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_29/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_29/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_3/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_3/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_4/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_4/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_5/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_5/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_6/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_6/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_7/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_7/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_8/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_8/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_9/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_0/moe/parallel_9/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_0_single/depthwise_kernel         shape    (3, 1, 512, 1)         size    1536
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_0_single/pointwise_kernel         shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_1_single/depthwise_kernel         shape    (3, 1, 2048, 1)        size    6144
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_1_single/pointwise_kernel         shape    (1, 1, 2048, 2048)     size    4194304
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_norm_0/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_norm_0/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_norm_1/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual1/conv_block_norm_1/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_0_single/depthwise_kernel         shape    (15, 1, 2048, 1)       size    30720
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_0_single/pointwise_kernel         shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_1_single/depthwise_kernel         shape    (15, 1, 512, 1)        size    7680
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_1_single/pointwise_kernel         shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_norm_0/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_norm_0/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_norm_1/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/conv_res_step/residual2/conv_block_norm_1/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/layer_norm/layer_norm_bias                                           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/layer_norm/layer_norm_scale                                          shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_1/moe/noisy_top_k_gating/w_gate                                        shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_1/moe/noisy_top_k_gating/w_noise                                       shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_0/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_0/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_1/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_1/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_10/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_10/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_11/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_11/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_12/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_12/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_13/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_13/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_14/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_14/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_15/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_15/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_16/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_16/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_17/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_17/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_18/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_18/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_19/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_19/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_2/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_2/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_20/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_20/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_21/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_21/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_22/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_22/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_23/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_23/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_24/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_24/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_25/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_25/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_26/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_26/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_27/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_27/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_28/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_28/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_29/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_29/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_3/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_3/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_4/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_4/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_5/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_5/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_6/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_6/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_7/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_7/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_8/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_8/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_9/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_1/moe/parallel_9/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_0_single/depthwise_kernel         shape    (3, 1, 512, 1)         size    1536
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_0_single/pointwise_kernel         shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_1_single/depthwise_kernel         shape    (3, 1, 2048, 1)        size    6144
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_1_single/pointwise_kernel         shape    (1, 1, 2048, 2048)     size    4194304
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_norm_0/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_norm_0/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_norm_1/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual1/conv_block_norm_1/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_0_single/depthwise_kernel         shape    (15, 1, 2048, 1)       size    30720
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_0_single/pointwise_kernel         shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_1_single/depthwise_kernel         shape    (15, 1, 512, 1)        size    7680
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_1_single/pointwise_kernel         shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_norm_0/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_norm_0/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_norm_1/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/conv_res_step/residual2/conv_block_norm_1/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/layer_norm/layer_norm_bias                                           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/layer_norm/layer_norm_scale                                          shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_2/moe/noisy_top_k_gating/w_gate                                        shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_2/moe/noisy_top_k_gating/w_noise                                       shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_0/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_0/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_1/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_1/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_10/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_10/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_11/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_11/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_12/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_12/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_13/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_13/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_14/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_14/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_15/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_15/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_16/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_16/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_17/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_17/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_18/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_18/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_19/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_19/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_2/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_2/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_20/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_20/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_21/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_21/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_22/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_22/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_23/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_23/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_24/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_24/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_25/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_25/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_26/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_26/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_27/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_27/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_28/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_28/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_29/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_29/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_3/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_3/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_4/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_4/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_5/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_5/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_6/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_6/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_7/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_7/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_8/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_8/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_9/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_2/moe/parallel_9/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_0_single/depthwise_kernel         shape    (3, 1, 512, 1)         size    1536
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_0_single/pointwise_kernel         shape    (1, 1, 512, 2048)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_1_single/depthwise_kernel         shape    (3, 1, 2048, 1)        size    6144
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_1_single/pointwise_kernel         shape    (1, 1, 2048, 2048)     size    4194304
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_norm_0/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_norm_0/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_norm_1/layer_norm_bias            shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual1/conv_block_norm_1/layer_norm_scale           shape    (2048,)                size    2048
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_0_single/depthwise_kernel         shape    (15, 1, 2048, 1)       size    30720
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_0_single/pointwise_kernel         shape    (1, 1, 2048, 512)      size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_1_single/depthwise_kernel         shape    (15, 1, 512, 1)        size    7680
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_1_single/pointwise_kernel         shape    (1, 1, 512, 512)       size    262144
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_norm_0/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_norm_0/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_norm_1/layer_norm_bias            shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/conv_res_step/residual2/conv_block_norm_1/layer_norm_scale           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/layer_norm/layer_norm_bias                                           shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/layer_norm/layer_norm_scale                                          shape    (512,)                 size    512
INFO:tensorflow:Weight    body/enc_layer_3/moe/noisy_top_k_gating/w_gate                                        shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_3/moe/noisy_top_k_gating/w_noise                                       shape    (512, 30)              size    15360
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_0/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_0/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_1/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_1/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_10/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_10/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_11/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_11/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_12/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_12/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_13/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_13/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_14/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_14/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_15/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_15/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_16/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_16/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_17/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_17/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_18/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_18/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_19/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_19/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_2/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_2/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_20/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_20/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_21/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_21/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_22/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_22/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_23/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_23/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_24/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_24/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_25/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_25/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_26/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_26/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_27/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_27/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_28/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_28/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_29/w_0                                                  shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_29/w_1                                                  shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_3/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_3/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_4/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_4/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_5/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_5/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_6/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_6/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_7/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_7/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_8/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_8/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_9/w_0                                                   shape    (512, 2048)            size    1048576
INFO:tensorflow:Weight    body/enc_layer_3/moe/parallel_9/w_1                                                   shape    (2048, 512)            size    1048576
INFO:tensorflow:Weight    body/target_space_embedding/kernel                                                    shape    (32, 512)              size    16384
INFO:tensorflow:Weight    class_label_modality_2_512/class_label_modality_2_512/dense/bias                      shape    (2,)                   size    2
INFO:tensorflow:Weight    class_label_modality_2_512/class_label_modality_2_512/dense/kernel                    shape    (512, 2)               size    1024
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_0                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_10                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_11                                        shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_12                                        shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_13                                        shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_14                                        shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_15                                        shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_1                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_2                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_3                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_4                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_5                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_6                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_7                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_8                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/input_emb/weights_9                                         shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_0                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_10                                          shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_11                                          shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_12                                          shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_13                                          shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_14                                          shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_15                                          shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_1                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_2                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_3                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_4                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_5                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_6                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_7                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_8                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/softmax/weights_9                                           shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_0                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_10                                       shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_11                                       shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_12                                       shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_13                                       shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_14                                       shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_15                                       shape    (1890, 512)            size    967680
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_1                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_2                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_3                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_4                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_5                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_6                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_7                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_8                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_30251_512/target_emb/weights_9                                        shape    (1891, 512)            size    968192
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_0                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_10                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_11                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_12                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_13                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_14                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_15                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_1                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_2                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_3                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_4                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_5                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_6                                         shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_7                                         shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_8                                         shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/input_emb/weights_9                                         shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_0                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_10                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_11                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_12                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_13                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_14                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_15                                          shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_1                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_2                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_3                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_4                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_5                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_6                                           shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_7                                           shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_8                                           shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/softmax/weights_9                                           shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_0                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_10                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_11                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_12                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_13                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_14                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_15                                       shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_1                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_2                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_3                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_4                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_5                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_6                                        shape    (2038, 512)            size    1043456
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_7                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_8                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32599_512/target_emb/weights_9                                        shape    (2037, 512)            size    1042944
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_0                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_10                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_11                                        shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_12                                        shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_13                                        shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_14                                        shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_15                                        shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_1                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_2                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_3                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_4                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_5                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_6                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_7                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_8                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/input_emb/weights_9                                         shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_0                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_10                                          shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_11                                          shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_12                                          shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_13                                          shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_14                                          shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_15                                          shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_1                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_2                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_3                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_4                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_5                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_6                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_7                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_8                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/softmax/weights_9                                           shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_0                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_10                                       shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_11                                       shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_12                                       shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_13                                       shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_14                                       shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_15                                       shape    (2056, 512)            size    1052672
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_1                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_2                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_3                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_4                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_5                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_6                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_7                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_8                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_32907_512/target_emb/weights_9                                        shape    (2057, 512)            size    1053184
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_0                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_10                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_11                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_12                                        shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_13                                        shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_14                                        shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_15                                        shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_1                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_2                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_3                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_4                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_5                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_6                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_7                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_8                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/input_emb/weights_9                                         shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_0                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_10                                          shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_11                                          shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_12                                          shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_13                                          shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_14                                          shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_15                                          shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_1                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_2                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_3                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_4                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_5                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_6                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_7                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_8                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/softmax/weights_9                                           shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_0                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_10                                       shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_11                                       shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_12                                       shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_13                                       shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_14                                       shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_15                                       shape    (2169, 512)            size    1110528
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_1                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_2                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_3                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_4                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_5                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_6                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_7                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_8                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_34716_512/target_emb/weights_9                                        shape    (2170, 512)            size    1111040
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_0                                            shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_10                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_11                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_12                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_13                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_14                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_15                                           shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_1                                            shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_2                                            shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_3                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_4                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_5                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_6                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_7                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_8                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/softmax/weights_9                                            shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_0                                         shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_10                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_11                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_12                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_13                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_14                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_15                                        shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_1                                         shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_2                                         shape    (259, 512)             size    132608
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_3                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_4                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_5                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_6                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_7                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_8                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_4131_512/target_emb/weights_9                                         shape    (258, 512)             size    132096
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_0                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_10                                         shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_11                                         shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_12                                         shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_13                                         shape    (514, 512)             size    263168
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_14                                         shape    (514, 512)             size    263168
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_15                                         shape    (514, 512)             size    263168
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_1                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_2                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_3                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_4                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_5                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_6                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_7                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_8                                          shape    (515, 512)             size    263680
INFO:tensorflow:Weight    symbol_modality_8237_512/input_emb/weights_9                                          shape    (515, 512)             size    263680
INFO:tensorflow:Trainable Variables Total size: 564753922
INFO:tensorflow:Diet Variables Total size: 0
INFO:tensorflow:Computing gradients for global model_fn.
/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:95: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Global model_fn finished.
INFO:tensorflow:Create CheckpointSaverHook.
2017-10-06 10:22:16.098485: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-06 10:22:16.098541: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-06 10:22:16.098550: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-06 10:22:16.098557: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-06 10:22:16.098565: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-10-06 10:22:17.194209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:06:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:17.896096: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22e9e7b80 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:17.897825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 1 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:07:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:18.603318: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22c90f220 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:18.604970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 2 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:0a:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:19.331394: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22bddf2d0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:19.333271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 3 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:0b:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:20.090214: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22bcce2b0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:20.091995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 4 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:85:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:20.859943: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22bcc8e20 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:20.861730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 5 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:86:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:21.628546: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22bc403d0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:21.630313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 6 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:89:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:22.428728: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x22bbbd270 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-06 10:22:22.430552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 7 with properties: 
name: Tesla P100-SXM2-16GB
major: 6 minor: 0 memoryClockRate (GHz) 1.4805
pciBusID 0000:8a:00.0
Total memory: 15.89GiB
Free memory: 15.61GiB
2017-10-06 10:22:22.441548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 0 and 5
2017-10-06 10:22:22.441577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 0 and 6
2017-10-06 10:22:22.441591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 0 and 7
2017-10-06 10:22:22.445179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 1 and 4
2017-10-06 10:22:22.447026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 1 and 6
2017-10-06 10:22:22.447048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 1 and 7
2017-10-06 10:22:22.448883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 2 and 4
2017-10-06 10:22:22.448905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 2 and 5
2017-10-06 10:22:22.450738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 2 and 7
2017-10-06 10:22:22.450765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 3 and 4
2017-10-06 10:22:22.450802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 3 and 5
2017-10-06 10:22:22.450833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 3 and 6
2017-10-06 10:22:22.452676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 4 and 1
2017-10-06 10:22:22.452699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 4 and 2
2017-10-06 10:22:22.452712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 4 and 3
2017-10-06 10:22:22.460701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 5 and 0
2017-10-06 10:22:22.460730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 5 and 2
2017-10-06 10:22:22.460743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 5 and 3
2017-10-06 10:22:22.464256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 6 and 0
2017-10-06 10:22:22.464280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 6 and 1
2017-10-06 10:22:22.464296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 6 and 3
2017-10-06 10:22:22.466110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 7 and 0
2017-10-06 10:22:22.466131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 7 and 1
2017-10-06 10:22:22.466143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:847] Peer access not supported between device ordinals 7 and 2
2017-10-06 10:22:22.466302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 1 2 3 4 5 6 7 
2017-10-06 10:22:22.466315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y Y Y Y Y N N N 
2017-10-06 10:22:22.466325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 1:   Y Y Y Y N Y N N 
2017-10-06 10:22:22.466334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 2:   Y Y Y Y N N Y N 
2017-10-06 10:22:22.466343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 3:   Y Y Y Y N N N Y 
2017-10-06 10:22:22.466352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 4:   Y N N N Y Y Y Y 
2017-10-06 10:22:22.466361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 5:   N Y N N Y Y Y Y 
2017-10-06 10:22:22.466370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 6:   N N Y N Y Y Y Y 
2017-10-06 10:22:22.466379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 7:   N N N Y Y Y Y Y 
2017-10-06 10:22:22.466564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0000:06:00.0)
2017-10-06 10:22:22.466689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla P100-SXM2-16GB, pci bus id: 0000:07:00.0)
2017-10-06 10:22:22.466700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:2) -> (device: 2, name: Tesla P100-SXM2-16GB, pci bus id: 0000:0a:00.0)
2017-10-06 10:22:22.466710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:3) -> (device: 3, name: Tesla P100-SXM2-16GB, pci bus id: 0000:0b:00.0)
2017-10-06 10:22:22.466719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:4) -> (device: 4, name: Tesla P100-SXM2-16GB, pci bus id: 0000:85:00.0)
2017-10-06 10:22:22.466736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:5) -> (device: 5, name: Tesla P100-SXM2-16GB, pci bus id: 0000:86:00.0)
2017-10-06 10:22:22.466744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:6) -> (device: 6, name: Tesla P100-SXM2-16GB, pci bus id: 0000:89:00.0)
2017-10-06 10:22:22.466753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:7) -> (device: 7, name: Tesla P100-SXM2-16GB, pci bus id: 0000:8a:00.0)
INFO:tensorflow:Saving checkpoints for 1 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:loss = 9.47266, step = 1
INFO:tensorflow:global_step/sec: 0.340327
INFO:tensorflow:loss = 7.58639, step = 101 (293.831 sec)
INFO:tensorflow:global_step/sec: 0.54257
INFO:tensorflow:loss = 6.66588, step = 201 (184.308 sec)
INFO:tensorflow:Saving checkpoints for 204 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.317322
INFO:tensorflow:loss = 0.0952963, step = 301 (315.137 sec)
INFO:tensorflow:global_step/sec: 0.542334
INFO:tensorflow:loss = 5.62133, step = 401 (184.389 sec)
INFO:tensorflow:Saving checkpoints for 466 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.33766
INFO:tensorflow:loss = 0.178572, step = 501 (296.155 sec)
INFO:tensorflow:global_step/sec: 0.550264
INFO:tensorflow:loss = 4.09841, step = 601 (181.731 sec)
INFO:tensorflow:global_step/sec: 0.537941
INFO:tensorflow:loss = 3.7804, step = 701 (185.895 sec)
INFO:tensorflow:Saving checkpoints for 732 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.351408
INFO:tensorflow:loss = 3.96623, step = 801 (284.569 sec)
INFO:tensorflow:global_step/sec: 0.528658
INFO:tensorflow:loss = 4.70143, step = 901 (189.158 sec)
INFO:tensorflow:Saving checkpoints for 1000 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.325086
INFO:tensorflow:loss = 5.33783, step = 1001 (307.610 sec)
INFO:tensorflow:global_step/sec: 0.54199
INFO:tensorflow:loss = 4.64965, step = 1101 (184.506 sec)
INFO:tensorflow:global_step/sec: 0.514352
INFO:tensorflow:loss = 5.29311, step = 1201 (194.420 sec)
INFO:tensorflow:Saving checkpoints for 1254 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.346023
INFO:tensorflow:loss = 3.02916, step = 1301 (288.998 sec)
INFO:tensorflow:global_step/sec: 0.534459
INFO:tensorflow:loss = 4.3703, step = 1401 (187.105 sec)
INFO:tensorflow:global_step/sec: 0.522147
INFO:tensorflow:loss = 0.154149, step = 1501 (191.517 sec)
INFO:tensorflow:Saving checkpoints for 1520 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.333356
INFO:tensorflow:loss = 3.93742, step = 1601 (299.979 sec)
INFO:tensorflow:global_step/sec: 0.538192
INFO:tensorflow:loss = 4.51017, step = 1701 (185.809 sec)
INFO:tensorflow:Saving checkpoints for 1780 into /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt.
INFO:tensorflow:global_step/sec: 0.344887
INFO:tensorflow:loss = 4.74185, step = 1801 (289.949 sec)
INFO:tensorflow:global_step/sec: 0.535042
INFO:tensorflow:loss = 4.79096, step = 1901 (186.901 sec)
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/languagemodel_wiki_full32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/programming_desc2code_py-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/sentiment_imdb-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/summarize_cnn_dailymail32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_encs_wmt32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_encs_wmt32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_ende_wmt32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_ende_wmt32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_enfr_wmt32k-dev*
INFO:tensorflow:batching_scheme = {'boundaries': [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 24, 26, 28, 30, 33, 36, 39, 42, 46, 50, 55, 60, 66, 72, 79, 86, 94, 103, 113, 124, 136, 149, 163, 179, 196, 215, 236, 259, 284, 312, 343, 377, 414, 455, 500, 550, 605, 
665, 731, 804, 884, 972, 1069, 1175, 1292, 1421, 1563, 1719, 1890], 'window_size': 5760, 'max_length': 1000000000, 'batch_sizes': [1920, 1440, 1440, 1440, 1152, 1152, 1152, 960, 960, 960, 720, 720, 720, 720, 640, 576, 576, 480, 480, 384, 384, 384, 320, 320, 288, 240, 240,
 192, 192, 160, 160, 144, 144, 128, 120, 96, 96, 80, 80, 72, 64, 48, 48, 48, 40, 40, 32, 32, 32, 24, 24, 24, 16, 16, 16, 16, 8, 8, 8, 8, 8, 8, 8, 8], 'shuffle_queue_size': 2160}
INFO:tensorflow:Reading data files from /home/ga53huq2/t2t_data/translate_enfr_wmt32k-dev*
INFO:tensorflow:datashard_devices: ['gpu:0', 'gpu:1', 'gpu:2', 'gpu:3', 'gpu:4', 'gpu:5', 'gpu:6', 'gpu:7']
INFO:tensorflow:caching_devices: None
INFO:tensorflow:This model_fn took 19.018 sec.
INFO:tensorflow:This model_fn took 37.142 sec.
INFO:tensorflow:This model_fn took 9.904 sec.
INFO:tensorflow:This model_fn took 16.782 sec.
INFO:tensorflow:This model_fn took 16.779 sec.
INFO:tensorflow:This model_fn took 16.476 sec.
INFO:tensorflow:This model_fn took 16.962 sec.
INFO:tensorflow:This model_fn took 16.771 sec.
INFO:tensorflow:This model_fn took 41.490 sec.
INFO:tensorflow:This model_fn took 16.865 sec.
INFO:tensorflow:Starting evaluation at 2017-10-06-10:02:17
2017-10-06 12:02:19.801853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla P100-SXM2-16GB, pci bus id: 0000:06:00.0)
2017-10-06 12:02:19.801938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla P100-SXM2-16GB, pci bus id: 0000:07:00.0)
2017-10-06 12:02:19.801985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:2) -> (device: 2, name: Tesla P100-SXM2-16GB, pci bus id: 0000:0a:00.0)
2017-10-06 12:02:19.802013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:3) -> (device: 3, name: Tesla P100-SXM2-16GB, pci bus id: 0000:0b:00.0)
2017-10-06 12:02:19.802041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:4) -> (device: 4, name: Tesla P100-SXM2-16GB, pci bus id: 0000:85:00.0)
2017-10-06 12:02:19.802052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:5) -> (device: 5, name: Tesla P100-SXM2-16GB, pci bus id: 0000:86:00.0)
2017-10-06 12:02:19.802063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:6) -> (device: 6, name: Tesla P100-SXM2-16GB, pci bus id: 0000:89:00.0)
2017-10-06 12:02:19.802073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:7) -> (device: 7, name: Tesla P100-SXM2-16GB, pci bus id: 0000:8a:00.0)
INFO:tensorflow:Restoring parameters from /home/ga53huq2/t2t_train/multimodel_base_all_1_4/model.ckpt-1780
2017-10-06 12:03:11.544988: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545013: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545065: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545088: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545036: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545019: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545157: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545080: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545027: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545092: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545147: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545148: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545322: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545152: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545155: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545159: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545633: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545073: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545680: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545170: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545723: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545171: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545179: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545180: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545229: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545148: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545768: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545909: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545916: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545951: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545958: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545986: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.545998: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546018: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546034: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546080: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546132: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546173: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546210: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546250: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546278: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546320: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546350: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546378: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.546406: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.557920: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.557973: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558005: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558042: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558089: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558135: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558164: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558192: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558222: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558270: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558299: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558327: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558354: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558385: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558433: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558506: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558538: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558582: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558592: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558616: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558677: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558678: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558770: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558809: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558841: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558869: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558897: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558926: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558971: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.558973: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559006: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559038: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559070: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559098: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559126: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559157: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559191: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559219: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559247: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559274: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559302: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559329: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559354: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559381: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559405: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559448: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559509: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559535: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559565: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559644: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559680: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.559707: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570204: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570268: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570312: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570343: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570375: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
2017-10-06 12:03:11.570405: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
Traceback (most recent call last):
  File "/home/ga53huq2/tensorflow/bin/t2t-trainer", line 96, in <module>
    tf.app.run()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/ga53huq2/tensorflow/bin/t2t-trainer", line 92, in main
    schedule=FLAGS.schedule)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/trainer_utils.py", line 352, in run
    hparams=hparams)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 209, in run
    return _execute_schedule(experiment, schedule)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 46, in _execute_schedule
    return task()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 502, in train_and_evaluate
    self.train(delay_secs=0)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 280, in train
    hooks=self._train_monitors + extra_hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 672, in _call_train
    hooks=hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 241, in train
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 862, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in run
    run_metadata=run_metadata))
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1198, in after_run
    induce_stop = m.step_end(self._last_step, result)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 358, in step_end
    return self.every_n_step_end(step, output)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 696, in every_n_step_end
    validation_outputs = self._evaluate_estimator()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 667, in _evaluate_estimator
    name=self.name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 292, in evaluate
    name=name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 739, in _evaluate_model
    config=self._session_config)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/evaluation.py", line 182, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 862, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 972, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
         [[Node: cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_34716_512/input_emb/Squeeze/_82493 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:6", send_device="/job:localhost/replica:0/task:0/gpu:6", send_device_inca
rnation=1, tensor_name="edge_371380_cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_34716_512/input_emb/Squeeze", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:6"](^cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_3471
6_512/input_emb/Squeeze/_82492)]]

Caused by op u'cond_1/cond/cond/cond/split', defined at:
  File "/home/ga53huq2/tensorflow/bin/t2t-trainer", line 96, in <module>
    tf.app.run()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/ga53huq2/tensorflow/bin/t2t-trainer", line 92, in main
    schedule=FLAGS.schedule)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/trainer_utils.py", line 352, in run
    hparams=hparams)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 209, in run
    return _execute_schedule(experiment, schedule)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 46, in _execute_schedule
    return task()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 502, in train_and_evaluate
    self.train(delay_secs=0)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 280, in train
    hooks=self._train_monitors + extra_hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 672, in _call_train
    hooks=hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 241, in train
    loss = self._train_model(input_fn=input_fn, hooks=hooks)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 862, in run
    run_metadata=run_metadata)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 818, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in run
    run_metadata=run_metadata))
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1198, in after_run
    induce_stop = m.step_end(self._last_step, result)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 358, in step_end
    return self.every_n_step_end(step, output)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 696, in every_n_step_end
    validation_outputs = self._evaluate_estimator()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 667, in _evaluate_estimator
    name=self.name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 292, in evaluate
    name=name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 713, in _evaluate_model
    features, labels, model_fn_lib.ModeKeys.EVAL)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/model_builder.py", line 348, in wrapping_model_fn
    return model_fn(model, features, mode, hparams, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/model_builder.py", line 163, in model_fn
    max_idx=len(hparams.problems) - 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in cond_on_index
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 296, in new_func
    return func(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1828, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1694, in BuildCondBranch
    original_result = fn()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in <lambda>
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in cond_on_index
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 296, in new_func
    return func(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1828, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1694, in BuildCondBranch
    original_result = fn()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in <lambda>
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in cond_on_index
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 296, in new_func
    return func(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1828, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1694, in BuildCondBranch
    original_result = fn()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in <lambda>
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 185, in cond_on_index
    lambda: cond_on_index(fn, index_tensor, max_idx, cur_idx + 1)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 296, in new_func
    return func(*args, **kwargs)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1819, in cond
    orig_res_t, res_t = context_t.BuildCondBranch(true_fn)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1694, in BuildCondBranch
    original_result = fn()
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/input_fn_builder.py", line 184, in <lambda>
    lambda: fn(cur_idx),
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/model_builder.py", line 131, in nth_model
    features, skip=(skipping_is_on and skip_this_one))
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 463, in model_fn
    sharded_features = self._shard_features(features)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensor2tensor/utils/t2t_model.py", line 444, in _shard_features
    0))
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1232, in split
    split_dim=axis, num_split=num_or_size_splits, value=value, name=name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3490, in _split
    num_split=num_split, name=name)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ga53huq2/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 13) and num_split 8
         [[Node: cond_1/cond/cond/cond/split = Split[T=DT_INT32, num_split=8, _device="/job:localhost/replica:0/task:0/cpu:0"](cond_1/cond/cond/cond/split/split_dim, cond_1/cond/cond/cond/split/Switch_1:1)]]
         [[Node: cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_34716_512/input_emb/Squeeze/_82493 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:6", send_device="/job:localhost/replica:0/task:0/gpu:6", send_device_inca
rnation=1, tensor_name="edge_371380_cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_34716_512/input_emb/Squeeze", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/gpu:6"](^cond_1/cond/symbol_modality_34716_512/parallel_6/symbol_modality_3471
6_512/input_emb/Squeeze/_82492)]]

Closing comment (2017-10-06T11:33:31Z): Duplicate of #266.
The T2T developers have not commented this issue and our attempts to workaround it, but they refer to #266 when closing other issues, so probably they are aware of this problem.