Repository: pytorch
Issue: [caffe2]mpirun multi-node multi GPU in Distributed mode ,run resnet50_trainer.py get RuntimeError
Labels: caffe2

run with MPI
MPI is for coordinating machine rendezvous through MPI
so is it possible just using MPI as rendezvous but not redis or nfs?
i do not use redis and nfs 
as my caffe2 is included in the container singularity ，
i run the under command:
mpirun -np 4 singularity exec /public/DL_Data/cnic_ai_20180316.img python resnet50_trainer.py --train_data  ~/ILSVRC/ilsvrc12_train_lmdb --num_gpus 8  --batch_size 256 --num_epochs 90 --num_shards 4 --shard_id 1 --run_id 50002 
got runtime erro :

INFO:data_parallel_model:Adding gradient operators
INFO:data_parallel_model:Adding gradient operators
INFO:data_parallel_model:Adding gradient operators
INFO:data_parallel_model:Adding gradient operators
INFO:data_parallel_model:Add gradient all-reduces for SyncSGD
INFO:data_parallel_model:Add gradient all-reduces for SyncSGD
INFO:data_parallel_model:Add gradient all-reduces for SyncSGD
INFO:data_parallel_model:Add gradient all-reduces for SyncSGD
WARNING:data_parallel_model:Distributed broadcast of computed params is not implemented yet
WARNING:data_parallel_model:Distributed broadcast of computed params is not implemented yet
WARNING:data_parallel_model:Distributed broadcast of computed params is not implemented yet
WARNING:data_parallel_model:Distributed broadcast of computed params is not implemented yet
INFO:data_parallel_model:Post-iteration operators for updating params
INFO:data_parallel_model:Calling optimizer builder function
INFO:data_parallel_model:Post-iteration operators for updating params
INFO:data_parallel_model:Calling optimizer builder function
INFO:data_parallel_model:Post-iteration operators for updating params
INFO:data_parallel_model:Calling optimizer builder function
INFO:data_parallel_model:Post-iteration operators for updating params
INFO:data_parallel_model:Calling optimizer builder function
INFO:data_parallel_model:Add initial parameter sync
INFO:data_parallel_model:Add initial parameter sync
INFO:data_parallel_model:Add initial parameter sync
INFO:data_parallel_model:Add initial parameter sync
Traceback (most recent call last):
  File "resnet50_trainer.py", line 602, in <module>
    main()
  File "resnet50_trainer.py", line 598, in main
    Train(args)
  File "resnet50_trainer.py", line 436, in Train
    data_parallel_model.OptimizeGradientMemory(train_model, {}, set(), False)
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/data_parallel_model.py", line 1688, in OptimizeGradientMemory
    input_shapes_all_devices,
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 257, in InferShapesAndTypes
    net_protos, blob_dimensions
RuntimeError: [enforce fail at operator.cc:492] . Invalid shape inference for operator Broadcast Expected 8 outputs, but got 1 
Traceback (most recent call last):
  File "resnet50_trainer.py", line 602, in <module>
    main()
  File "resnet50_trainer.py", line 598, in main
    Train(args)
  File "resnet50_trainer.py", line 436, in Train
    data_parallel_model.OptimizeGradientMemory(train_model, {}, set(), False)
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/data_parallel_model.py", line 1688, in OptimizeGradientMemory
    input_shapes_all_devices,
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 257, in InferShapesAndTypes
    net_protos, blob_dimensions
RuntimeError: [enforce fail at operator.cc:492] . Invalid shape inference for operator Broadcast Expected 8 outputs, but got 1 
Traceback (most recent call last):
  File "resnet50_trainer.py", line 602, in <module>
    main()
  File "resnet50_trainer.py", line 598, in main
    Train(args)
  File "resnet50_trainer.py", line 436, in Train
    data_parallel_model.OptimizeGradientMemory(train_model, {}, set(), False)
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/data_parallel_model.py", line 1688, in OptimizeGradientMemory
    input_shapes_all_devices,
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 257, in InferShapesAndTypes
    net_protos, blob_dimensions
RuntimeError: [enforce fail at operator.cc:492] . Invalid shape inference for operator Broadcast Expected 8 outputs, but got 1 
Traceback (most recent call last):
  File "resnet50_trainer.py", line 602, in <module>
    main()
  File "resnet50_trainer.py", line 598, in main
    Train(args)
  File "resnet50_trainer.py", line 436, in Train
    data_parallel_model.OptimizeGradientMemory(train_model, {}, set(), False)
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/data_parallel_model.py", line 1688, in OptimizeGradientMemory
    input_shapes_all_devices,
  File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 257, in InferShapesAndTypes
    net_protos, blob_dimensions
RuntimeError: [enforce fail at operator.cc:492] . Invalid shape inference for operator Broadcast Expected 8 outputs, but got 1 
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[43321,1],1]
  Exit code:    1
-----------------------------------------------
if run with single node 8 gpus ,# success !
-----------------------------------------------
i have 4 nodes ,8 gpus per node
so how to run distributed caffe2 with mpi, whether redis or nfs is necessary? my machine does not install redis and nfs。