Repository: pytorch
Issue: [Caffe2] Segmentation faults in multithreading Caffe2
Labels: caffe2

I wrote these scripts in reference to the Caffe2 tutorials to check if Caffe2 is thread-safe.

caffe2_trial.py:

```
#!/usr/bin/python

from caffe2.python import workspace

import threading
import caffe2_multithreading

NUM_THREADS = 2

def worker():
    caffe2_multithreading.main()

def run_workers():

    threads = [threading.Thread(name="Thread{}".format(i), target=worker) for i in range(NUM_THREADS)]
    for thread in threads:
        thread.start()

workspace.GlobalInit(['caffe2', '--caffe2_log_level=0'])
run_workers()
```

caffe2_multithreading.py:

```
from caffe2.python import workspace, model_helper

import numpy as np 
import threading

def main():

    data = np.random.rand(16, 100).astype(np.float32)
    label = (np.random.rand(16) * 10).astype(np.int32)

    workspace.FeedBlob("data", data)
    workspace.FeedBlob("label", label)

    m = model_helper.ModelHelper(name="my first net")

    weight = m.param_init_net.XavierFill([], "fc_w", shape=[10, 100])
    bias = m.param_init_net.ConstantFill([], "fc_b", shape=[10, ])

    fc_l = m.net.FC(["data", "fc_w", "fc_b"], "fcl")
    pred = m.net.Sigmoid(fc_l, "pred")
    softmax, loss = m.net.SoftmaxWithLoss([pred, "label"], ["softmax", "loss"])

    workspace.RunNetOnce(m.param_init_net)
    workspace.CreateNet(m.net)

    for _ in range(100):
        data = np.random.rand(16, 100).astype(np.float32)
        label = (np.random.rand(16) * 10).astype(np.int32)
        workspace.FeedBlob("data", data)
        workspace.FeedBlob("label", label)
        workspace.RunNet(m.name, 10)
    
    print("{} in {} fetched blob softmax:\n{}".format(workspace.CurrentWorkspace(), threading.current_thread().name, workspace.FetchBlob("softmax")))
    print("{} in {} fetched blob loss:\n{}".format(workspace.CurrentWorkspace(), threading.current_thread().name, workspace.FetchBlob("loss")))

if __name__ == '__main__':
    main()
```

When I run these, I get segfaults as below:

```
E0814 16:35:11.237641 13596 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0814 16:35:11.237673 13596 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0814 16:35:11.237675 13596 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
*** Aborted at 1534278911 (unix time) try "date -d @1534278911" if you are using GNU date ***
PC: @     0x7f98556c85b8 caffe2::math::Gemm<>()
*** SIGSEGV (@0x0) received by PID 13596 (TID 0x7f9815001700) from PID 0; stack trace: ***
    @     0x7f98ab66e390 (unknown)
    @     0x7f98556c85b8 caffe2::math::Gemm<>()
    @     0x7f9855c08c58 caffe2::FullyConnectedOp<>::DoRunWithType<>()
    @     0x7f98556d89d2 caffe2::Operator<>::Run()
    @     0x7f98557848bb caffe2::SimpleNet::Run()
    @     0x7f985578f49a caffe2::Workspace::RunNet()
    @     0x7f98564e9477 _ZZN8pybind1112cpp_function10initializeIZN6caffe26python16addGlobalMethodsERNS_6moduleEEUlRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEibE19_bJSD_ibEJNS_4nameENS_5scopeENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESV_
    @     0x7f985652ba0e pybind11::cpp_function::dispatcher()
    @           0x4c30ce PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c1e6f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4c16e7 PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4d55f3 (unknown)
    @           0x4a577e PyObject_Call
    @           0x4bed3d PyEval_EvalFrameEx
    @           0x4c136f PyEval_EvalFrameEx
    @           0x4c136f PyEval_EvalFrameEx
    @           0x4b9ab6 PyEval_EvalCodeEx
    @           0x4d54b9 (unknown)
    @           0x4eebee (unknown)
    @           0x4a577e PyObject_Call
    @           0x4c5e10 PyEval_CallObjectWithKeywords
    @           0x589172 (unknown)
    @     0x7f98ab6646ba start_thread
default in Thread0 fetched blob softmax:
[[0.10406707 0.0958132  0.11186256 0.10592665 0.0817344  0.09865279
  0.09048746 0.08944649 0.10203679 0.11997259]
 [0.10739651 0.09508494 0.10893054 0.10854136 0.0935948  0.09612396
  0.10101433 0.08424021 0.09613933 0.10893402]
 [0.10163862 0.09200761 0.10511011 0.109869   0.09156405 0.10474087
  0.1022335  0.0703143  0.09938712 0.12313475]
 [0.08985671 0.09775824 0.11547512 0.09998353 0.09359182 0.10500769
  0.08895223 0.08376996 0.10898601 0.11661875]
 [0.08954862 0.08843845 0.11500646 0.09689793 0.09185582 0.10589205
  0.10810956 0.08333685 0.10339628 0.11751796]
 [0.09852833 0.09209983 0.11036412 0.10263348 0.09042244 0.10394713
  0.10160396 0.07928675 0.10388856 0.11722537]
 [0.10337994 0.09972429 0.11327668 0.08736703 0.09438661 0.10448781
  0.0878675  0.08105434 0.10431094 0.12414487]
 [0.09534382 0.09335071 0.11011291 0.10551803 0.08672015 0.10744442
  0.10062116 0.07553869 0.09846447 0.12688567]
 [0.09399191 0.10102197 0.11472756 0.09188537 0.08315386 0.11782556
  0.0986233  0.08276524 0.10503418 0.11097106]
 [0.09374088 0.09551394 0.11585982 0.11314663 0.09346773 0.09375941
  0.09588219 0.08359513 0.1058373  0.10919697]
 [0.10109688 0.0993735  0.11368633 0.08764622 0.09819658 0.11559883
  0.09014595 0.0805226  0.10728084 0.10645223]
 [0.10267685 0.09026854 0.11018284 0.10741374 0.09042422 0.11028332
  0.09176522 0.08200574 0.09634313 0.11863648]
 [0.10321208 0.09730779 0.10979314 0.10604937 0.09673787 0.10101081
  0.09481944 0.08262547 0.09426087 0.11418316]
 [0.10170192 0.09810354 0.11086501 0.09821761 0.09967485 0.10490072
  0.09211882 0.08475582 0.09108799 0.11857382]
 [0.09822202 0.10342799 0.10083076 0.10264489 0.09806627 0.10075969
  0.10523681 0.08029877 0.09441201 0.11610077]
 [0.09419835 0.10204436 0.10571007 0.11321441 0.09489898 0.1064717
  0.08542782 0.09340751 0.09008439 0.11454245]]
default in Thread0 fetched blob loss:
2.26629066467
    @     0x7f98ab39a41d clone
    @                0x0 (unknown)
Segmentation fault (core dumped)
```

Does this mean that Caffe2 is not thread-safe? I have been working hard for days just to get multithreading working in Caffe2. Thanks!