Repository: pytorch
Issue: BUILD_BINARY is a lie
Labels: module: build,triaged

## 🐛 Bug

I expect `BUILD_BINARY=0` to not build any binaries. It still builds lots of binaries.

## To Reproduce

Steps to reproduce the behavior:

1. `BUILD_BINARY=0 python setup.py build_deps develop`
2. `ls build/bin`

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

```
(/scratch/ezyang/pytorch-build-env) ezyang@devfair040:/scratch/ezyang/pytorch-build$ ls build/bin 
AlgorithmsTest            common_subexpression_elimination_test  event_test                             net_gpu_test                string_ops_test
apply_test                common_test                            fatal_signal_asan_no_sig_test          net_test                    SubgraphMatcherTest
apply_utils_test          context_gpu_test                       fixed_divisor_test                     NeuralNetTest               TarjansImplTest
atest                     context_test                           generate_proposals_op_test             nnpack_test                 tbb_init_test
backend_cutting_test      converter_nomigraph_test               generate_proposals_op_util_boxes_test  observer_test               test_api
basic                     conv_op_cache_cudnn_test               generate_proposals_op_util_nms_test    operator_fallback_gpu_test  test_jit
batch_matmul_op_gpu_test  conv_to_nnpack_transform_test          graph_test                             operator_gpu_test           test_parallel
batch_matmul_op_test      conv_transpose_op_mobile_test          GraphTest                              operator_schema_test        text_file_reader_utils_test
BinaryMatchImplTest       cpuid_test                             half_test                              operator_test               time_observer_test
blob_gpu_test             cuda_half_test                         init_test                              parallel_net_test           timer_test
blob_test                 cuda_optional_test                     integer_divider_test                   pattern_net_transform_test  transform_test
boolean_unmask_ops_test   cuda_packedtensoraccessor_test         js_embed                               predictor_test              typeid_test
broadcast_test            cuda_rng_test                          logging_test                           protoc                      undefined_tensor_test
c10_Array_test            cudnn_test                             MatchTest                              proto_utils_test            utility_ops_gpu_test
c10_flags_test            dead_code_elim_test                    math_gpu_test                          reshape_op_gpu_test         utility_ops_test
c10_Metaprogramming_test  depthwise3x3_conv_op_test              math_test                              roi_align_op_gpu_test       verify_api_visibility
c10_registry_test         device_test                            mobile_test                            scalar_tensor_test          weakref_test
c10_TypeList_test         dispatch_test                          module_test                            scalar_test                 workspace_test
c10_TypeTraits_test       distributed_test                       mpi_gpu_test                           simple_queue_test           wrapdim_test
c10_utils_cpu_test        dlconvertor_test                       mpi_test                               smart_tensor_printer_test
c10_utils_gpu_test        elementwise_op_gpu_test                native_test                            ssa_test
c10_utils_hip_test        elementwise_op_test                    net_async_tracing_test                 stats_test
cast_test                 event_gpu_test                         net_dag_utils_test                     stream_test
```

## Expected behavior

Nothing

<!-- A clear and concise description of what you expected to happen. -->

## Environment

This is on a devfair.


cc @malfet @seemethere @walterddr