Repository: pytorch
Issue: TensorBoard SummaryWriter to remote storage unreasonably slow
Labels: triaged,module: tensorboard

## 🐛 Bug

Writing scalars to TensorBoard event files on remote storage using the SummaryWriter is unreasonably slow.

## To Reproduce

Steps to reproduce the behavior:

I am not an expert on data handling and the lingo involved so forgive me some inaccuracies. From my understanding (e.g. from [here](https://www.tensorflow.org/guide/data_performance#parallelizing_data_extraction)), the main reason I observe this has possibly something to do with the time-to-first-byte, so to reproduce this issue, you need some remote storage with high bandwidth but also high time-to-first-byte, to observe this effect. You can then execute the following lines:

```python
import os
import os.path as osp
from time import time

from torch.utils.tensorboard import SummaryWriter

N = 100000

OUTPUT_DATADIR = os.getenv(
    "OUTPUT_DATADIR", "OUTPUT_DATADIR_NOT_SPECIFIED"
)  # directory on remote storage
writer = SummaryWriter(
    osp.join(OUTPUT_DATADIR, "misc", "tb_writer_speed_test"),
    max_queue=N,  # or 10 (default)
)

t0 = time()
for i in range(N):
    writer.add_scalar("count", i, i)
writer.flush()
writer.close()
print("SummaryWriter time per scalar:", 1e3 * (time() - t0) / N, "ms")

t0 = time()
with open(osp.join(OUTPUT_DATADIR, "misc", "file_writer_speed_test"), "w") as fout:
    for i in range(N):
        fout.write("some long line with some data maybe: %d\n" % i)
print("Simple file writer time per line:", 1e3 * (time() - t0) / N, "ms")
```

The output I get (N=5000) is:
```
SummaryWriter time per scalar: 10.980198955535888 ms
Simple file writer time per line: 0.011434125900268554 ms
```

## Expected behavior

I expect the SummaryWriter to be of similar speed as the file writer or maybe an order of magnitude slower because of some protobuf encoding. However, the speed is so slow (3 orders of magnitude), that a single scalar (the loss) logging for each iteration is the bottleneck in my mini-experimental training. Coming from TensorFlow I used to log on the order of 100 scalars per iteration (which would amount in 1 sec.) and this was never an issue regarding runtime on the same computational infrastructure which I am still working on now.

## Environment

From the [environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py):

```
PyTorch version: 1.7.0
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration: GPU 0: Tesla V100-PCIE-16GB
Nvidia driver version: 440.100
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] torch==1.7.0
[pip3] torch-cluster==1.5.8
[pip3] torch-geometric==1.6.3
[pip3] torch-scatter==2.0.5
[pip3] torch-sparse==0.6.8
[pip3] torch-spline-conv==1.2.0
[conda] Could not collect
```

## Additional context

Like I pointed out above, this happens for me only on the remote cluster and storage system. When I train (or execute the script above) locally on my machine locally, I have no runtime issues when writing to local storage. This is why I suspect SummaryWriter is doing something aweful with respect to the time-to-first-byte. I put it here to PyTorch, as I worked already a lot with TensorBoard and TensorFlow and never experienced any issues.

Any hints or help on options or system settings to improve on this behavior are greatly appreciated :)