docker container cannot launch tensorboard
State: closed | Pull request: no
Labels: name: bug

Hi all,

I was running the docker container using this command

```shell
docker run --gpus all -it --rm -v /mnt/NeMo:/NeMo --shm-size=8g \
-p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit \
stack=67108864 --device=/dev/snd nvcr.io/nvidia/nemo:1.0.0b3
```

and the NeMo repository was on branch 1.0.0b3. 

In this container, I have successfully trained my model but when I was trying to monitor my training using tensorboard using the following command,
```shell
tensorboard --bind_all --logdir .
```
the error appeared.
```
TensorFlow installation not found - running with reduced feature set.
Traceback (most recent call last):
  File "/opt/conda/bin/tensorboard", line 8, in <module>
    sys.exit(run_main())
  File "/opt/conda/lib/python3.6/site-packages/tensorboard/main.py", line 65, in run_main
    default.get_plugins(),
  File "/opt/conda/lib/python3.6/site-packages/tensorboard/default.py", line 108, in get_plugins
    return get_static_plugins() + get_dynamic_plugins()
  File "/opt/conda/lib/python3.6/site-packages/tensorboard/default.py", line 146, in get_dynamic_plugins
    "tensorboard_plugins"
  File "/opt/conda/lib/python3.6/site-packages/tensorboard/default.py", line 145, in <listcomp>
    for entry_point in pkg_resources.iter_entry_points(
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2472, in load
    return self.resolve()
  File "/opt/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2478, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/conda/lib/python3.6/site-packages/tensorboard_plugin_dlprof/plugin.py", line 31, in <module>
    from tensorboard.plugins.graph import dlprof_pb2
ImportError: cannot import name 'dlprof_pb2'
```

Any help is appreciated. Thanks.

You managed to solve this?
I'm having the same issue

not yet

Hi, i think i solved the problem. 
After you pull the container and run it, send a pip list command and search for something called tensorboard-plugine-dlproof.
Uninstall it.
Then:

pip uninstall -y tensorboard
pip install nvidia-pyindex
pip install nvidia-tensorboard-plugin-dlprof

As nvidia said here: https://docs.nvidia.com/deeplearning/frameworks/tensorboard-plugin-user-guide/

Care about uninstall the tensorboard dlprof plugin before follow the guide. If you don't uninstall nemo will return you an error because dlprof is duplicated. 

Bye

Hi, @lodm94 , thanks for the response

I've tried your method but `tensorboard` command does not exist anymore. Do you face the same problem? Isn't `nvidia-tensorboard-plugin-dlprof` supposed to installed tensorboard?

Yeah i guess i faced that too.
Try  this pipeline:

1. pull the container
2. run it
3. pip uninstall -y tensorboard
4. pip uninstall tensorboard-plugine-dlproof (check if i spelled correctly!!!)
5. pip install nvidia-pyindex
6. pip install nvidia-tensorboard-plugin-dlprof
7. pip install tensorboard

then i can launch tensorboard with:
`tensorboard --logdir yourpathtofiles --bind_all --port yourport`

and i open a browser typing 192.168.xxx.xxx:yourport

i guess the this two commands are persoonal. for start, apologize if i am not accurate with the istruction. i'm kinda noob about all this. i am working on my thesis so I don't have much experience. 
if you dont solve on monday i can log into my server again and i will copy all the steps.

it will be great if someone experienced could help you. and also me of course.

cheers



This method works well! Thank you!