Repository: Cloud-CV/EvalAI
Issue: docker issues: website, database persistence, and example project
Root cause: Environment Incompatibility

I'm excited about using + contributing to this project and because I can't quite release my challenge data so I'm going the docker route.

After creating and running the docker image, a number of high-level issues were found:
* [x] it looks like some of the javascript on the website isn't functioning as expected because all of the graphics are native size and cluttered (see visual examples)
    * I have tried this on a current version of Firefox, Chrome, and Safari (on a Mac)
* [x] when creating teams, no team persists beyond a sample account; for example, if I create as `admin`, I would expect those teams to be visible by `host`, but this isn't happening
* ~~[x] the example challenge directory and zip file are not in line with what the backend is expecting, so there's some possible code shift that occurred; not only would bringing/testing this script back up to a working example help, but it may be a challenge for some users~~ (see #1590)
    * one example of how this isn't quite right is that the evaluation script is referred to as `data.py` in the yaml file, but it exists in another subdirectory in the distributed code
* ~~[x] as a question for the public website (https://evalai.cloudcv.org), is there the ability to keep the participant list closed or restricted? I know this goes against the goals of the project, but I'm looking for back-up solutions if the docker option can't be resolved in time.~~ (see #1606)
* [x] it was hard to understand where I should go to debug interactions on the system because it is split into many sub modules; for example, where can I go to test the database or web front interaction?

![UX sizing example 1](https://user-images.githubusercontent.com/12480297/38163159-e14209fa-34b4-11e8-9c93-fcb1b43060d1.png)
![UX sizing example 2](https://user-images.githubusercontent.com/12480297/38163165-f8b946a2-34b4-11e8-9a31-f6c604495e7f.png)
![UX sizing example 3](https://user-images.githubusercontent.com/12480297/38163167-15c88e6a-34b5-11e8-92b9-1a7f31d74e1b.png)



Hi @ezavesky, thanks for your interest in using EvalAI. We are happy either you use EvalAI (http://evalai.cloudcv.org) or your own forked version of EvalAI to host a challenge. We would be happy to help you with the problems that you are facing with the docker installation.

>  it looks like some of the javascript on the website isn't functioning as expected because all of the graphics are native size and cluttered (see visual examples)

It seems like there is some issue with the current docker setup. Let me try building the container again and see if I can reproduce this error. Meanwhile, can you please check what does the log show in the node container. Please try this command and let me know what you get in the response `docker logs nodejs`

>  When creating teams, no team persists beyond a sample account; for example, if I create as admin, I would expect those teams to be visible by the host, but this isn't happening

The challenge hosts can see all the team members submissions along with the team names in `View All Submissions` tab under a particular challenge. Let me know if that is not sufficient and you think that having a separate page to show the team name with other details would be useful for the challenge organizers. We can prioritize that and get it done soon. :)

>  the example challenge directory and zip file are not in line with what the backend is expecting, so there's some possible code shift that occurred; not only would bringing/testing this script back up to a working example help, but it may be a challenge for some users

I really apologize for the inconvenience. We are working on adding new examples to the repository so that it is even easier for the challenge hosts. Please see #1590 for more details. Let me know if you want to schedule a short meeting to know more about the challenge creation process or want some other help.

> one example of how this isn't quite right is that the evaluation script is referred to as data.py in the yaml file, but it exists in another subdirectory in the distributed code

It looks you misunderstood the example. The `data.py` is just the helper python files that would be used in the `main.py` where the main evaluation function resides. Please see the example zip file given in #1590. That will make things clear. 

>  as a question for the public website (https://evalai.cloudcv.org), is there the ability to keep the participant list closed or restricted? I know this goes against the goals of the project, but I'm looking for back-up solutions if the docker option can't be resolved in time.

Do you mean that you don't want to disclose the list of participants and the team details to the other EvalAI users? You just want these details to be accessible by challenge host, right? If this is the question, then the answer is yes. The only way participants can see the other team names which participated in the challenge by seeing the leaderboard. If you make the leaderboard private, participants cannot see who all are participating in the challenge. Please let me know if I misunderstood your question. 

>  it was hard to understand where I should go to debug interactions on the system because it is split into many sub modules; for example, where can I go to test the database or web front interaction?

If I get your right, then you want to perform the CRUD operations on the database. In this case, you have to do following things:

1. Open http://locahost:8000/admin
2. Enter the username: `admin` and password: `password`
3. You will be able to access all the details in the database

For debugging the frontend, you just need to start hacking into the codebase in `frontend/` part of the repository. For more details, if you want to see the directory structure, the best way would be to go through the docs given here: http://evalai.readthedocs.io/en/latest/directory_structure.html

Please let us know if you have more questions. We would be happy to help. 

* **Docker image building**: Thanks.  My experience with composed docker images is not a strong, so I wasn't sure where to dig into each image.  Attached is the current node-js log, but nothing looked like an error to me: [node-js_log.txt](https://github.com/Cloud-CV/EvalAI/files/1866327/node-js_log.txt)
* **Team Display**: Adding a few more details, I did not understand the difference between a *host team* and a *participant team*.   Further, it looked like the dashboard view (as an `admin`) was displaying all teams, but I realize now this is just for those teams that the user is a member of.  I have checked off this part of my original issue as it seems it was a misunderstanding and one that can be resolved when challenge creation/membership is verified.
* **Example Challenge**: Thanks, for the reference to #1590. My initial confusion about `data.py` was from the previous sample challenge, where this file was included in the yaml challenge definition; the new example helps to understand the structure a little more. Unfortunately, when using the attachment from #1590, but still see no created challenge after submission.  When inspecting the django instance file structure, it does appear that the assets (under `/media/` the following directories were created: `zip_configuration_files`, `evaluation_scripts`, `test_annotations`, `logos` ) were created, but these assets don't appear visible from any of the challenge lists. 
* **team/challenge visibility**: Okay, thanks.  Aside from making the leaderboard private, I was curious if I could host a "private" challenge that can be joined by invite or hidden URL only. Leaderboard visibility (and possibly challenge visibility itself) is less of a concern.  I realize this is a little against the spirit of the open source contributions, but that's that the near-term need was.
* **frontend hacking**: This make sense, thanks for your response.


> Docker image building: Thanks. My experience with composed docker images is not a strong, so I wasn't sure where to dig into each image. Attached is the current node-js log, but nothing looked like an error to me: node-js_log.txt

I was able to reproduce this error and it seems like there is some issue when we try to build the nodejs container. I am looking into this issue. I will let you know once I have resolved this issue. Meanwhile, feel free to setup using the manual instructions given in the docs: http://evalai.readthedocs.io/en/latest/setup.html#ubuntu-installation-instructions

> Team Display: Adding a few more details, I did not understand the difference between a host team and a participant team. Further, it looked like the dashboard view (as an admin) was displaying all teams, but I realize now this is just for those teams that the user is a member of. I have checked off this part of my original issue as it seems it was a misunderstanding and one that can be resolved when challenge creation/membership is verified.

Yes, you got it right. :)

> Example Challenge: Thanks, for the reference to #1590. My initial confusion about data.py was from the previous sample challenge, where this file was included in the yaml challenge definition; the new example helps to understand the structure a little more. Unfortunately, when using the attachment from #1590, but still see no created challenge after submission. When inspecting the django instance file structure, it does appear that the assets (under /media/ the following directories were created: zip_configuration_files, evaluation_scripts, test_annotations, logos ) were created, but these assets don't appear visible from any of the challenge lists.

Sorry I forgot to inform you that once you have uploaded the challenge then you have to go to the admin interface http://localhost:8000/admin to approve the challenge by setting the flag `is_approved_by_admin` to True using Django Admin. Once you approve it, it will be visible to the other participants.

> team/challenge visibility: Okay, thanks. Aside from making the leaderboard private, I was curious if I could host a "private" challenge that can be joined by invite or hidden URL only. Leaderboard visibility (and possibly challenge visibility itself) is less of a concern. I realize this is a little against the spirit of the open source contributions, but that's that the near-term need was.

We are working on something similar functionality where the challenge hosts can restrict the participant team email domains. For example, you can set that participants with email domain `['@cloudcv.org', '@gatech.edu']` can only join. Similarly, you can block certain email domains for a challenge that you are hosting. For reference, here is the issue #1606.

Please let me know if I am unclear in any of the above responses. 


I regret to push on this directly, but I will be launching an internal hackathon next week.  Are there expectations on fixing this issue (or getting any ability to privately deploy) by ~Wednesday (4/11/18) or should I look elsewhere for the near term?

@ezavesky I understand your concern and I think we have figured out the issue. Earlier what was happening is that we were not able to replicate the issue. Now, we have found it and we would be able to resolve this by today. I will let you know once this is done so that you can use EvalAI for your internal hackathon. I apologize for the delay. 

@ezavesky all the issues that you mentioned in this issue are fixed. Please make sure to pull the latest changes from the master branch. #1645 #1646  fixes the remaining issues that you have mentioned. I am closing this issue for now. Please feel free to reopen if things don't work for you. 

Thanks for all of the extra effort here. I get a new error when trying to start from a clean docker environment (no containers, no images, etc).  It looks like the node commands still have some issues.

@deshraj (PS, I can't reopen this issue)
```
npm ERR! path /root/.npm/_cacache/index-v5/02/90
npm ERR! code ENOTEMPTY
npm ERR! errno -39
npm ERR! syscall rmdir
npm ERR! ENOTEMPTY: directory not empty, rmdir '/root/.npm/_cacache/index-v5/02/90'

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-04-10T10_50_55_258Z-debug.log
ERROR: Service 'nodejs' failed to build: The command '/bin/sh -c npm cache clean -f' returned a non-zero code: 217
```

@ezavesky I think it's npm cache issue. Try running `npm cache verify` then do `npm install` etc. Please let me know in case of anything.

This could be a caching issue on a fresh docker image?  Not quite sure where to poke into an image to do that.  There are no running containers at this point for me to log into.  This was encountered while running the docker compose command `docker-compose up --build`.  I'm just getting the hang of compose testing, so if you have an instruction set to try, I'd be happy to.

@spyshiv knowing you're quite busy, pushing this in with an auto-notification

@ezavesky I deleted again all the containers, images and volumes in my system and tried to build the docker container and it seems to work well now. Please see the log here: https://pastebin.com/6eDrCNV7

Just as an extra precaution, can you please remove `node_modules/`, `bower_components` and `package-lock.json` files before building the Docker containers.

Also, did you try installing through the normal method? Did it work? 

@deshraj Okay, oddly it worked on a mac workstation but not on a deployed linux server running docker-ce as well.  For 'normal' method install, no there were other challenges and limitations that prevented that install from working when starting from a simple ubuntu docker image.  Perhaps I should try again now that the main docker build works.

Thank you for the continued work and check-in.  I will attempt to experiment locally now, but it seems the nodejs docker image is consistently eating some ~20% of CPU, is that normal?   

Given the above two challenges, I have to admit that as a user in my own environment maybe just handing out a canned docker where everything runs would be easiest?  As is, it's pretty cumbersome to launch several and shuttle around many docker images.  

> @deshraj Okay, oddly it worked on a mac workstation but not on a deployed linux server running docker-ce as well. 

This is odd. If it helps, the current docker version that I am using is `Docker version 18.03.0-ce, build 0520e24`.

> For 'normal' method install, no there were other challenges and limitations that prevented that install from working when starting from a simple ubuntu docker image. Perhaps I should try again now that the main docker build works.

Sounds good. Please let me know how it goes.

> Thank you for the continued work and check-in. I will attempt to experiment locally now, but it seems the nodejs docker image is consistently eating some ~20% of CPU, is that normal?

Yeah I just noticed that it is taking ~20% CPU on my machine as well. This might be case because of `gulp dev:runserver`. If you are trying to run this on a server, you might want to use the task `gulp prod` instead of `gulp dev:runserver` and handle the requests through Nginx instead of running it as a Gulp task. Let me know if that makes sense. I can send you the nginx config that we are using to make this happen.

> Given the above two challenges, I have to admit that as a user in my own environment maybe just handing out a canned docker where everything runs would be easiest? As is, it's pretty cumbersome to launch several and shuttle around many docker images.

I strongly agree. I will prioritize this and get this done soon.