Repository: Cloud-CV/EvalAI
Issue: Cannot create a challenge on hosted EvalAI 
Root cause: Configuration Error

Hello, 

I just discovered your exciting project and I am trying to set it up locally. However, I have the following problem when I try to create a new challenge.

The setup:
1. I forked and downloaded the EvalAI project locally
2. Started using `docker-compose up --build`
3. Created a new `challenge_config.zip` by cloning EvalAI-Starters and running ./run.sh
4. I created the challenge via the EvalAI web using the host account
5. I logged into the admin panel and approved the challenge
6. I do `docker-compose restart worker` (tried also docker restart eval_ai_worker) and when the worker restarts I get the following error:
```1  | Traceback (most recent call last):
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 216, in extract_challenge_data
worker_1  |     challenge_module = importlib.import_module(CHALLENGE_IMPORT_STRING.format(challenge_id=challenge.id))
worker_1  |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
worker_1  |     return _bootstrap._gcd_import(name[level:], package, level)
worker_1  |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
worker_1  |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
worker_1  |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
worker_1  | ModuleNotFoundError: No module named 'challenge_data.challenge_3'
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
worker_1  |     "__main__", mod_spec)
worker_1  |   File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
worker_1  |     exec(code, run_globals)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 562, in <module>
worker_1  |     main()
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 543, in main
worker_1  |     load_challenge(challenge)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 230, in load_challenge
worker_1  |     extract_challenge_data(challenge, phases)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 216, in extract_challenge_data
worker_1  |     challenge_module = importlib.import_module(CHALLENGE_IMPORT_STRING.format(challenge_id=challenge.id))
worker_1  |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
worker_1  |     return _bootstrap._gcd_import(name[level:], package, level)
worker_1  |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
worker_1  |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
worker_1  |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
worker_1  | ModuleNotFoundError: No module named 'challenge_data.challenge_3'```

Help is much appreciated


@kosio instead of the last step, coulf you try these ?
1: ‘docker container ls’ while the server is running in the background
2: Check the container id of EvalAI worker and copy it.
3: docker stop container_id
4: docker start container_id


And please try doing it with the example test_config.zip in the examples folder

Hey @yashdusing 

Thank you for your suggestions, unfortunately I get the same error.

Here is what I did
1. I loaded the challenge from the example folder (only changed the end-date to 2019 instead of 2017)
2. I restarted the worker container using the steps you suggested. 

The error is still
```worker_1  | [2019-02-06 15:50:01] ERROR Exception raised while creating Python module for challenge_id: 2
worker_1  | Traceback (most recent call last):
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 216, in extract_challenge_data
worker_1  |     challenge_module = importlib.import_module(CHALLENGE_IMPORT_STRING.format(challenge_id=challenge.id))
worker_1  |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
worker_1  |     return _bootstrap._gcd_import(name[level:], package, level)
worker_1  |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
worker_1  |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
worker_1  |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
worker_1  | ModuleNotFoundError: No module named 'challenge_data.challenge_2'
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
worker_1  |     "__main__", mod_spec)
worker_1  |   File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
worker_1  |     exec(code, run_globals)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 562, in <module>
worker_1  |     main()
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 543, in main
worker_1  |     load_challenge(challenge)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 230, in load_challenge
worker_1  |     extract_challenge_data(challenge, phases)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 216, in extract_challenge_data
worker_1  |     challenge_module = importlib.import_module(CHALLENGE_IMPORT_STRING.format(challenge_id=challenge.id))
worker_1  |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
worker_1  |     return _bootstrap._gcd_import(name[level:], package, level)
worker_1  |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
worker_1  |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
worker_1  |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
worker_1  | ModuleNotFoundError: No module named 'challenge_data.challenge_2'```

@kosio are you sure your installation went smoothly?
Could you try rebuilding from scratch (take down the running containers first) ?

I tried cloning and rebuilding from scratch and everything seems to pass, but I still have the problem when I upload a new challenge. 

Just a note, I only have a problem with the worker when I upload a new challenge. When I submit a random file to the challenge, which is set up by default, the worker completes and I get an evaluation score. 

From the error message, it seems that the worker is not able to load the evaluation script for the new challenge on start, but I have not yet found where to fix this. 

Not sure if relevant, but I'm running this on MacOS

I just tried it on a clean Amazon Linux machine and got the same error

I finally fixed the issue. Seems like a problem that should be happening to more people. Here is the one-line fix in my fork, let me know if you'd like me to open a PR here as well
https://github.com/kosio/EvalAI/commit/2e546fb93ff24ef7e5b4d401fe08b48f228a4e4b

@kosio Thanks for fixing this.
Opening a PR would be great!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically closed as there is no further activity. Thank you for your contributions.