Repository: Cloud-CV/EvalAI
Issue: Add support for different versions of same packages for different Challenges
Root cause: Unimplemented Feature Gap

## Problem 

The current challenges on EvalAI use the same version of the libraries like `numpy`, `scipy`, `matplotlib` etc. But this needs to be changed since different challenges will want different version of the same libraries. 

Possible solutions that I can think of are:

- Different submission workers for each challenge
- Run the submission worker in different docker containers

Discussion/suggestions are welcome!  

@deshraj, Whenever we make a new challenge, the versions of the software required are given in the `requirements.txt` inside `evaluation_script`, right?

@RishabhJain2018 I'd like to work on this. :smile: 

@isht3: do you suggest a way to implement this? We have been brainstorming about this. Let us know if you have some ideas about how to do this. 

@RishabhJain2018 I will definitely look into this issue and get back to you within few days.

@deshraj I think splitting the submission_worker would be the best way to go. What do you think?

@isht3: correct. If you want to work on it and would like to propose the workflow (with SQS), feel free to do it. 

Hey, I'd like to work on this. I was thinking that instead of creating a docker container or running a different worker, I was thinking that while a submission is being evaluated, the `cwd` of the worker can be changed and the required packages can be locally installed in the submission directory using `pip install numpy -t .` command which will cause the version specified to be installed in the submission directory. These should then get used instead of the global packages.

Thanks for the input @party98. I think now we are pretty much clear on how we want to implement this. 

We will go with the challenge based worker approach where for each challenge, there will be dedicated workers so that if one goes down for some reason, it doesn't affect the other challenges. 

This is how the solution will look like:

1. Have a `requirements.txt` when creating a challenge
2. When we load the evaluation script in the submission worker, we will install the requirements in the docker container before extracting the evaluation script. This should be straightforward.

@RishabhJain2018 has already worked on the challenge based workers approach for running submission workers, not inside docker. You can find it in #1692 but a lot of code in this PR is deprecated since we have replaced RabbitMQ with SQS for better management. 

Please feel free to ask questions if you have any. If you are able to finish it, then you will get to understand the core of how EvalAI evaluates all the submissions. Looking forward to working with you on this issue. :) 


> We will go with the challenge based worker approach where for each challenge, there will be dedicated workers so that if one goes down for some reason, it doesn't affect the other challenges.

@deshraj 
One solution can be using multiple workers and each worker for all challenges. If one worker goes down, any challenge won't be affected.

Closing this since we now support challenge based worker.