Repository: Cloud-CV/EvalAI
Issue: Add support in submission worker to update file field changes for Challenge and Challenge Phase
Root cause: Unimplemented Feature Gap

__Problem__

If the Challenge Host updates any of `evaluation_script` or `test_annotation` file, then there is no way to propagate that change to the worker.

__Please be more clear__ ?

Presently the submission worker when starts fetches all the active challenges and loads `evaluation_script` for each challenge. It also fetches active challenge phase for each challenge and loads `test_annotation` for each phase. We need to extend that code to add support for handling update requests for these files.

__Why__ ?

Imagine a case where in a Challenge Hosts whose challenge is active and hence running, wants to update `evaluation_script` for a challenge. He logs in and updates the file but still sees that submission are being processed using the older version of `evaluation_script`. He wonders why is this happening ??

So we want to help him and add support to propagate the changes done by him to the worker, because it is the worker who is doing the heavy task of processing submission.

__Impact__ ?

It actually guarantees that whenever any submission is evaluated, it is evaluated against the write set of parameters.

__How__ ?

Overall, this will involve refactoring `scripts/workers/submission_worker.py`. The concerned function which extracts challenge and phase data presently is `extract_challenge_data`. This function may be modified or a new function may be added. This will actually require some **thoughts** from the contributor.

One very important thing which needs to be pointed out is how the worker will detect that the file is changed. Here RabbitMQ comes into action. If you see `add_challenge_queue` in `scripts/workers/submission_worker.py`, it binds on key `challenge.*.*`. The first `*` here will help us decide what the intent of the message is. 

Whenever any new challenge will be added, you can assume that `challenge.add.*` key will be there and whenever any update is done, you can assume that `challenge.edit.*` will be published.

So you can now add checks according to the key and then code accordingly.

For now being you can assume that keys published will be `challenge.add.*` and `challenge.edit.*`.

__TakeAways__

About how the submission worker works in detail, RabbitMQ and its related terminologies. This issue will provide you more depth about RabbitMQ and how queue function.

Also I will prefer that before coding, its better to discuss the approach over here only, so that we can get an idea and update if something is not right.

@TroJan Can I work on this?

@mayank-agarwal-96 : sure :)

@mayank-agarwal-96 : Let me know if you are struck somewhere. Will be happy to help :)

@TroJan : sure :) I am currently reading about RabbitMQ, as I am not experienced with it. I will ask if I get stuck.


@TroJan I was reading this [pull request](https://github.com/Cloud-CV/EvalAI/pull/833/files) created by you for documentation about workers. I noticed that all the evaluation scripts are loaded in a dictionary before a worker fully starts. So whenever a challenge.edit.*  is published, we need to update the `EVALUATION_SCRIPTS` dictionary? Also what needs to be done about previously evaluated submissions?

@mayank-agarwal-96 : Yeah whenever a message for `challenge.edit.*` is published, you need to update `EVALUATION_SCRIPTS` dictionary. If you clearly see the structure of the dictionary, it uses challenge id as its keys. Challenge id has to be available in the message and then you can dynamically import the module and then reload the module in `EVALUATION_SCRIPTS`.

As of now, we are going with the fact that we wont do anything for submissions which are already submitted and evaluated. Hence no need of considering them.

@mayank-agarwal-96 : is this in progress? If you are struck somewhere you can ask it here!

Yes I am still trying to fix this one. I just wasn't able to dedicate proper time. I'll keep posting if I feel stuck.

@mayank-agarwal-96 : any update ?

@TroJan As of now, I am not able to work on the issue because of time constraints. Please reassign if someone wants to take this up. Sorry for the delay.

@TroJan  @mayank-agarwal-96 is this taken?

@yadavankit I think @mayank-agarwal-96 is busy with his exams. So, if you are interested, then let me know. I can assign you the issue. 

@deshraj yeah sure, am interested in doing this one :)

Assigning it to you. :) 

@taranjeet @deshraj . Hello, can this issue be assigned to me. I have read the docs regarding the worker, and it has made some sense. I'll try to clear it within 4-5 days and come up with a working solution.
Thanks

@taranjeet @deshraj . I have read the docs and have drawn some conclusions, please rectify if I'm wrong.
1. `apps/jobs/sender.py` behaves as the `sender/producer` for the `RabbitMQ` broker (which is called by API). 
2. the `main` function in `submission_worker.py` behaves as the `receiver/consumer`.
3. In the docs it is mentioned that `submission_worker.py` works as a daemon. Does this mean it runs continuously in the background? Please clarify.
4. Lastly,since the  challenge can be added and edited from admin panel only (as of now) , how do I **emulate/demonstrate**  `challenge.add.*`  and `challenge.edit.*`.

Please clarify. 
Thanks

yeah on production environment, we run it as daemon process. On development environment, we simply run it as a python process.

Regarding the `challenge.add.*` and `challenge.edit.*`, you will need to write a small script yourself, that creates a connection and then publishes a message. This part is actually missing. We are planning to write test case for it, so that it can be completed and tested end to end.

@taranjeet @deshraj Okay I've a solution (just a prototype). I'll create a script similar to `jobs/sender.py` which will have two functions `challenge_add_publish_message` and `challenge_edit_publish_message`. Since, currently there is no view to invoke these functions, they'll be invoked using command line arguments and they will route the message to the `add_challenge_queue` . Is this the right approach ? 

 

@farziengineer yeah, I too have followed kinda same approach to cater this issue, a better way to invoke the message publishing functions for now would be using `post_save` `signal` from `Challenge` with `created` as `False`. This would invoke the message publishing function whenever `Challenge` details are changed (which can be done from superuser dashboard for now). I will be adding a WIP PR asap. 

@yadavankit Sure :+1: 

@taranjeet @yadavankit Can I take over?

@isht3 feel free to take this up. 

@deshraj Okay, thanks!

@deshraj @RishabhJain2018 : this one also. Can we please close it if not required?

Yeah I agree. We are moving towards using AWS SQS instead of RabbitMQ and hence we won't need this now. Closing this issue.