Repository: Cloud-CV/EvalAI
Issue: Submission Queue receives NoneType
Root cause: Validation Gap

I recently cloned and stood up evalAI via the documented docker compose method for development.
I also cloned the challenge template as described in the documentation.

After submitting the template with some minor changes (I can provide if needed), the challenge uploads successfully and is present at http://localhost:8888/web/challenges/challenge-page/21/submission

However, upon attempting to submit a result file for evaluation, I get the following error from docker compose's worker 1 container:
```django_1     | [2020-11-16 20:04:06] INFO Submission is below for user 1 form participant_team 3 for challenge_phase 41
worker_1     | [2020-11-16 20:04:06] INFO WORKER_LOG Processing message body: {"challenge_pk": "21", "phase_pk": "41", "submission_pk": 401}
worker_1     | [2020-11-16 20:04:06] INFO WORKER_LOG Processing message body: {"challenge_pk": "21", "phase_pk": "41", "submission_pk": 401}
worker_1     | [2020-11-16 20:04:06] INFO SUBMISSION_LOG [x] Received submission message {"challenge_pk": "21", "phase_pk": "41", "submission_pk": 401}
worker_1     | [2020-11-16 20:04:06] INFO SUBMISSION_LOG [x] Received submission message {"challenge_pk": "21", "phase_pk": "41", "submission_pk": 401}
django_1     | [2020-11-16 20:04:06] INFO "GET /media/submission_files/submission_401/54f26004-68c8-4c0e-afd5-4ec5515fefc1.json HTTP/1.1" 200 3
worker_1     | [2020-11-16 20:04:06] ERROR SUBMISSION_LOG Exception while receiving message from submission queue with error 'NoneType' object has no attribute 'get'
worker_1     | Traceback (most recent call last):
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 652, in process_submission_callback
worker_1     |     process_submission_message(body)
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 628, in process_submission_message
worker_1     |     user_annotation_file_path,
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 374, in run_submission
worker_1     |     challenge_id
worker_1     | AttributeError: 'NoneType' object has no attribute 'get'
worker_1     | [2020-11-16 20:04:06] ERROR SUBMISSION_LOG Exception while receiving message from submission queue with error 'NoneType' object has no attribute 'get'
worker_1     | Traceback (most recent call last):
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 652, in process_submission_callback
worker_1     |     process_submission_message(body)
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 628, in process_submission_message
worker_1     |     user_annotation_file_path,
worker_1     |   File "/code/scripts/workers/submission_worker.py", line 374, in run_submission
worker_1     |     challenge_id
worker_1     | AttributeError: 'NoneType' object has no attribute 'get'
django_1     | [2020-11-16 20:04:06] INFO "POST /api/jobs/challenge/21/challenge_phase/41/submission/ HTTP/1.1" 201 649```



Hi @tokotchd, Upon creation of the challenge, you will need to restart the worker container which will load the required challenge files for the evaluation to work.

Please feel free to ask if the issue doesn't resolve after restarting the worker container. 

Thanks, @RishabhJain2018, this indeed did solve my issue.

I'm guessing this doesn't happen in production... How exactly does it avoid this though?

Does the worker auto restart in production after a new challenge is added?  Or are there other measures in place that prevent needing a restart that aren't in the development compose file?

EDIT: Looks like it's the same issue/solution with changing/updating the evaluation script for a challenge not just creating a new challenge.  I imagine your answer will be the same in both cases though.

Also on that note, I'm having an issue where submission results are always marked as "Failed" despite the fact that the script in question runs successfully, printing "success" to standard out just before returning a dictionary object that has key "result":

```
result['result'] = <some_value>
print('success')
return result
```

~~Does this also have to do with changes on the development branch or should I open a new issue for this?~~

I kept digging due to above issue and found the following error in stderr output of the worker:

```ORGINIAL EXCEPTION: No such relation between Challenge Phase and DatasetSplit specified by Challenge Host 
Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 499, in run_submission
    dataset_split__codename=split_code_name,
  File "/usr/local/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
challenges.models.DoesNotExist: ChallengePhaseSplit matching query does not exist.
```

I'm having trouble debugging this as well though, because I'm using the default challenge provided in:
https://github.com/cloud-CV/evalai-starters

Any ideas?

Looks like it was an issue having to restart most of the services via docker compose in order for the challenge-phase split to be found by the workers.  

My question remains:
Is there a better solution for these issues than continuously restarting the worker containers in dev mode?

@tokotchd for dev worker to pick up new challenges restart is needed.

Closing this issue

 @tokotchd Hi, I have the same problem as you. And after restart worker container, it is work. But do I need to restart the worker container every time after creating a new competition? Have you solve it?