Repository: Cloud-CV/EvalAI
Issue: Improve Validate Challenge Config
Root cause: Validation Gap

### Description

Currently, there are lots of issues with the validate challenge configuration utility present [here](https://github.com/Cloud-CV/EvalAI/blob/19130726790e4cb79b0d2e8603dd6817b16dba69/apps/challenges/challenge_config_utils.py#L223).

Here are some issues which this has been reported in the past:
- Removing `test_annotation_file` from `challenge_phases` throws Internal Server Error.
- Removing `is_leaderboard_order_descending` from `challenge_phase_splits` also throws errors.

The issue with such cases is that the hosts often remove these things and reach out to us with errors and it becomes challenging for us to figure out where the script is exactly failing.

Ideally, we would like for the validation script to handle each corner case properly and handle the exceptions appropriately so as to have verbose errors when something is missing or is failing. Additionally, if something is not really required by the configuration, we should allow the hosts to skip it from the configuration. For example `test_annotation_file` is not necessarily needed in code-upload challenges.

### Pre-requisites/Learnings
This issue essentially requires background on a few things:
- An understanding of the [challenge configuration](https://evalai.readthedocs.io/en/latest/configuration.html).
- A high-level understanding of [EvalAI challenges](https://evalai.readthedocs.io/en/latest/host_challenge.html).
- How the [challenge validation](https://github.com/Cloud-CV/EvalAI/blob/19130726790e4cb79b0d2e8603dd6817b16dba69/apps/challenges/challenge_config_utils.py#L223) works.

### How to Approach
- You can start by setting up EvalAI locally or hosting a challenge on staging
    - To host a challenge on staging, please follow the steps [here](https://evalai.readthedocs.io/en/latest/host_challenge.html#host-challenge-using-github).
    - (**Recommended**) Set up the dev version locally (preferred) using the steps [here](https://github.com/Cloud-CV/EvalAI/tree/master#installation-instructions).
- Play around with the challenge configuration to see what fails, put up the errors in a draft PR as a comment, along with the appropriate changes to fix it.
- Reach out to us regarding any issues that you face or if you need any help with understanding any of the above.


Hi! 
I want to take on the issue. I have set up the dev version of EvalAI locally and started working on it.

Please assign the issue to me.

Hi,
I am working on it.

Hey @AyushR1, Since you commented first, you can work on this one.

@b9aurav Please take up any other issue. We will open issues soon.

cc: @gchhablani @Ram81 

Hey! I have been working on the issue and after successfully setting up everything locally,  I modified the host challenge using GitHub script to work locally, I have recreated the issues.

I have been playing around to see what more things have been throwing Internal Service Errors on removing them from the YAML file (challenge_config.yaml)  and have found many more things with the same as we have seen on removing `test_annotation_file`. 

**I wanted to know, How should I be telling the verbose errors now? Should I tell it as soon as something is missing from the YAML (challenge_config.yaml) or should I only tell the ones which throw the Internal Service Error?** 

**Also removing `is_leaderboard_order_descending` didn't cause any errors to show up for me. Please tell on how to proceed with this**

@AyushR1

The best way to go about this would be to create a shared document (markdown or docs) and add:
- What are you removing/changing
- The effect / do you face any errors?
- If yes, what is the exact error log?
- Link to the corresponding build on your public build

After that, we can evaluate what all needs change and it'll help us with your upcoming PR as well.

CC @RishabhJain2018 @Ram81


@gchhablani 
This is the shared notion document with the current findings:-
https://flax-beaufort-4d1.notion.site/Improve-Validate-Challenge-Config-def62fd7cbf4480989a7f30b906a10be

Also, I have made some changes in the util function to fix the Internal service error caused by removing test_annotation_file from challenge_phases and it fixes it:- 
https://github.com/AyushR1/EvalAI/commit/1a26d615bb2469696019343d0c453c3257f5944a
Please check.

@RishabhJain2018 @gchhablani @Ram81 

Based on my findings here https://flax-beaufort-4d1.notion.site/Improve-Validate-Challenge-Config-def62fd7cbf4480989a7f30b906a10be in which I tested what fails.

I have implemented fixes... 

I have tried to implement it in 2 ways  

1st way
https://github.com/AyushR1/EvalAI/commit/6dce143114e774371c06a24208415cbe2f48b7c7
In this, I have added a simple conditional check to see from expected keys and find missing keys in the challenge config YAML file.


2nd way. I have handled the missing keys individually for each modifying the existing code to handle corner cases...
https://github.com/AyushR1/EvalAI/commit/20088fe8752ffd0b7a6599f89468739037d2e756
https://github.com/AyushR1/EvalAI/commit/a97ec01ac555b0edf55af664800b80298f594e07
https://github.com/AyushR1/EvalAI/commit/836b7d254c5bf6df4d601f6d5abb7bf2994e9904
https://github.com/AyushR1/EvalAI/commit/4c9fae1cf33c992e0bdc8499e2771659e16b76c8

I believe the 2nd way is more verbose and tells all the errors in the YAML files. 

Please provide further insights.

Edit:- I have implemented the fixes for all the corner cases going with the 2nd wayl.