zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: submission worker error : 'NoneType' object has no attribute 'get'
Root cause: Validation Gap

When I upload an evaluation script, the system shows the following error messages at the server side
(I follow the instructions in https://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-configuration-examples to create the challenge) 

> worker_1_c60bbe8e7733 | [2019-05-27 17:50:28] ERROR Exception while receiving message from submission queue with error 'NoneType' object has no attribute 'get'
> worker_1_c60bbe8e7733 | Traceback (most recent call last):
> worker_1_c60bbe8e7733 |   File "/code/scripts/workers/submission_worker.py", line 600, in process_submission_callback
> worker_1_c60bbe8e7733 |     process_submission_message(body)
> worker_1_c60bbe8e7733 |   File "/code/scripts/workers/submission_worker.py", line 579, in process_submission_message
> worker_1_c60bbe8e7733 |     user_annotation_file_path,
> n process_submission_callback
> worker_1_c60bbe8e7733 |   File "/code/scripts/workers/submission_worker.py", line 326, in run_submission
> worker_1_c60bbe8e7733 |     challenge_id
> worker_1_c60bbe8e7733 | AttributeError: 'NoneType' object has no attribute 'get'

I know this problem has been discussed in #2254. However, it is close but I cannot find any solution on this problem. 


I found the fix #2277 that makes the submission_worker.py working in DEV environment.


This is fixed.
Repository: Cloud-CV/EvalAI
Issue: Add api to disable a particular challenge
Root cause: Unimplemented Feature Gap

Api should live in challenge app. For adding this api, a field `is_disabled` needs to be present in _Challenge_ model. 

Logic behind the api will be to fetch `creator` from `challenge` object and then check if the current user is challenge_host_team creator or not. This is for v1, may change in v2(where we may add clause something related to status or permissions).

Test and migrations should also be included.
Repository: Cloud-CV/EvalAI
Issue: Cannot submit results into a challenge
Root cause: Validation Gap

- the expected behavior
A challenge was created following the steps provided in [EvalAI Documentation release 1.1.] link here https://buildmedia.readthedocs.org/media/pdf/evalai/latest/evalai.pdf
When logged as host, we should be able to submit into a challenge.

- the observed behavior
  1. I logged as host (also for the participant), when I clicked on the phase a warning came up as shown in the picture below

 	![](https://drive.google.com/uc?export=view&id=1f5ioAz4kMmeziiV5zqqxA_QhnzNlVRr8)

 	The server also reported this:
 	`django_1  | [2019-07-02 10:47:41] WARNING "GET /api/jobs/2/remaining_submissions/ HTTP/1.1" 401 58`.

  2. No submission was allowed. I wasn't able to click on the submission bottom as shown in the picture below
  ![](https://drive.google.com/uc?export=view&id=1KKy1Uzsb2N_T8dvRXByX9i20tTLE-H2U)

  I cleared all docker containers and redo all the steps using the example challenge provided [here](https://buildmedia.readthedocs.org/media/pdf/evalai/latest/evalai.pdf) and still got the same warning and same problem. Note that the example is from this repo.

- steps to reproduce the problem
  1. Get a copy from EvalAI master branch.
  1. Build and run Docker containers following the installation procedure in the master branch.
  1. Go localhost:8888 and uploaded the challenge as a host.
  1. Created the [challenge](https://github.com/Cloud-CV/EvalAI/blob/master/examples/example1/test_zip_file.zip) following the documentation provided by EvalAI 1.1.
  1. Approve the challenge as admin in Django-site, localhost:8000/admin 
  1. Log in as a participant and try to submit a file.

- which operating system and/or browser you're using, if relevant
  Mac OSx, safari

Thanks!



Hi @fhalhani,
Thanks for reporting the issue. We've fixed this issue with this PR: https://github.com/Cloud-CV/EvalAI/pull/2367. Once this is merged then it should not cause further issues. Please feel free to reopen the issue if it doesn't get resolved.

Thanks, @vkartik97 for the quick fix.
Repository: Cloud-CV/EvalAI
Issue: Add remaining integration tests for the worker 

(After Fargate deployment)
- test_extract_challenge_data_succesfully.
- test_run_submission_succesful
- test_load_challenge

Refer this commit: https://github.com/Cloud-CV/EvalAI/pull/2352/commits/493f7cdfcea8f37cd68ee8a6f6aa574ac9a015d9
Repository: Cloud-CV/EvalAI
Issue: Submission worker: Add successful_submission_flag==True condition before saving submission_output
Root cause: Validation Gap

- **Expected behaviour**:
If the submission evaluation is not succesful then the submission output in the database should not be updated.
_Example_: if the challenge splits in the output and the database don't match, this line should not run: https://github.com/Cloud-CV/EvalAI/blob/34cf4572d6938a5f16a4d97b1e8434ee5bb49d75/scripts/workers/submission_worker.py#L513
- **Observed behaviour**:
The submission output in the database is updated irrespective of the contents of the evaluated submission output.

@RishabhJain2018 

Can I work on this issue?

@KhalidRmb Is anyone working on this ? I'd like to take this up 


@yashdusing Sure.

This issue was posted as a task on Google Code-In. I claimed the task, and it seems like this issue has not been resolved yet, so I'll start working on it. Is that fine?

Is the issue still open?
Repository: Cloud-CV/EvalAI
Issue: submission error 
Root cause: Validation Gap

- the expected behavior: 
when submitting a file to the challenge it should show the details of the submission 
- the observed behavior
when a file is submitted to the challenge all submission details are non as shown below 
![](https://drive.google.com/uc?export=view&id=1YtD6-ytled3_pFA3VIP1ynHmr_3z_M2E)
and the following error shows in terminal:
" worker_1  | [2019-07-06 10:06:45] ERROR Exception while receiving message from submission queue with error 'NoneType' object has no attribute 'get'
worker_1  | Traceback (most recent call last):
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 599, in process_submission_callback
worker_1  |     process_submission_message(body)
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 578, in process_submission_message
worker_1  |     user_annotation_file_path,
worker_1  |   File "/code/scripts/workers/submission_worker.py", line 325, in run_submission
worker_1  |     challenge_id
worker_1  | AttributeError: 'NoneType' object has no attribute 'get'" 
This was done using different challenges including the example from EvalAI_starters and the same problem happens every time 
- steps to reproduce the problem
submit a file to the challenge 
- which operating system and/or browser you're using, if relevant
Mac OSx, safari 

Thanks!

Try to restart your worker 
sudo docker-compose restart worker and submit again
Repository: Cloud-CV/EvalAI
Issue: Add username field in response of participant_team_list view
GitHub labels: enhancement

GSOC-2019

easy_to_fix

backend

We also want to send `username` field to the client side so that the Team admin username is visible on the dashboard. 

@deshraj I think this is done in #239. We should close this.

Thanks @RishabhJain2018 for pointing out. 

Fixed in #239. Closing it.
Repository: Cloud-CV/EvalAI
Issue: Update submission API results in 500 error
Root cause: Validation Gap

**Issue:**

The update submission API results into the 500 error when the parameters passed aren't correct. Here is the error: https://sentry.io/share/issue/f8a2925b30604ba0b950abb04853d04d/

**Fix**
All the errors should be properly handled and the response should be displayed to the challenge host.

@RishabhJain2018  Can I work on this?


Hi @RishabhJain2018 isn't this already implemented in the repo [here](https://github.com/Cloud-CV/EvalAI/blob/ca0351dda61be800b2ee50a2b9d4f41b913e8ea9/apps/jobs/views.py#L854) with [tests](https://github.com/Cloud-CV/EvalAI/blob/ca0351dda61be800b2ee50a2b9d4f41b913e8ea9/tests/unit/jobs/test_views.py#L2309). Is something else expected out of this issue?

This is fixed!
Repository: Cloud-CV/EvalAI
Issue: Update emailError key of localStorage on error in the dashboard controller.
Root cause: Validation Gap

### Bug
![Screenshot from 2019-07-21 18-52-13](https://user-images.githubusercontent.com/31621523/61591838-ee487880-abe8-11e9-9c06-99a6c1c1c60c.png)
**Expected** -> For an unverified user, the message displayed should be _"Please verify your email first!"_

**Current** -> The message is "_false_". It os only updated to the former if the user attempts to move to other pages in the menu. (Where localStorage *is* updated on Error.)



@Ram81 @RishabhJain2018 PTAL.

This is fixed!
#2193
Repository: Cloud-CV/EvalAI
Issue: ClientException: An error occurred (ClientException) when calling the CreateService operation: TaskDefinition not ...

https://sentry.io/organizations/cloudcv/issues/1133979609/?referrer=github_plugin

```
ClientException: An error occurred (ClientException) when calling the CreateService operation: TaskDefinition not found.
  File "challenges/aws_utils.py", line 306, in create_service_by_challenge_pk
    response = client.create_service(**definition)

An error occurred (ClientException) when calling the CreateService operation: TaskDefinition not found.
```
Repository: Cloud-CV/EvalAI
Issue: how to access evalai webpage using ip address instead of localhost
Root cause: Architectural Constraint

How to configure local evalAI server to access it using ip address? Currently, it seem to work only with localhost:8888.

Thanks

actually it opens webspage, but when I enter the password nothing happens when opened on a different machine.

As a workaround, you can update the hardcoded `"API": "http://localhost:8000/api/"` to `"API": "http://YOUR_MACHINE_IP:8000/api/"` in the file `EvalAI/frontend/src/js/config.json`, this should solve your problem.

Hi @msubglb Can you please check two things -
1. Is the Django container running?
2. Please check the browser console for the error.


Closing due to inactivity.
Repository: Cloud-CV/EvalAI
Issue: Attribute Error during submission: "no attribute 'evaluate'"
Root cause: External Dependency Breakage

Hi there!
So I created a challenge, admin approved it, restarted the worker. When I make submissions they get shown as "Failed" and the Stderr reads:

```
Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 399, in run_submission
    submission_output = EVALUATION_SCRIPTS[challenge_id].evaluate(
AttributeError: module 'challenge_data.challenge_3' has no attribute 'evaluate'
```
I interpret that so that the evaluation script isn't even called in the first place, but to be sure, mine currently looks like this:

```
import sys
import traceback

def evaluate(test_annotation_file ,user_annotation_file, phase_name, **kwargs):

    original_values = []
    print("Loading Test annotation file ...")
    with open(test_annotation_file, "r") as f:
        for line in f:
            original_values.append(line)
    print("Successfully loaded the test annotation file")

    user_values = []
    print("Loading User annotation file ...")
    with open(user_annotation_file, "r") as f:
        for line in f:
            user_values.append(line)
    print("Successfully loaded the User annotation file")

    score = set(user_values).intersection(original_values).__len__()
    result = {}
    try:
        result['result'] = [
            {
                'main_split': {
                    'score': score,
                }
            },
        ]
        result['submission_metadata'] = {'foo': 'bar'}
        result['submission_result'] = ['foo', 'bar']
        return result
    except Exception as e:
        sys.stderr.write(traceback.format_exc())
        return e
```

Hi @bimtauer, Can you please confirm that you're using the latest master branch? There was a bug related to it earlier but it is fixed now.

Closing this as it is fixed.
Repository: Cloud-CV/EvalAI
Issue: Change color of radio buttons to EvalAI's primary color (Dark Gray)
GitHub labels: GSOC-2019

easy_to_fix

frontend

Change color of radio buttons in Create Team page.
Refer variable.scss file for color themes. Use $dark-gray for the radio button. Currently it is of green color and needs to be changed.

@aka-jain Can i work on this?

@jainarunarunjain sure.


Closing and fixed with #248
Repository: Cloud-CV/EvalAI
Issue: Upgrade challenge examples in EvalAI from Python2 to Python3
Root cause: Environment Incompatibility

**Issue:**

The challenge configuration examples here: https://github.com/Cloud-CV/EvalAI/tree/master/examples/example1 are written in Python2.

**Deliverables**

Update the examples from Python2 to Python3.

Can I work on this

Sure!

Hi @sachinmukherjee Are you still working on this issue? If not, can i take it up?

Hi @RishabhJain2018 I committed the changes and upgraded the code from python2 to python3.

is this issue open or close? 
if it is open can anyone assign it to me

Closing this since it is a very old issue.
Repository: Cloud-CV/EvalAI
Issue: Integrate with EvalAI-ngx

This may already be in the works but I couldn't find any information on it. We are trying to use EvalAI with https://github.com/Cloud-CV/EvalAI-ngx instead of the built in UI. This may be more of a question/request for help than an issue.

This is the Dockerfile we are using for the front end (pretty much just copied from the ngx project):
```
FROM node:12.2.0

RUN apt-get update
RUN apt-get install -y build-essential git curl libfontconfig
RUN apt-get install nodejs-legacy -y
RUN npm install -g @angular/cli@7.3.9

RUN mkdir /code

RUN git clone https://github.com/Cloud-CV/EvalAI-ngx /code/

WORKDIR /code
RUN npm install
RUN npm audit fix

EXPOSE 8888

CMD ["ng","serve", "--host", "0.0.0.0"]
```
When I do a `docker-compose --build` I get this error for the container:
`The serve command requires to be run in an Angular project, but a project definition could not be found.`

This error makes me think there is no `angular.json` file in the `/code` folder but there should be after cloning the repo.

Some documentation on how to connect the back end to the new front end would be very helpful.


Hi, @MichCest I recently set up the https://github.com/Cloud-CV/EvalAI-ngx, I didn't face such an error. Just use `docker-compose up` and would automatically build it for you.

We are actively working to solve this.
Repository: Cloud-CV/EvalAI
Issue: Add and update existing tests for challenge submission API
Root cause: Unimplemented Feature Gap

Reference: #2409 

- Some tests were removed in the above mentioned PR but we need those tests with some updation when we upload a file for challenge submission.

- Also need to add tests when we make submission using URL.

@Sanji515 Can I take up this issue? 


Sure @coldk91ght 

@Sanji515 Ohkay. On it. 👍  

@Sanji515 Can I take this issue?

I'm sorry for the delay. A close relative of mine had passed away and so I couldn't focus on the task. I have put some new tests. I need some help with the successful url tests and to make a sample input file url for testing. These are the updated tests for now: https://github.com/coldk91ght/EvalAI/commit/38480d99cedd377194fcf8d83e89add0b77750ff 

Hi @nikochiko These tests might help you out -  [test_submission_worker.py](https://github.com/Cloud-CV/EvalAI/blob/ca0351dda61be800b2ee50a2b9d4f41b913e8ea9/tests/integration/worker/test_submission_worker.py#L231)

Thanks, @pushkalkatara for the pointer.

@nikochiko Please open up the PR for it.

@pushkalkatara Thanks for the help. Looking into it.
Opening up PR now. Will add changes to it.

Hi @nikochiko . I think these two tests are also required which were removed - [status, file not submitted](https://github.com/Cloud-CV/EvalAI/pull/2409/files#diff-c6268b43fc26de9e08ed355975be069bL424) and [wrong form encoding](https://github.com/Cloud-CV/EvalAI/pull/2409/files#diff-c6268b43fc26de9e08ed355975be069bL445). @Sanji515 Can you comment on this?

> Hi @nikochiko . I think these two tests are also required which were removed - [status, file not submitted](https://github.com/Cloud-CV/EvalAI/pull/2409/files#diff-c6268b43fc26de9e08ed355975be069bL424) and [wrong form encoding](https://github.com/Cloud-CV/EvalAI/pull/2409/files#diff-c6268b43fc26de9e08ed355975be069bL445). @Sanji515 Can you comment on this?

Yes, @nikochiko Those tests need to be added with some updation. Thanks @pushkalkatara for pointing the missing tests

@pushkalkatara , thanks for pointing. @Sanji515 , I'm working on it. 

@Sanji515 Done. Please review.
Repository: Cloud-CV/EvalAI
Issue: Discrepancy in status codes in views.py and test_views.py
Root cause: Validation Gap

https://github.com/Cloud-CV/EvalAI/blob/f59eca2aa8ed8ecf01b340c972e4bde2915010a1/apps/jobs/views.py#L124-L125

The response returns ```status=status.HTTP_400_BAD_REQUEST``` for non-existing challenge.
In test_views.py (https://github.com/Cloud-CV/EvalAI/blob/master/tests/unit/challenges/test_views.py) , lines 369, 658, 1985, 2365, 2596, the code is 
```self.assertEqual(response.status_code, status.HTTP_406_NOT_ACCEPTABLE)``` 
Similarly, for 'challenge phase does not exist', line 134 in views.py.

Okay got it. I'm sorry. I confused jobs/views.py challenges/test_views.py
Repository: Cloud-CV/EvalAI
Issue: Error not displayed when filesize > 100 with url send
Root cause: Validation Gap

When submitting a json file via the url, and the file has more than 100 characters, it fails silently, while answering that the upload was successful.  

It would be better to answer with an explicit error message, like it is the case when uploading directly the file. 



Thanks, @cdancette for pointing it.
@Sanji515 Can you please look into this?

Closing this since we don't have this feature anymore.
Repository: Cloud-CV/EvalAI
Issue: Set DrfDocs to be only available on development server only
GitHub labels: enhancement

GSOC-2019

easy_to_fix

backend

See this: https://github.com/Cloud-CV/EvalAI/blob/master/evalai/urls.py#L24 . This URL pattern should be only available in the development environment. 

@deshraj I will fix this.

@deshra The PR is merged . I think we should close this.

Fixed in #254. Closing it.
Repository: Cloud-CV/EvalAI
Issue: update installation instruction in readme 
Root cause: Documentation Deficiency

When I was setting up EvalAI on my system then for Building and running the Docker containers I had to give the command with admin priviledges(in both windows & ubuntu). 
So ``` sudo  docker-compose up --build``` for ubuntu and running cmd with **Run as Administration** should be there.

Thanks!


Closing this since because it is an installation issue and if the docker is set up properly, it shouldn't need admin privileges.
Repository: Cloud-CV/EvalAI
Issue: The documentation page on 'challenge-creation-using-zip-configuration' does not exist
GitHub labels: priority-high

[This](https://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-creation-using-zip-configuration) link to the documentation page on 'challenge-creation-using-zip-configuration' does not exist.
- the expected behaviour: To open the documentation
- the observed behaviour: Gives page not found error
- steps to reproduce the problem: Click on 'Host Competition' on the main page, and go to your host team, and then got to the link where it says, 'To know how to create a challenge using zip file configuration, please see our documentation here'
-browser: Google Chrome


Thanks!


@VishwaasHegde , I think you are referring to this page: https://evalai.cloudcv.org/web/challenge-create
Also facing same issue here: https://github.com/Cloud-CV/EvalAI-Starters/blob/master/challenge_config.yaml
We can use this link instead: https://evalai.readthedocs.io/en/latest/configuration.html#challenge-configuration.
@RishabhJain2018 There are a few more similar issues with the docs pages. Like a missing '/blob/' in the 'Edit in GitHub' and also, one 'challenge phases' hyperlink refers to this non-existing page: https://evalai.readthedocs.io/glossary.html#challenge-phase
Shall I take up this issue? I will batch a few fixes in one PR.

Thanks, @nikochiko for picking up on this issue and opening a PR.

@VishwaasHegde @RishabhJain2018 , I think we can close this issue.

@nikochiko okay, Thank you :)
Repository: Cloud-CV/EvalAI
Issue: Does evalai push support adding different container registry?
Root cause: Unimplemented Feature Gap

Hello,

Can you provide information on how can one add a different container registry to evalai cli?

Thanks!


Hi @asm582 ,
I didn't quite get your point.
What do you mean by different container registry? Do you mean to point to the docker hub?

yes, how can I configure evalai-cli it to store containers in docker.io or quay.io

@RishabhJain2018 Could you please answer this ? I also have this doubt since most of the functionality which was added was for AWS. So does the CLI support other container registry hubs ? 

Hi @asm582 
Currently, EvalAI only supports AWS' ECR as container registry for challenge submissions, as the project is tightly coupled with AWS.
I hope this resolves your issue.


@vkartik97 is there any get started page for the AWS functionality, k8s code and GPU schedule that was implemented during GSOC? I’d like to understand what all was implemented to have an idea of the functionalities
Repository: Cloud-CV/EvalAI
Issue: UI Design : Signup page displays your password is weak even though password field is empty



**- the observed behaviour**
   When a user tries to sign up the web page, the first enters the password and then clears this field, the sign up page displays your password is weak 
**- the expected behaviour**

   Ideally, it should only display that password is required

**- steps to reproduce the problem**

1. Go to SignUP page
2. First try to enter a random password 
3. Clear this password field 
The screenshot below displays this error. I have circled the error(`weak`) with black marker on the screenshot below. 

<img width="709" alt="Screenshot 2019-11-23 at 11 58 49 PM" src="https://user-images.githubusercontent.com/31419210/69483601-50431400-0e4f-11ea-885b-a895766a34f0.png">

**Solution**
If there is no password in the password field it should only display` password is required`

**System Requirements**
Browser - Safari 
OS - MAC OS 

@RishabhJain2018 can I work on this?

@tusharfc Already there is an open PR for the issue. https://github.com/Cloud-CV/EvalAI-ngx/pull/222

@Kajol-Kumari We should fix this in ngx version.

> @Kajol-Kumari We should fix this in ngx version.

Hey @RishabhJain2018 I just checked it for ngx and this issue is not with ngx it's only with the EvalAI. So should i go for fixing it in EvalAI?