zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Add API endpoint to create challenge by uploading zip file.
Root cause: Unimplemented Feature Gap

endpoint: `/challenges/zip/$`
method: `POST`
success: `200/OK`
failure: `400/bad request` or `406/not acceptable`

Input: The user must upload a zip file containing the `.yaml` file which describes the challenge configuration. The zip folder will also contain other necessary files which are required to create a challenge.

Output: Upon requesting at this endpoint , a challenge must be created by the user.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to get details about a particular ChallengeHostTeam

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/
Header: Authorization: Token
Method: GET
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Merged via #105
Repository: Cloud-CV/EvalAI
Issue: RandomFileName class returns incorrect path when id is not provided.
GitHub labels: bug

easy_to_fix

backend

**Bug:** The `RandomFileName` class in [base.utils](https://github.com/Cloud-CV/EvalAI/blob/master/apps/base/utils.py#L22) returns incorrect path when `id` is not provided in the path.

**Fix:** The condition on the [line 28](https://github.com/Cloud-CV/EvalAI/blob/master/apps/base/utils.py#L28) needs to be changed to `or` from `and` as `instance.pk` will always be available for upon saving a model.

Can you please explain this in more detail along with an example of where it is failing? 

As per my understanding `and` perfectly makes sense as the string should be substituted with the value of `instance.id` if it exists. Still I want to know where am I missing?

@taranjeet Yes you are correct `and` makes sense here.I understood it incorrectly.Moreover, it is failing when we are calling the class like `RandonFileName("evaluation_script")` rather than `RandomFileName("evaluation_script_{id}") `without giving the `id` & hence due to hard-coded [path](https://github.com/Cloud-CV/EvalAI/blob/master/apps/base/utils.py#L31) all the files are getting saved in `submission_files/something`.
I think we should simply check for `id` parameter in the path, if it is present then we should append it else  save the files on the given path. What do you say?
Repository: Cloud-CV/EvalAI
Issue: Change color schemes for popup notifications
GitHub labels: easy_to_fix

frontend

Default colors for popup notification doesn't looks good and also not matching with our theme colors. We need different colors for popup notification.
@aka-jain : Assigning this to you. You have to share color codes only. Please choose colors according to notification category that's are given below
1. Success
2. Warning
3. Error
4. Info
also provide need four icons for the same.

Yeap, we have to redesign the whole notification system. I'm on it 👍

cool. Let me know if you need helping hands. :grin:  

@aka-jain I think using materialized toast would be a better idea as compared to the current toast we are using.

https://material.angularjs.org/latest/demo/toast

The current toast is quite buggy and lacks a lot of customization.

I think we wanted to shift to the materializecss toasts. @aka-jain can you please confirm this? 

@deshraj can I work on this?

This issue is still open. If someone is interested, feel free to take this up. 

@deshraj Fixed in #1472 .

@aka-jain @spyshiv It was fixed in #1472 I think so, Can we close this issue?

Thanks @gautamjajoo for the update.
Closing this issue
Repository: Cloud-CV/EvalAI
Issue: Change the email verified template.
GitHub labels: bug

easy_to_fix

frontend

priority-high

**Issue:**: The current `email verified` template is according to the old theme.
**Fix**: We need to modify it according to the new theme that we are using it now.
Old Template 
![shot](https://user-images.githubusercontent.com/12206047/30498958-53c6c5d0-9a76-11e7-9bc6-d17d289fbc3e.png)


Can I take this?

@xamfy Sure go ahead!


Since @xamfy seems to be inactive from a long time, @16116073 you can take this task up. Also, feel free to ask your doubts.

Thanks for noticing @obtin. Closing this.
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to get details about a particular ChallengeHost

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: GET
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Merged via #114
Repository: Cloud-CV/EvalAI
Issue: Change the pagination limit in leaderboard
Root cause: Unimplemented Feature Gap

Currently, in the VQA challenge, only 10 entries are visible on the leaderboard and there is no pagination on the client side. To show all the entries of the leaderboard, there are two ways to do that:
1. Add a pagination on the client side
2. Show all the entries on a single page (involves adding a new Pagination Class)

The second approach is a better way since then others will be able to compare their results with everyone else in the team. 


@deshraj Can we add the support for filtering the results on the basis of `participant team` as it will help the participants to be more specific and the overhead of searching team members manually will get reduced. Moreover, for showing all the results we can use pagination. What are your views about it ? 

@RishabhJain2018 we can surely do that but this issue addresses a different problem. Please feel free to open a new issue.
Repository: Cloud-CV/EvalAI
Issue: Add APIs for challenge creation using UI.
GitHub labels: medium-difficulty

backend

new-feature

The API's include:
1. API for creating challenge.
2. API for creating leader board.
3. API for creating challenge phase.
4. API for creating dataset split.
5. API for creating challenge phase split.

Please add tests for the same.

@RishabhJain2018 How is this coming through? Can I help?

@isht3 I am working on this feature. Please feel free to take up any other unassigned issue. :)

@RishabhJain2018 Okay, cool! :D
Repository: Cloud-CV/EvalAI
Issue: Improve the check for if the user is a host of a challenge
GitHub labels: easy_to_fix

backend

critical

new-feature

Currently, at https://github.com/Cloud-CV/EvalAI/blob/master/apps/hosts/utils.py#L12 we only check if the user has created the challegne and based on that we allow the challenge host to see all the submissions (for example, see this https://github.com/Cloud-CV/EvalAI/blob/master/apps/challenges/views.py#L703). We would want to change it so that anyone who is a part of the challenge host team is allowed to access information such as 'View all submissions' etc. 

@deshraj What if we have an API that takes `user` and `challenge_pk` as input and gives the result  as true or false stating whether the user is challenge  host or not. What do you say?

@RishabhJain2018 : Even if we create api, this does not solve our purpose here. At the end, for this issue, you will be calling a function only and not making an internal request to check if user is challenge host or not.
Repository: Cloud-CV/EvalAI
Issue: Update dependencies
Root cause: External Dependency Breakage

- Lot of python dependencies seems outdated'
Please check https://requires.io/github/Cloud-CV/EvalAI/requirements/?branch=master

@aayusharora, thanks for opening this issue. 👍  

If anyone is willing to work on this issue, please take care of the following features:

1. Make sure that nothing is breaking after updating the package
2. Look for the improvements that have been done in the library so that things don't get mess up once merged
3. Please provide the link to the library's latest version and list the latest features added to that version

Hi @deshraj, I would like to work on this issue, could you please assign me to it?

@the-dagger Please go ahead.

@the-dagger Can I take over? :smile: 

@isht3 I've fixed this in #1136 already.
Maybe @deshraj can review it :)


Closing this since we decided to move with the existing version of packages.
Repository: Cloud-CV/EvalAI
Issue: Not able to click on any button in Chrome 

- This issue is related to a specific version of Chrome.

## Steps to Reproduce
-  Setup project in Chrome ( Mac OS X )
- Chrome Version 59.0.3071.115 (Official Build) (64-bit)
- Buttons are non-clickable.

**Note: The website works correctly in Safari**



@aayusharora Can you capture a short video for this please ?

Yes @aka-jain, please check https://drive.google.com/file/d/0BylUgB3l8Lc6VlhBWUNFMXlGQ3M/view?usp=sharing

@aayusharora: Any error in console while clicking on a button??

@spyshiv Hi, the buttons are working fine now. But I can see the error.
![main](https://user-images.githubusercontent.com/12194719/28985861-2e2b5350-7982-11e7-9535-b97232625c80.png)


@aayusharora: also please check the cache issues on your browser. 

@deshraj Can confirm this.
Happens when you open the console and try to click anything on the page.
I believe this is issue related to the responsiveness of the website.

Please check the screen record linked below.
[https://youtu.be/hzQtf68QDfw](https://youtu.be/hzQtf68QDfw)


Yeah, I tried on my latest chrome and it seems to be not working on my mobile phone. 

 I have tried and figure out some points 
 - The issue is related to `animations`
 - The CSS3 animations introduce a `painting` layer on the screen which does not allow any clicks.
 - On mobile, as the page is not responsive, this `painting` layer go above the buttons which seems to be the major cause.
- To check this, remove all animations from CSS and run the project again.
 - Still, I am not able to figure out why it is not working on `iPhone6` while it works on `iPhone6 plus`. The reason for this seems the size of the button on a screen of 313 px ( iPhone 6 ), we can try to set the initial-scale=2 in the viewport to get this fixed ( not sure on this ).

Please suggest your viewpoint @aka-jain @spyshiv @deshraj 


@aayusharora Does this error still exists ? Can you please check once.

@aayusharora Any update on this ?

FYI: using `initial-scale=2` will zoom content of viewport by 200% which is not a solution for this issue.
@aayusharora : IDK if this is fixed but I am closing this issue. Please feel free to reopen this if this issue is not fixed.
Repository: Cloud-CV/EvalAI
Issue: RabbitMQ Worker Forced Shutdown
Root cause: Unimplemented Feature Gap

## Error Log

```shell
  File "scripts/workers/submission_worker.py", line 512, in main
    channel.start_consuming()
  File "/home/ubuntu/.virtualenvs/EvalAI/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming
    self.connection.process_data_events(time_limit=None)
  File "/home/ubuntu/.virtualenvs/EvalAI/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 647, in process_data_events
    self._flush_output(common_terminator)
  File "/home/ubuntu/.virtualenvs/EvalAI/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 420, in _flush_output
    result.reason_text)
pika.exceptions.ConnectionClosed: (320, "CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'")
Sentry is attempting to send 2 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
```

@deshraj I think this means that RabbitMQ was told to shut down (by a human or another program). 
In order to solve it we need to add a shutdown listener which will reconnect it upon any failure or shutdown. Should I go ahead with this approach? 

@deshraj Does this error still exists?

Seems to be fixed. Sorry for the inactivity on this thread.
Repository: Cloud-CV/EvalAI
Issue: Add documentation for preparing the evaluation script 
Root cause: Documentation Deficiency

Currently, the documentation for preparing evaluation script for different kind of challenges (X number of phases and y number of dataset splits) is missing. 

## Deliverables

1. Write documentation about creating simple evaluation script from scratch for a simple challenge of string comparison (see https://github.com/Cloud-CV/EvalAI/blob/master/examples/example1/string_matching.zip for more details)
2. Write documentation for a challenge with 2 challenge phases and 2 dataset splits
3. Add a section with heading `How to make the evaluation faster for my challenge?` giving tips about making the evaluation faster. 

Let me know if there are issues/concerns. 

@deshraj Can we close this issue since it is fixed in #1526 ?

Thanks @gautamjajoo for pointing out. Closing this now.
Repository: Cloud-CV/EvalAI
Issue: Separate Configuration file for API Endpoints
Root cause: Architectural Constraint

All the endpoints used in the project are written inside the files. 
### Problems
- This addresses two major issues
    - Code quality is affected due to the redundancy of same API endpoints
    - Code Readability 
### Proposed Approach 
 - Make a single config.js file containing all End points
 - Call All Endpoints from this Config file.

@spyshiv @deshraj @aka-jain Please suggest if you have some better approaches.


Closing this since we are planning to rewrite most of the part in CloudCV/EvalAI-ngx repository. Feel free to contribute to that repository. :)
Repository: Cloud-CV/EvalAI
Issue: Syntax error while compiling scss
Root cause: Validation Gap

<img width="1437" alt="screen shot 2017-08-05 at 2 15 40 am" src="https://user-images.githubusercontent.com/6565270/28986325-1c4fd37a-7984-11e7-826b-e96152e8069a.png">


Welcome @the-dagger. I think this happens after the last merge.
Do you want to work on this ?

Sure, would love to 🙂 

@the-dagger Please take help from this commit https://github.com/Cloud-CV/EvalAI/pull/1110/commits/3fab1a8ca96f2e6a4dd4ed9a53d8ca23d91566aa.
Repository: Cloud-CV/EvalAI
Issue: Url fix for the remaining submissions url.
GitHub labels: bug

easy_to_fix

frontend

critical

Fix: Please change the url in the [line](https://github.com/Cloud-CV/EvalAI/blob/master/frontend/src/js/controllers/challengeCtrl.js#L912) from 
`"jobs/"+ vm.challengeId + "/phases/"+ vm.phaseId + "/remaining-submissions/";` to 
`"jobs/"+ vm.challengeId + "/phases/"+ vm.phaseId + "/remaining_submissions/";`
Repository: Cloud-CV/EvalAI
Issue: UI issues on signup/login page
GitHub labels: easy_to_fix

I found some UI/UX related issues in the signup page.

- [x] Upon updating profile, the notification displayed has a minor UI issue.
<img width="685" alt="screen shot 2017-08-05 at 1 35 22 pm" src="https://user-images.githubusercontent.com/6565270/28994093-2174d896-79e3-11e7-9cf1-df9c0374a33c.png">

Ideally the exclamation mark should come in the same line as the update message.

- [x] Same issue exists when you sign up

<img width="703" alt="screen shot 2017-08-05 at 1 50 30 pm" src="https://user-images.githubusercontent.com/6565270/28994154-20904f26-79e5-11e7-81fc-93781459fa0b.png">

- [x] In the Sign up page, the password field only blocks copying by keyboard shortcut.
It allows you to copy the password if you do it via mouse (right click and then copy).
This is not a good UX approach and the copying should be either totally allowed or totally blocked.

Browser : Safari
OS : OSX
Repository: Cloud-CV/EvalAI
Issue: Error on Console `Cannot read property 'description' of undefined`
Root cause: Validation Gap

The error appears on the Home page Console, after running the project.

`There should be handling of description variable on client side`

@aayusharora : Are you working on this issue? Assign this to me if you are busy with other issues.

Ok, Go ahead @spyshiv !!
Repository: Cloud-CV/EvalAI
Issue: Add automatic reloading of rabbitmq worker from django admin.
GitHub labels: enhancement

backend

critical


@RishabhJain2018  If its unresolved, can I take this issue?

@RishabhJain2018 Is this available for taking up?

@isht3 No, since we are now planning to shift to AWS SQS instead of RabbitMQ. Thanks for pointing it. I am closing it for now.

@RishabhJain2018 So the whole submission_worker have to be renewed yes?

@isht3: yeah, the submission worker will undergo certain changes where we will remove the RabbitMQ part and use SQS instead. Also, one important thing to notice here is that since SQS cannot run on local machine for testing, so we would have to use some kind of mock messaging service. (There are plenty of them on Github)

So, I guess it will need a common script, development script and a production script in a structure like the settings is done now? The mock messaging service would go in the development script, yes? If it's needed, I can work on the setting up the mock server and testing using something like https://code.google.com/archive/p/mock/
Repository: Cloud-CV/EvalAI
Issue: Sending Email locally
Root cause: Documentation Deficiency

- While Creating a challenge locally, I get notified as `Please Verify your email` while the email has not been sent.

## What is expected?
 
- Email configurations changes for `dev.py` should be mentioned in Readme.md.
- Also, `RESEND EMAIL` feature should be there, in case the user does not get an email.



also when signing up the text "Setting up your details" shows up in white color which is not visible

> also when signing up the text "Setting up your details" shows up in white color which is not visible
@xamfy can you please open an issue for this? 

@aayusharora: please see your terminal console to see the email content. It gets printed there instead of sending a real email that will need a smtp server.
Repository: Cloud-CV/EvalAI
Issue: Fix Notification Message Styling

- [ ] Fix notification on sign up
<img width="414" alt="screen shot 2017-08-06 at 6 24 33 pm" src="https://user-images.githubusercontent.com/6565270/29003583-d7236dbc-7ad6-11e7-8c7c-567ea83b2a12.png">

- [ ] Fixed notification on profile update
Repository: Cloud-CV/EvalAI
Issue: EVALAI Performance Metrics
GitHub labels: bug

easy_to_fix

frontend

new-feature

As the client code base is increasing, we need to focus more on Performance Issues along with additional features, each animation, each `div span` adds to the performance of the page. Hence, to drive more users and to provide the app functionalities on slower connections, we should `measure` the performance and improve it time to time.

I have generated only Homepage Metrics, will add more when we are done with this.
### Lighthouse Report 
[Lighthouse Report.pdf](https://github.com/Cloud-CV/EvalAI/files/1205702/Lighthouse.Report.pdf)

### Google Page Insights Performance Metrics

![evalai](https://user-images.githubusercontent.com/12194719/29040338-399647ee-7bcc-11e7-9a47-af3cab2a300f.png)

 
Some initial Pointers to work upon

- [ ]  Improving the Critical Rendering Path

- [ ]  Following RAIL model to load the animations, instead of loading them on the first load.

- [ ]  Serve images that are appropriately-sized to save cellular data.

- [ ]  Enabling text-compression.

- [ ]  Removing render-blocking stylesheets

- [ ]  Serve WebP images for better compression.

- [ ]  Loading resources asynchronously and deferring the download of unnecessary resources.

- [ ] Improving Accessibility of the app
           - [ ]  Using `alt` tag in images.
           - [ ]  Adding `lang` attribute to `html`
           - [ ]  Using proper color contrast Ratio for text visibility ( UX )

@spyshiv @deshraj Please guide.

 

Thanks @aayusharora. @aka-jain and I had a discussion about some of the points sometime back but we didn't take any action on this since this wasn't the priority. This looks like an exhaustive list of bugs that we definitely want to fix. :) Feel free to take lead on this if you are interested in solving this. 

Thanks. Will try to do my best @deshraj 

Closing this issue for now as we are working on @Cloud-CV/evalai-ngx . @live-wire can you please take care of this in ngx?
Repository: Cloud-CV/EvalAI
Issue: Indefinite progress bar on logging out
GitHub labels: bug

frontend

critical

priority-high

Once you log out, the progressbar is displayed indefinitely and the user is unable to click anywhere on the screen.

Video Capture : 

https://drive.google.com/open?id=0B6OtIpAL6oa6ZWNoTTY0M0NocVU

Browsers : Safari and Chrome


I have seen this but forgot to report. Would you like to work on this @the-dagger ?

Sure. 👍 

@the-dagger Any update on this ?

@RishabhJain2018 Wasn't able to work on it, will take a look tonight.

Okay sure, no problem. :+1: 

Need update, anyone working on this? @RishabhJain2018 

I want to

Sure @xamfy, please go ahead and let me know if you find any difficulty.

Sorry, couldn't work, you can assign this to someone else.

Ok sure. :+1: 

@RishabhJain2018 Since I'm not able to work on the submission issues due to a migration error, can I look into this meanwhile?

@RishabhJain2018 The progress bar is controlled through angular right? Is `startLoader` the command to start the loader?

@RishabhJain2018 If nobody is working on this, can I take this issue?

@anuj-rathore: please work on the other issues assigned to you before working on this one. :) 

@deshraj @aka-jain @spyshiv I think this is resolved. Can you please verify it?

Yeah, I just checked and it seems to be fixed. Thanks for pointing this out.
Repository: Cloud-CV/EvalAI
Issue: Disable the Challenge Phase buttons which are inactive while making submissions
Root cause: Validation Gap

## Observed Behaviour
Currently, if a challenge phase is inactive then also a participant can select the phase to make submissions. 

See screenshot for example:
<img width="1440" alt="screen shot 2017-08-08 at 7 21 16 pm" src="https://user-images.githubusercontent.com/2945708/29098709-ca4c67a8-7c6e-11e7-8729-73122eb9982e.png">

## Expected Behaviour

The challenge phases that are not active now should be ideally disabled and the user shouldn't be allowed to select those challenge phases. 

@deshraj can I take this up?

@Ram81 Yes sure. go ahead!
Repository: Cloud-CV/EvalAI
Issue: Add support for filtering challenges on the basis of challenge phase for host.
Root cause: Unimplemented Feature Gap

**Current Behaviour**: The filtering of submissions under the `View All Submissions` tab for a challenge host is on the basis of `challenge_pk` only. The results are same even if challenge host selects any `challenge_phase`.
**Expected Behaviour**: We want to filter the submissions on the basis of the both `challenge_pk` and `challenge_phase` corresponding to that challenge, so that the result set will only contain the submissions corresponding to the `challenge_phase` that the challenge host selects.

Please let me know in case of further queries.

Can I take this up? :)

@isht3: yeah sure. Assigning it to you. 

@isht3: please accept the invitation as a contributor to CloudCV so that I can assign this issue. 

@deshraj Thank you very much! I did accept.

@RishabhJain2018 Hi, could you provide a small description about this issue? 

@isht3 I have updated the description of the issue. Please take a look and let me know if you have further queries! Moreover, I am assigning this issue to you :)


@RishabhJain2018 Thanks for the clear-cut description, got it done in a giffy!
Repository: Cloud-CV/EvalAI
Issue: Add approved by admin check in seed script
Root cause: Configuration Error

## Observed Behaviour

Currently, when the seed script is ran, then the challenges are created but they are not shown on the client side since the `approved_by_admin` atribute for those challenges is False by default.

## Objective

Set `approved_by_admin=True` for the dummy challenges in the seed script so that they can be viewed on the client side. 

@deshraj can I work on this? 

Sure, go ahead.
Repository: Cloud-CV/EvalAI
Issue: Add link for challenge creation using zip file examples in read the docs. 
GitHub labels: enhancement

easy_to_fix

docs

Add the link for https://github.com/Cloud-CV/EvalAI-Examples repository in evalai docs under the section https://evalai.readthedocs.io/en/latest/challenge_creation.html#challenge-creation-using-zip-configuration

Go ahead :)

@RishabhJain2018 Working on it!