zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: Migrate from Travis CI to GitHub Actions for CI/CD
Root cause: Unimplemented Feature Gap

**Description:**

We should migrate the EvalAI CI/CD pipeline from **Travis CI** to **GitHub Actions**, which is now the more widely adopted and integrated CI solution for GitHub repositories and is free.

A draft migration was already initiated in [[PR #4576](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576), which includes the basic setup of GitHub Actions workflows for backend testing.

**Motivation:**

* Travis CI has reduced OSS support and slower performance.
* GitHub Actions provides better integration, caching, and visibility.
* Unifying CI workflows with the GitHub ecosystem will simplify development and collaboration.

**Tasks:**

* [ ] Finalize and merge [[PR #4576](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576) with necessary fixes and enhancements.
* [ ] Ensure the backend test suite runs reliably on GitHub Actions (including database services, coverage, etc.).
* [ ] Set up CI checks for linting, migrations, and static type checks (if any).
* [ ] The deployment runs from the Github CI.
* [ ] Decommission `.travis.yml` and remove Travis-specific config/integration from the repository.
* [ ] Update documentation to reflect the CI migration (e.g., in `README.md` or `CONTRIBUTING.md`).

**Acceptance Criteria:**

* GitHub Actions fully replaces Travis CI for all pull requests and merges.
* All tests and jobs pass successfully and consistently on GitHub Actions.
* Travis CI integration is removed both from the repo and any external services.

**References:**

* Draft PR for GitHub Actions setup: [[#4576](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576)
Repository: Cloud-CV/EvalAI
Issue: Add search and filter functionality for challenges

**Description:**

To improve the usability and discoverability of challenges on the platform, we should implement a search and filter feature on the frontend challenges list page.

A starting point for this work was laid out in [[PR #4175](https://github.com/Cloud-CV/EvalAI/pull/4175)](https://github.com/Cloud-CV/EvalAI/pull/4175), which introduces a basic search capability. This can be expanded and refined to include advanced filtering options and improve the UX.

**Motivation:**

* As the number of hosted challenges grows, users need an efficient way to find specific challenges.
* A clean and functional UI/UX for filtering by attributes (e.g., category, status, submission type) will significantly enhance the experience for both participants and organizers.

**Tasks:**

* [ ] Review and build upon [[PR #4175](https://github.com/Cloud-CV/EvalAI/pull/4175)](https://github.com/Cloud-CV/EvalAI/pull/4175) (which introduced a basic search box).
* [ ] Add filters for challenge attributes (e.g., ongoing/past, evaluation type, organization, etc.).
* [ ] Integrate search and filter with existing API endpoints (or update backend endpoints if necessary).
* [ ] Ensure filters and search work together smoothly (e.g., combined filtering and keyword search).
* [ ] Make the UI responsive and accessible.

**Reference:**

* Initial implementation PR: [[#4175](https://github.com/Cloud-CV/EvalAI/pull/4175)](https://github.com/Cloud-CV/EvalAI/pull/4175)

@RishabhJain2018 /assign
Repository: Cloud-CV/EvalAI
Issue: Send EvalAI Paid Plan Details Email on "Request for Approval" Action

**Description:**

When a challenge host clicks on the **"Request for Approval"** button in the EvalAI UI, we should automatically send them an email containing the details of EvalAI's paid subscription plans.

This will ensure transparency about the new billing model and provide hosts with clear next steps before their challenge is reviewed and approved.

**Motivation:**

* With the move to a subscription-based model, challenge hosts must be aware of the paid plans before approval.
* Automating this communication ensures consistency and reduces manual intervention by the team.

**Tasks:**

* [ ] Hook into the existing "Request for Approval" flow on the backend.
* [ ] Create or update an email template with plan details (cost, what's included, Stripe payment link, etc.).
* [ ] Automatically send the email to the challenge creator (and any co-hosts if applicable) when the request is made.
* [ ] Log or track the email-sending status for audit/debugging.
* [ ] Optionally include a contact email or support link for questions.
Repository: Cloud-CV/EvalAI
Issue: Add Missing Test Cases for Key API Endpoints and Backend Apps

**Description:**

This issue is to audit and improve test coverage for important API endpoints and backend modules that currently lack sufficient test cases. Ensuring robust coverage will enhance EvalAI's reliability, make future changes safer, and reduce the likelihood of regressions.

**Checklist:**

Please check whether tests exist for each of the following, and if not, add them:

---

### ✅ **API Endpoints**

#### **Auth & Accounts**

* [ ] `POST /auth/register/`
* [ ] `POST /auth/login/`
* [ ] `GET /accounts/user/get_auth_token`
* [ ] `GET /accounts/user/refresh_auth_token`
* [ ] `POST /auth/password/change/`
* [ ] `POST /accounts/user/resend_email_verification/`

#### **Challenges**

* [ ] `PUT /challenges/{challengeId}/manage_worker/{action}/`
* [ ] `GET /challenges/{challengeId}/request_approval`
* [ ] `GET /challenges/{challengeId}/get_worker_logs/`
* [ ] `GET /challenges/challenge/{challengeId}/prizes/`
* [ ] `GET /challenges/challenge/{challengeId}/sponsors/`
* [ ] `POST /challenges/{challengeId}/participate/`
* [ ] `PUT /challenges/{challengeId}/scale_resources/`
* [ ] `GET /challenges/challenge/{challengeId}/get_participant_teams`
* [ ] `POST /challenges/challenge/{challengeId}/deregister/`

#### **Jobs**

* [ ] `GET /jobs/phase_split/{phaseSplitId}/public_leaderboard_all_entries/`
* [ ] `PATCH /jobs/challenges/{challengeId}/submissions/{submissionId}/update_submission_meta/`

---

### ✅ **Backend Apps & Files**

#### **Models**

* [ ] `apps/hosts/models.py`
* [ ] `apps/jobs/models.py`
* [ ] `apps/base/models.py`
* [ ] `apps/participants/models.py`
* [ ] `apps/accounts/models.py`
* [ ] `apps/challenges/models.py`

#### **Serializers**

* [ ] `apps/challenges/serializers.py`
* [ ] `apps/participants/serializers.py`
* [ ] `apps/accounts/serializers.py`

#### **Views & Admin**

* [ ] `apps/web/views.py`
* [ ] `apps/challenges/admin.py`
* [ ] `apps/web/admin.py`
* [ ] `apps/participants/admin.py`
* [ ] `apps/jobs/admin.py`
* [ ] `apps/hosts/admin.py`

#### **Utilities & Celery Tasks**

* [ ] `apps/base/utils.py`
* [ ] `apps/jobs/utils.py`
* [ ] `apps/jobs/tasks.py`
* [ ] `apps/jobs/sender.py`
* [ ] `apps/challenges/utils.py`
* [ ] `apps/hosts/utils.py`
* [ ] `apps/challenges/aws_utils.py`
* [ ] `evalai/celery.py`

---

**Expected Outcome:**

* Identify all components above that lack tests.
* Add unit tests as appropriate.
* Ensure that new tests follow existing conventions.
* Verify increased coverage via `pytest --cov` or Codecov integration (once available).

Hey @RishabhJain2018 i have rebased all the PRs
Repository: Cloud-CV/EvalAI
Issue: Integrate Postman API Tests into GitHub Actions CI Pipeline

**Description:**

To improve backend API reliability, we should integrate **Postman API tests** into EvalAI’s **GitHub Actions CI pipeline**. This will ensure that core API endpoints remain functional and are not unintentionally broken by future pull requests — especially since current CI primarily validates backend unit tests and frontend builds.

This was originally proposed for Travis CI, but since we are migrating to **GitHub Actions**, the integration should be done there.

**Plan:**

### ✅ Step 1: Create Postman Test Assets

* [ ] Create a folder at the root of the repo (e.g., `postman-tests/`)
* [ ] Add the following files:

  * `evalai-api-tests.postman_collection.json`: Contains the API test cases.
  * `evalai-env.postman_environment.json`: Holds base URL, auth tokens, etc.

### ✅ Step 2: Add GitHub Actions Workflow

* [ ] Use a GitHub Actions workflow to:

  * Set up Node.js
  * Install [[Newman](https://github.com/postmanlabs/newman)](https://github.com/postmanlabs/newman), the CLI tool for Postman
  * Run the collection:

    ```bash
    newman run postman-tests/evalai-api-tests.postman_collection.json \
      -e postman-tests/evalai-env.postman_environment.json
    ```
  * Fail the build if any API test fails

**Additional Notes:**

* This should be coordinated with or follow the [[Travis → GitHub Actions migration](https://github.com/Cloud-CV/EvalAI/pull/4576)](https://github.com/Cloud-CV/EvalAI/pull/4576).

Hi! I'm a beginner contributor interested in DevOps and CI, and this issue looks like a great learning opportunity. May I please take this up and try it out? I may ask for help along the way. 🙂


@RishabhJain2018  /assign


Hi @RishabhJain2018 , I’d love to take this up if it’s still available. I can set up the Postman tests and integrate them with GitHub Actions as described. Could you please assign this issue to me?
Repository: Cloud-CV/EvalAI
Issue: Improve Test Coverage for authCtrl.js (Current: 53.92%)

**Description:**

The file `EvalAI/frontend/src/js/controllers/authCtrl.js` currently has **incomplete test coverage**, with the latest Codecov report showing only **53.92%** coverage. This controller handles critical user-facing functionality like sign-up, login, password reset, and JWT token handling, so untested logic introduces a significant risk of regressions.

This issue aims to **identify and test the remaining untested flows**, including UI state changes, form logic, and asynchronous API responses.

---

### ✅ Areas to Improve Test Coverage

#### 1. **UI State Toggles and Utilities**

* [ ] Add unit tests for:

  * `togglePasswordVisibility()`
  * `toggleConfirmPasswordVisibility()`
* [ ] Add test for `checkStrength()` to validate:

  * Password strength detection
  * Updates to `vm.showPasswordStrength`, `vm.message`, `vm.color`

#### 2. **Asynchronous Request Callbacks**

* [ ] Add tests for:

  * `userSignUp()` and `userLogin()` using `utilities.sendRequest()` mock
  * Simulate success and error callbacks
  * Test post-registration auto-login and redirection
  * Validate 400 error responses and field error rendering

#### 3. **JWT Refresh Token Logic**

* [ ] Add tests for:

  * `resetPassword()`: success (email sent) and failure (invalid or inactive account)
  * `resetPasswordConfirm()`: success case and error parsing for `token`, `new_password1`, and `new_password2`

#### 4. **Password Reset Flows**

* [ ] Duplicate coverage with above if needed:

  * Comprehensive test for both reset initiation and confirmation paths

#### 5. **Loader and Form State Management**

* [ ] Add tests for:

  * `startLoader()` and `stopLoader()` — test `$rootScope.loaderTitle`, and `.page-loading-container` class manipulation
  * `resetForm()` — test reset of login/signup/reset form state variables

#### 6. **State Change Listener**

* [ ] Add test for:

  * `$rootScope.$on('$stateChangeStart')` to verify that `resetForm()` is triggered correctly during route transitions

---

### 📈 Goal

Increase test coverage of `authCtrl.js` from **53.92% to at least 90%**, covering all major interaction flows and edge cases.
Repository: Cloud-CV/EvalAI
Issue: Improve Test Coverage for `permissionCtrl.js` (Current: 36.36%)

**Description:**

The file `EvalAI/frontend/src/js/controllers/permissionCtrl.js` currently has **low test coverage**, with only **36.36%** according to the latest Codecov report. This controller includes important logic such as the ability for users to request a new email verification link, which is currently **under-tested**.

The `requestLink()` function, in particular, handles a critical piece of user interaction and should be thoroughly tested to prevent regressions and ensure a reliable user experience.

---

### ✅ Focus Area: `requestLink()`

The `requestLink()` method sends a **POST** request to trigger a new email verification. It handles different response scenarios and interacts with the UI via `$rootScope.notify()`.

#### Tests to Add:

* [ ] Ensure `utilities.sendRequest()` is called with the correct:

  * URL
  * HTTP method (`POST`)
  * Token
* [ ] Simulate **successful POST** response:

  * Assert that `vm.sendMail` is set to `true`
  * Assert that `$rootScope.notify()` displays a **success message**
* [ ] Simulate **400/500 error** response:

  * Assert that a **generic error notification** is triggered via `$rootScope.notify()`

---

### 📈 Goal

Increase test coverage of `permissionCtrl.js` to **above 90%**, with full test validation of `requestLink()`'s behavior across success and failure flows.
Repository: Cloud-CV/EvalAI
Issue: Improve Test Coverage for `challengeCtrl.js` (Current: 61.24%)

**Description:**

The file `EvalAI/frontend/src/js/controllers/challengeCtrl.js` currently has **61.24%** test coverage (as per Codecov). To improve this, comprehensive unit tests need to be added targeting untested and partially covered functionality — especially around API-driven logic, asynchronous behaviors, and UI state transitions.

This issue proposes enhancing coverage through **mocking, controlled inputs**, and **Jasmine’s `spyOn`** to simulate and verify the behavior of key functions.

---

### ✅ Key Areas to Cover

#### 1. **API Interactions**

Functions: `manageWorker`, `sendApprovalRequest`, `getTeamName`

* [ ] Mock `utilities.sendRequest`
* [ ] Validate:

  * Proper API parameters
  * Success callback handling
  * Error handling with appropriate alerts or state fallback

#### 2. **UI Controls & Toggles**

Functions: `toggleParticipation`, `toggleLeaderboard`, `toggleShowLeaderboardByLatest`

* [ ] Mock:

  * `$mdDialog`
  * `$interval`
  * `$rootScope.notify`
* [ ] Validate correct toggling of UI states and user feedback messaging

#### 3. **Leaderboard & Time-based Logic**

Function: `getAllEntriesOnPublicLeaderboard`

* [ ] Mock timestamps using `moment.js`
* [ ] Validate:

  * Proper leaderboard grouping by time periods (e.g., "Today", "Yesterday")
  * Accurate date formatting and category rendering

#### 4. **Form & Submission Metadata Handling**

Functions: `loadPhaseAttributes`, `isCurrentSubmissionMetaAttributeValid`, `toggleSelection`

* [ ] Simulate realistic phase and metadata input
* [ ] Test for correct validation logic and state updates

---

### 📈 Goal

Increase test coverage for `challengeCtrl.js` to **90%+**, ensuring:

* All asynchronous behaviors are properly tested
* UI state and transitions are fully verified
* External dependencies (e.g., `$mdDialog`, `moment`, `utilities.sendRequest`) are reliably mocked
* Future refactors or regressions are caught early

---
Repository: Cloud-CV/EvalAI
Issue: Improve Test Coverage for `profileCtrl.js` (Current: 66.51%)

**Description:**

The file `EvalAI/frontend/src/js/controllers/profileCtrls.js` currently has **66.51%** test coverage according to the latest Codecov report. Key functions such as `editprofileDialog()` and `changePassword()` remain either **untested** or **partially tested**, leaving important UI and security-related workflows vulnerable to regressions.

This issue proposes targeted unit tests using **Jasmine** and **Karma**, aiming to validate all control paths, UI interactions, and error handling logic.

---

### ✅ Target Functions for Testing

#### 1. **`editprofileDialog(ev)`**

This method configures and opens a modal dialog based on the clicked field. It uses a `switch-case` block driven by `ev.currentTarget.id`.

* [ ] Simulate DOM click events with different `id`s (e.g. `first_name`, `github_url`)
* [ ] Validate that:

  * `vm.titleInput` and `vm.editid` are set correctly
  * `$mdDialog.show()` is called with correct arguments
* [ ] Use `spyOn($mdDialog, 'show')` to assert modal behavior

#### 2. **`changePassword()`**

This function submits an API request to change the user's password and handles validation errors for different fields.

* [ ] Mock `utilities.sendRequest()` to simulate:

  * Success case: test `$rootScope.notify()` and `$state.go()`
  * Error cases for individual fields:

    * `old_password`
    * `new_password1`
    * `new_password2`
  * Generic/unexpected errors
* [ ] Validate that:

  * `vm.FormError` is updated correctly
  * Error messages are passed properly to `$rootScope.notify()`

---

### 🧪 Testing Strategy

* Implement tests using **Jasmine + Karma**
* Use mocking/spying for:

  * `$mdDialog.show()`
  * `utilities.sendRequest()`
  * `$state.go()` and `$rootScope.notify()`

---

### 📈 Goal

Increase test coverage of `profileCtrl.js` to **90%+** by covering all key logic paths, modal workflows, and API result handling.
Repository: Cloud-CV/EvalAI
Issue: 🗂️ Implement Submission & Logs Retention Policy Post-Challenge (1-Month Retention)

**Description:**

To improve data storage efficiency and reduce long-term infrastructure costs, we propose implementing a **retention policy** for **submission files and logs** after a challenge has ended on EvalAI.

### 🎯 Objective

Once a challenge has ended, **submission metadata should be preserved**, but the following associated artifacts should be **retained for only 1 month**:

* Submission input files (e.g., prediction files, zip uploads)
* Submission output files (e.g., logs, result archives, stdout/stderr logs)

After the 1-month grace period, the storage used by these large artifacts should be released automatically.

---

### 🛠️ Proposed Implementation

#### 1. **Retention Trigger**

* Trigger based on the `end_date` of the challenge phase.
* Start countdown only if the `challenge_phase.is_public` is `False` (i.e., not accepting any more submissions).

#### 2. **Scheduled Cleanup**

* A periodic background job (e.g., via Celery beat or AWS Lambda/CloudWatch Events) that:

  * Identifies expired submissions.
  * Deletes associated large files from storage (AWS S3 or others).
  * Keeps submission records in the database for audit/reference.

#### 3. **Audit Trail / Flags**

* Add a flag like `is_artifact_deleted` to each submission to avoid repeated deletion attempts.
* Log the deletion with timestamps for traceability.

#### 4. **Notification (Optional Enhancement)**

* Notify challenge hosts 14 days before deletion starts via email.
* Allow manual override if a host wants to retain data longer.

---

### 🔐 Considerations

* Ensure challenge host consent is accounted for — either as a configurable retention policy per challenge or platform-wide with clear documentation.
* Submissions from **host users** or **evaluation script outputs** may need a different policy if used for benchmarking or demo purposes.


---

### 📌 Related Work / References
Ref: https://github.com/Cloud-CV/EvalAI/pull/4586
Repository: Cloud-CV/EvalAI
Issue: Add Example for codename Mapping in Challenge Configuration
Root cause: Documentation Deficiency

**Description:**

The `codename` field is a required string used to uniquely identify each challenge phase. It is critical for correctly mapping results returned by the evaluation script to the respective challenge phase.

Currently, documentation mentions the usage of `codename` but lacks a concrete example. To improve clarity for challenge organizers, we should add an example in the relevant documentation section along with it's usage in the evaluation script.

This addition will help prevent confusion and ensure more robust challenge configurations, especially for first-time users.
Repository: Cloud-CV/EvalAI
Issue: Create and Document Challenge Configuration Examples in a Separate Section
Root cause: Documentation Deficiency

**Description:**

To help challenge organizers better understand how to structure their configuration files, we propose creating a dedicated documentation section that showcases common challenge setup examples. This will make it easier for users to get started and avoid errors when customizing their own configurations.

### 📚 Proposed Examples to Include

Each example should include a fully functional `config.yaml` file with explanations for each component:

---

#### ✅ Example 1: One Challenge, One Phase, One Leaderboard, One Phase Split

* Single challenge setup
* One challenge phase
* One leaderboard
* One phase split for evaluation

---

#### ✅ Example 2: One Challenge, Two Phases, One Leaderboard, Two Phase Splits

* Single challenge
* Two different phases (e.g., dev and test)
* One shared leaderboard
* Two phase splits, one for each phase

---

#### ✅ Example 3: One Challenge, Three Phases, Two Leaderboards

* Three different phases (e.g., dev, test-public, test-private)
* Two leaderboards:

  * Phase 1 (dev) uses Leaderboard A and Leaderboard B
  * Phase 2 and 3 use Leaderboard B only

---

### 🧾 Deliverables

* [ ] Add a new section in the documentation: `Challenge Configuration Examples`
* [ ] Write clear and complete `config.yaml` examples for the 3 cases above
* [ ] Add inline comments in YAML files for explanation
* [ ] Link these examples from the main challenge configuration docs
* [ ] Think of more cases which I might have missed here.
* [ ] Run these configs on local to make sure that they work fine if someone copy and paste them.
Repository: Cloud-CV/EvalAI
Issue: Remove Code Upload/Docker Image Submission from CLI Docs and Add Reference Note in Main
Root cause: Documentation Deficiency

**Description:**

The current EvalAI CLI documentation at [https://cli.eval.ai/](https://cli.eval.ai/) contains outdated instructions for **code upload** and **docker image-based submissions** (e.g., `evalai push`). These are no longer actively supported.

This issue proposes cleaning up the CLI docs and updating the main documentation accordingly — while ensuring support for large submission uploads remains intact.

---

### 🧹 Tasks

#### 1. **Update CLI Documentation**

* [ ] **Remove** all references and sections related to:

  * `evalai push`
  * Code upload
  * Docker image-based submission modes
* [ ] **Do not remove** or modify any commands related to **large file submissions** (e.g., multipart uploads or `evalai upload` with `--large` flag)
* [ ] Add a note in the CLI readme if needed to clarify the distinction

#### 2. **Update Main EvalAI Documentation**

* [ ] Under a new heading (e.g., **"Deprecated Submission Modes"**), mention that:

  * Docker/code upload submissions are deprecated
  * Users should not use the `evalai push` flow
* [ ] Link to the CLI documentation for historical reference, if necessary