zhiminy/EvalEng huggingface dataset:
Repository: Cloud-CV/EvalAI
Issue: 401 Unauthorized Error When Validating Challenge Configuration
Root cause: Validation Gap

## Issue
I encountered a 401 Unauthorized error while trying to validate a challenge configuration via the EvalAI API:

```
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://eval.ai/api/challenges/challenge/challenge_host_team/<challenge_id>/validate_challenge_config/
```
This occurred while following the challenge hosting flow, specifically at the validation step.

## Root Cause
After investigation, I found that the issue was due to an expired authentication token. The API returned a 401 error because the token used for the request was no longer valid.


## Solution
I resolved the issue by:

- Re-authenticating to generate a new auth token.
- Replacing the expired token in my API request headers.

After doing this, the challenge configuration validation worked as expected.

## TODO:
- Return a clearer error message in this case (e.g., "Token expired" instead of generic 401).

@RishabhJain2018 can I work on it?

Hey @aditi-dsi , Sure, please do it.

Hey @RishabhJain2018 raised a fix here: [https://github.com/Cloud-CV/EvalAI-Starters/pull/112](https://github.com/Cloud-CV/EvalAI-Starters/pull/112)
Repository: Cloud-CV/EvalAI
Issue: Bad Request in update_submission
Root cause: Validation Gap

I'm hosting a challenge with remote evaluation enabled.

My leaderboard has the following schema

```
leaderboard:
  - id: 1
    schema:
      {
        "labels": ["Metric1", "Metric2", "Metric3", "Metric4"],
        "default_order_by": "Metric1",
        "metadata": {
          "Metric1": {
            "sort_ascending": False,
            "description": "The Metric1 against ground truth",
          },
          "Metric2": {
            "sort_ascending": False,
            "description": "The Metric2 against ground truth",
          },
          "Metric3": {
            "sort_ascending": False,
            "description": "The Metric3 against ground truth",
          },
          "Metric4": {
            "sort_ascending": False,
            "description": "The Metric4 score of the video",
          }
        }
      }
```

I have 2 phases (`interp_test` and `vr_test`) and 2 splits (`interp_split` and `vr_split`). I plan to use the first split with the first phase and the second phase with the second split, in the final challenge.

My `evaluate()` function returns based on the following schema
```
output = {}

if phase_codename == "interp_test":
    print("Evaluating for interpolated cameras")
    output["result"] = [
        {
            "interp_split": {
                "Metric1": 10, # random value
                "Metric2": 10, # random value
                "Metric3": 10, # random value
                "Metric4": 10, # random value
            }
        }
    ]
    output["submission_result"] = output["result"][0]["interp_split"]

elif phase_codename == "vr_test":
    print("Evaluating for Test Phase")
    output["result"] = [
        {
            "vr_split": {
                "Metric1": 10, # random value
                "Metric2": 10, # random value
                "Metric3": 10, # random value
                "Metric4": 10, # random value
            }
        },
    ]
    output["submission_result"] = output["result"][0]["vr_split"]
```

In my `remote_challenge_evaluation/main.py`, I call the submission API as follows
```
update_finished(evalai, phase_pk, submission_pk, json.dumps(results["result"])
```

However, I keep getting the below error 
```
400 Client Error: Bad Request for url: https://eval.ai/api/jobs/challenge/2535/update_submission/
```

For additional reference, this is how my payload looks like
```
{'challenge_phase': 5031, 'submission': 515132, 'stdout': '', 'stderr': '', 'submission_status': 'FINISHED', 'result': '[{"interp_split": {"Metric1": 10, "Metric2": 10, "Metric3": 10, "Metric4": 10}}]', 'metadata': ''}
```

I checked the [API docs](https://eval.ai/api/docs/#tag/jobs/operation/update_submission_2) and found that results for both the splits should be contained in the request, but will that be feasible? Either way, when I use dummy values to POST for both the splits, I have the same error.
Repository: Cloud-CV/EvalAI
Issue: a custom python pip packages error
Root cause: Environment Incompatibility

Why when I execute subprocess.check_call([sys.executable, "-m", "pip", "install", "zhipuai"]) in main.py of the evaluation_script, the installed package still can't be detected? For example, from zhipuai import ZhipuAI throws an error.I also get errors when running the related code in __init__.py.
Repository: Cloud-CV/EvalAI
Issue: Updating a challenge (leaderboard, phases) after its creation
Root cause: Architectural Constraint

Hello

Thank you for the great work! I am in the process of creating a new challenge and I am using the EvalAI-Starters template. I was testing my challenge development by pushing the code onto the challenge branch. However, when I make updates to the challenge_config.yaml, I face errors which do not allow me to update the challenge on EvalAI. I did not submit a request for approval of the challenge and have kept the leaderboard and challenge private. However I still cannot pass the challenge validation step. My error log looks like this - 

ERROR: The leaderboard with ID: 2 doesn't exist. Addition of a new leaderboard after challenge creation is not allowed.
ERROR: Challenge phase 3 doesn't exist. Addition of a new challenge phase after challenge creation is not allowed.
ERROR: Challenge phase split (leaderboard_id: 1, challenge_phase_id: 3, dataset_split_id: 2) doesn't exist. Addition of challenge phase split after challenge creation is not allowed.
ERROR: Challenge phase split (leaderboard_id: 1, challenge_phase_id: 2, dataset_split_id: 2) not found in config. Deletion of existing challenge phase split after challenge creation is not allowed. 

I would appreciate any help, thank you for your consideration!
Repository: Cloud-CV/EvalAI
Issue: Remove outdated documentation for challenge creation using ZIP

**Description:**

The section in the README describing challenge creation using a ZIP file is outdated and should be removed to avoid confusion.

🔗 **Reference:**
[[Create Challenge Using Config - EvalAI-Starters](https://github.com/Cloud-CV/EvalAI-Starters?tab=readme-ov-file#create-challenge-using-config)](https://github.com/Cloud-CV/EvalAI-Starters?tab=readme-ov-file#create-challenge-using-config)

**Suggested Action:**

* Remove the section **"Create Challenge Using Config"** that refers to ZIP file uploads.
* Ensure all references to the deprecated ZIP method are cleaned from the README.

This will help keep the documentation clear and focused on the currently supported workflows.
Repository: Cloud-CV/EvalAI
Issue: Remove outdated documentation for code-upload challenges
Root cause: Documentation Deficiency

**Description:**

The documentation includes outdated instructions that should be removed to avoid confusion for challenge hosts.

**Code Upload Challenge Hosting:**

📄 Location: [EvalAI Docs - Code Upload Challenge](https://evalai.readthedocs.io/en/latest/host_challenge.html#host-code-upload-based-challenge)

🛠️ Action: Remove the section detailing how to host a code-upload-based challenge.

These features are no longer supported, and removing their documentation will help reduce confusion and streamline onboarding for new hosts.
Repository: Cloud-CV/EvalAI
Issue: Remove Code Upload Challenge Documentation from EvalAI Docs

Description:

The documentation currently includes a section on hosting code upload challenges, which is no longer supported or maintained. This section should be removed to keep the documentation accurate and up to date.

📄 Location:
[EvalAI Docs – Host Code Upload-Based Challenge](https://evalai.readthedocs.io/en/latest/host_challenge.html#host-challenge)

🛠️ Suggested Action:

Remove the section titled “Code-upload based challenges:” from the EvalAI documentation.

Double-check for any internal references or links pointing to this section and remove or update them accordingly.

Removing this outdated content will help prevent confusion for new challenge hosts and ensure the documentation reflects only the supported challenge types.

May I work on this

Hey @RishabhJain2018, I'm interested in working on this. Could you please assign it to me?
Repository: Cloud-CV/EvalAI
Issue: Remove Evaluation Script Documentation for Code Upload and Static Code Upload Challenges
Root cause: Documentation Deficiency

Description:

The documentation currently includes guidance for writing evaluation scripts for code upload and static code upload challenges, which are no longer supported. This section should be removed to reflect the current supported evaluation methods.

📄 Location:
[EvalAI Docs – Writing Code Upload Challenge Evaluation](https://evalai.readthedocs.io/en/latest/evaluation_scripts.html#writing-code-upload-challenge-evaluation)

🛠️ Suggested Action:

Remove the section titled “Writing Code Upload Challenge Evaluation” and "Writing Static Code-Upload Challenge Evaluation Script"

Review and remove any other references to static code upload or code upload evaluation scripts in the docs.

Cleaning up these obsolete instructions will improve clarity and reduce confusion for users writing evaluation scripts for supported challenge types.
Repository: Cloud-CV/EvalAI
Issue: Add Documentation for Missing Fields in Challenge and Challenge Phase Models and configuration.md (Excluding Code Upload Fields)

**Description:**

Some fields used in the Challenge model are missing from the public documentation, particularly the [`[configuration.md](https://github.com/Cloud-CV/EvalAI/blob/master/docs/configuration.md)`](https://github.com/Cloud-CV/EvalAI/blob/master/docs/configuration.md) file. These undocumented fields make it difficult for challenge hosts to fully understand and utilize the configuration options available.

📄 **Areas to Update:**

1. **Challenge Configuration Guide:**

   * 📂 File: `configuration.md`
   * 🛠️ Action: Add detailed explanations, allowed values, and example usage for all relevant fields used in challenge configuration.

2. **Challenge Model Documentation:**

   * Ensure all documented fields are up to date with the current `Challenge`, `ChallengePhase`, `Leaderboard`, and `DatasetSplit` models.

🚫 **Important Note:**
Do **not** include any fields that were previously used for **code upload** or **static code upload** challenges, as those features are no longer supported.

📌 **Examples of Missing or Under-Documented Fields (non-code-upload related):**

* `remote_evaluation`
* `submission_public`
* Others present in the current model definitions

Improving this documentation will streamline challenge onboarding, reduce confusion, and ensure consistency across hosted challenges.
Repository: Cloud-CV/EvalAI
Issue: Add Documentation on Editing Fields in Challenge Configuration on EvalAI
Root cause: Documentation Deficiency

**Description:**

There is currently no clear documentation on how to **edit or update existing fields** in the challenge configuration once a challenge has been created on EvalAI. This makes it difficult for challenge hosts to make changes or correct mistakes after deployment.

📄 **Suggested Addition:**

* A new section in the `configuration.md` or a separate guide titled **“Editing Challenge Configuration”**.
* Include instructions on:

  * Which fields can be updated after challenge creation
  * How to request updates (e.g., via the admin panel or support team)
  * How to test config changes locally before submission
  * Limitations or irreversible fields (e.g., dataset splits or phases that cannot be deleted once submissions are made)

✅ **Bonus:**
Add a checklist or common use cases, such as:

* Updating `start_date` or `end_date`
* Changing phase visibility (`is_public`)
* Modifying submission limits or evaluation script paths

📌 This documentation will help improve the experience for challenge organizers and reduce the dependency on direct support for minor edits.
Repository: Cloud-CV/EvalAI
Issue: Restructure and Prettify `configuration.md` Documentation with Sections, Examples, and Better Formatting

**Description:**

The current [[challenge configuration documentation](https://evalai.readthedocs.io/en/latest/configuration.html)](https://evalai.readthedocs.io/en/latest/configuration.html) is lengthy, difficult to navigate, and lacks clear structure and real-world examples. This can be overwhelming for new users and hard to reference for returning challenge organizers.

📄 **Proposed Improvements:**

1. **Break Into Logical Sections:**

   * Organize the content into separate, clearly labeled parts:

     * Challenge Metadata
     * Challenge Phases
     * Leaderboard Configuration
     * Dataset Splits
     * Submission Settings
     * Evaluation Configuration
     * Remote Evaluation (if applicable)

2. **Add Examples for Each Section:**

   * Include **realistic, copy-paste-ready examples** for each block of the configuration
   * Mark optional vs required fields in the examples
   * Where applicable, contrast different values with variations (e.g., `is_public: true` vs `false`)

3. **Improve Formatting:**

   * Use consistent heading levels and markdown style
   * Highlight field names, types, and default values
   * Add collapsible sections if supported by the platform
   * Fix long code blocks for readability and structure

4. **Include Table of Contents:**

   * Add a clickable ToC for navigating to specific sections quickly

🛠️ **Goal:**
Enhance usability of the `configuration.md` documentation by making it more approachable, example-driven, and scannable — helping both new and experienced challenge hosts build configs confidently and correctly.
Repository: Cloud-CV/EvalAI
Issue: Add Documentation for Building and Running Docs Locally

**Description:**

There is currently no clear guide on how to **build and view the EvalAI documentation locally**, which can make it difficult for contributors to preview changes before submitting a pull request.

📄 **Suggested Addition:**

* Add a new section in the main `README.md` or in a dedicated `docs/README.md` file with the following details:

### ✅ Steps to Include:

1. **Install Requirements:**

   ```bash
   pip install -r requirements/readthedocs.txt
   ```

2. **Build Docs:**

   ```bash
   cd docs
   make html
   ```

3. **View Locally:**
   Open `docs/_build/html/index.html` in your browser.

4. **Live Preview:**
   Add instructions for live preview using `sphinx-autobuild`:

   ```bash
   pip install sphinx-autobuild
   sphinx-autobuild docs docs/_build/html
   ```

5. **Troubleshooting Tips:**

   * Mention common issues like missing packages or build errors.
   * Include supported Python/Sphinx versions if relevant.

📌 **Goal:**
Enable contributors to preview and test documentation changes locally, improving the contribution experience and reducing documentation-related errors in PRs.
Repository: Cloud-CV/EvalAI
Issue: 🧹 Clean Up and Update Developer FAQ Documentation

**Description:**

The current [[Developer FAQ](https://github.com/Cloud-CV/EvalAI/blob/master/docs/source/faq(developers).md)](https://github.com/Cloud-CV/EvalAI/blob/master/docs/source/faq%28developers%29.md) document is outdated and requires cleanup and restructuring for clarity and usefulness.

---

## Tasks

### ✅ Cleanup Goals:

* Fix formatting issues (e.g., broken markdown, inconsistent headers)
* Remove outdated or irrelevant questions
* Merge or rephrase redundant entries
* Standardize tone and writing style

### 🧩 Suggested Improvements:

* Group related questions under sections like:

  * Setup & Installation
  * Development & Contribution
  * Docker & Workers
  * Debugging & Logs
* Add internal links for easier navigation (e.g., using `#anchor-links`)
* Add links to relevant updated documentation sections where appropriate
* Remove any content already duplicated in newer guides or tutorials

### 🔍 Optional Enhancements:

* Add a table of contents at the top for better UX
* Include FAQs for commonly asked GitHub issues (e.g., Redis errors, Celery issues)
* Ensure all commands use code formatting for consistency

---

## Related Files:

* [`[docs/source/faq(developers).md](https://github.com/Cloud-CV/EvalAI/blob/master/docs/source/faq(developers).md)`](https://github.com/Cloud-CV/EvalAI/blob/master/docs/source/faq%28developers%29.md)

---

Let’s make the FAQ genuinely helpful for new contributors and developers working with EvalAI!

Hi @RishabhJain2018 👋,

Thank you for outlining the improvements so clearly! I’ve gone through the current faq(developers).md and would love to take this up. I plan to:
	•	Fix all formatting inconsistencies and broken markdown
	•	Organize FAQs under structured sections (Setup, Docker, Debugging, etc.)
	•	Remove outdated/duplicated content and standardize tone
	•	Add internal anchor links and a table of contents for better navigation

Let me know if you’d like me to follow any specific style guide or refer to any sample documentation for tone and structure. Looking forward to contributing!

Best,
@vtushar06
Repository: Cloud-CV/EvalAI
Issue: 📚 Add Documentation for Installing Custom Dependencies via init.py in EvalAI-Starters
Root cause: Documentation Deficiency

**Description:**

Installing custom dependencies using an `init.py` script in the `evaluation_script/` directory within EvalAI challenge bundles. We should add proper documentation to the **EvalAI-Starters** repository to guide challenge hosts on how to use this feature effectively.

---

## 📂 Target Location for Documentation

* Repository: [[Cloud-CV/EvalAI](https://github.com/Cloud-CV/EvalAI)](https://github.com/Cloud-CV/EvalAI-Starters)
 *Add a new markdown file: `dependency-installation.md`

---

## 📝 Documentation Contents

### 🔹 Purpose

Explain why a custom `init.py` is useful—for installing packages, preparing the environment, or downloading required files before evaluation begins.

### 🔹 Example Structure

```plaintext
evaluation_script/
├── init.py
├── main.py
```

### 🔹 Sample `init.py`

```python
import subprocess

# Install required pip packages
subprocess.check_call(["pip", "install", "nltk==3.8.1"])

# Download NLTK data
import nltk
nltk.download("punkt")
```

### 🔹 Execution Behavior

* `init.py` is run **before** `main.py` during submission evaluation.
* Any stdout/stderr output will be visible in the submission logs.


---

## ✅ Task

* [ ] Add the new section or file to document the use of `init.py`.

---

This will greatly help challenge organizers using EvalAI to customize their evaluation environment without modifying the base worker image.
Repository: Cloud-CV/EvalAI
Issue: ✅ Add test coverage for seed.py management command

Here’s a GitHub issue you can use to track adding test cases for the seed script:

**Description:**

The `apps/base/management/commands/seed.py` script currently has **0% test coverage**. This script defines a custom Django management command `seed` that delegates execution to the `runscript` command with optional arguments.

We need to add unit tests to ensure proper functionality and prevent regressions.

---

**📂 File:**
`apps/base/management/commands/seed.py`

---

**🔧 Functions to Cover:**

1. `add_arguments`

   * **Approach:**
     Mock `argparse.ArgumentParser` and assert that `add_argument("-nc", ...)` is called correctly.

2. `handle`

   * **Approach:**
     Use `unittest.mock.patch` to mock `call_command`.
     Test both default behavior (no `-nc`) and custom values (e.g., `--nc 5`) to ensure `call_command("runscript", "seed", "--script-args", <nc>)` is called as expected.

---

**✅ Acceptance Criteria:**

* [ ] Test that `-nc` argument is registered in `add_arguments`
* [ ] Test default behavior of `handle` when `-nc` is not provided
* [ ] Test custom behavior of `handle` when `-nc` is specified
* [ ] Achieve > 95% coverage for `seed.py` via `coverage run` and `coverage report`

---

**📌 Notes:**

* Tests should go in a corresponding test file (e.g., `tests/test_seed_command.py` or similar).
* Use Django's `override_settings` and `call_command` utilities as needed.
* Add the test module to the test discovery if not already picked up.
Repository: Cloud-CV/EvalAI
Issue: 🧩 Resolve conflicts in #4403 and open a fresh PR after testing
Root cause: Others

**Description:**

The PR [[#4403](https://github.com/Cloud-CV/EvalAI/pull/4403)](https://github.com/Cloud-CV/EvalAI/pull/4403) adds the tests for the base app, but it currently has **merge conflicts** and is outdated with the `master` branch.

---

**🛠️ Tasks:**

* [ ] Pull latest changes from `master`
* [ ] Resolve merge conflicts in the branch from #4403
* [ ] Verify that all tests pass locally
* [ ] Link the new PR in a comment on #4403 and close the old PR if everything is carried over
Repository: Cloud-CV/EvalAI
Issue: Approval Request for RoboSense Challenge 2025 - Track #2: Social Navigation (ID: 2557)

Hi EvalAI Team,

We have submitted an approval request for our challenge via the EvalAI dashboard. Could you please help review and approve it?

**Challenge Information:**

- **Competition Name:** The RoboSense Challenge 2025 – Track # 2: Social Navigation  
- **Challenge Title on EvalAI:** RoboSense Social Navigation Challenge  
- **Challenge ID:** 2557  
- **EvalAI Page:** https://eval.ai/web/challenges/challenge-page/2557/overview  
- **External Website:** https://robosense2025.github.io  
- **Expected Start Date:** June 15, 2025

@RishabhJain2018 We would really appreciate your help reviewing and approving this challenge. Please let us know if any changes or additional steps are required. Thank you for your support!



Hi @Zeying-Gong , Thank you for reaching out to us and it should be approved now!

Thank you so much for the prompt response — really appreciate your support! I’ll go ahead and close this issue.
Repository: Cloud-CV/EvalAI
Issue: argoverse2 challenge click problem
Root cause: Environment Incompatibility

I can not select phase because the button could not be selected.Maybe it is because the competition is over.But i just want to test the performance of my model.

<img width="533" alt="Image" src="https://github.com/user-attachments/assets/83afb265-5174-47c5-a6f8-b26f0f87381a" />
Repository: Cloud-CV/EvalAI
Issue: Platform selection
Root cause: Documentation Deficiency

Can we use the Harbor warehouse on the intranet for remote evaluation in the form of local deployment, or must we use the AWS platform

<html>
<body>
<!--StartFragment--><html><head></head><body><p>Here is the <strong>clear, direct answer</strong>:</p>
<h3>✅ <strong>Yes — you <em>can</em> use a local Harbor warehouse on the intranet for remote evaluation</strong>,</h3>
<p><strong>BUT only if the evaluation rules explicitly allow on-premise / intranet deployment.</strong></p>
<h3>❗ <strong>If the competition/evaluation guidelines say the deployment must be done on AWS</strong>,</h3>
<p>then you <strong>must</strong> use the AWS platform — local Harbor deployment will <strong>not</strong> be accepted.</p>
<hr>
<h1>🔍 <strong>General Rule</strong></h1>

Deployment Option | Allowed?
-- | --
Local / Intranet Harbor registry | ✔ Allowed only if remote evaluators can access it and guidelines allow it
AWS platform deployment | ✔ Required if mentioned by the organizers


<hr>
<h1>🔑 <strong>Important Points</strong></h1>
<ul>
<li>
<p>If the evaluation team needs <strong>remote access</strong>, your local Harbor registry must be <strong>publicly reachable</strong> (not blocked by firewall).</p>
</li>
<li>
<p>Many hackathons or competitions <strong>mandate AWS</strong>, so intranet deployment is usually <strong>not acceptable</strong> unless stated.</p>
</li>
<li>
<p>Always check:</p>
<ul>
<li>
<p>platform requirements</p>
</li>
<li>
<p>deployment instructions</p>
</li>
<li>
<p>access rules</p>
</li>
</ul>
</li>
</ul>
<hr>

</body>
</html>