Repository: Cloud-CV/EvalAI
Issue: Dataset Health & Drift Analysis Tool for EvalAI / CloudCV
Root cause: Unimplemented Feature Gap

Hello EvalAI / CloudCV maintainers,

My name is Jobin Sabu. While exploring EvalAI workflows, I noticed that dataset quality checks (duplicates, corruption, imbalance, drift) are largely handled manually before training or evaluation.

I’m currently building a standalone tool tentatively called Dataset Drifter, focused on dataset health and drift analysis for computer vision datasets. The goal is to help users identify dataset issues early, before model training or challenge submissions.

Initial scope (MVP)

Detect duplicate and near-duplicate images

Identify corrupted or unreadable files

Analyze class distribution and imbalance

Flag outliers using embedding-based similarity

Generate a concise dataset health report (CLI + basic visual summary)


Design goals

Modular, extensible architecture

Works with common CV dataset formats (image folders + annotations)

Usable as a CLI or Python library

Potentially integrable with EvalAI / CloudCV workflows in the future


I’m completing a working MVP first and wanted to check:

whether this direction aligns with EvalAI / CloudCV goals, and

if there are any existing tools, preferred formats, or constraints I should align with early.


Once the MVP is ready, I’d be happy to share the repository and discuss possible integration or next steps.

Thank you for maintaining EvalAI and CloudCV — it’s a very valuable platform for the community.

Best regards,
Jobin Sabu

This is an interesting feature direction.
If your standalone Dataset Drifter project is public, I’d love to explore contributing.
In parallel, I can draft a small prototype for how EvalAI might integrate dataset-health checks (API, worker tasks, or report UI).
Let me know what direction would be most useful.

will you  please assigm me this issue


This isn’t a tracked issue yet — it’s currently a proposed tool / direction I’m exploring. I’ll create a formal issue once the scope is clearer and will update there

ok 



This sounds like a very relevant problem to address 

From an EvalAI / CloudCV workflow point of view, early dataset validation could definitely reduce noisy submissions and failed evaluations.

A few thoughts / questions that might be worth considering early:
- For duplicate / near-duplicate detection, are you planning classical perceptual hashing (pHash, aHash) or embedding-based similarity?
- Do you plan to support common formats used in EvalAI challenges (e.g., COCO-style annotations, image-folder + CSV/JSON labels)?
- Would the initial integration target be pre-submission validation (user-side CLI) or server-side checks within EvalAI pipelines?

Also, some overlap might exist with tools like cleanlab or FiftyOne — it may help to clarify how Dataset Drifter differentiates or complements them.

Happy to follow along and test the MVP once it's public.


Hi 

This is a really interesting proposal — dataset health and drift analysis is definitely a missing piece in many ML workflows.

I’d love to contribute to this if it’s open for collaboration.

I have experience working with:
- Python-based data analysis pipelines
- Computer vision datasets (image validation, basic stats, class distribution)
- Visualization dashboards for dataset insights

If helpful, I can start with a small prototype (for example, corrupted file detection or class imbalance analysis) and iterate based on feedback.

Please let me know if this aligns with the project direction or if there’s a preferred way to contribute here.  
Happy to follow guidance from the maintainers 

Thanks!