Spaces:

competitions

/

wyze-rule-recommendation

like 43

Running

I think it should be related to not using the private data for submission probably. Please use the new test data on the dataset repo https://huggingface.co/datasets/wyzelabs/RuleRecommendation/tree/main, labeled as test_rule_private.csv and test_device_private.csv to generate your submission file and let us know if the problem persists.
Also tagging @abhishek to follow up on the problem.

Yeah I imagine it had to do with what you answered in the other topic. Weirded me out because I still could submit after the public lb was done, but you hadn't released the new set. Thanks

I tried again with the new data and it's just bugged I think, all of them are stuck in 'pending'.

I think the problem is the private phase limit for submission. You only have two submissions for the private round. I did reset your submissions in this round. Can you try, whenever you think your submission is ready and see if it works?

I tried again and still, stuck in pending.

Also, the wording is a bit weird. The timeline says: 'October 21, 2023 - Private leaderboard opens. Participants can make final submissions for evaluation on the full test set.' and on the 'My submissions' tab it says: 'You can select upto 2 submissions for private leaderboard.' It sounded to me like you can submit several but then select two, choosing the IDs of the submissions.

Hello, my private submission is also stuck in 'pending' status.

Also, for your information, the 'test_rule_private.csv' file has not the same columns as the public file 'test_rule_csv', which is not ideal for participants ! Missing columns are : rule, action_id, trigger_state_id.

Hi. Thanks for noticing that. It was the redundant part of the data that could be added using a map function. Anyway, I added a new file that contains those columns as well.

As for the submission page, I am waiting to hear from the HF team on how to handle it. Tagging @abhishek here to see if he can help us with this issue.

hi, whats your submission id for pending submission?

we have fixed the issue now. please re-submit the pending submissions.
apologies for the inconvenience.

Thanks @abhishek .

Please let me know if you have exhausted your quota of submission for the private phase with this issue, so I can fix that.

It is fixed but I think it just rolledback to the public test set. I uploaded a file and got 0, so I tried an earlier submission file and it worked and got an old score.

I think the private score will not be revealed to you before the deadline finishes. Also, since the private test data is different from the public data, the score you will get for that submission (which is probably the public score) would be zero. @abhishek could you please confirm this behaviour of the scores for public and private sets?

I selected 2 submissions for the private leaderboard, and none for the public one, as indicated in the submission tab. I hope there won't be any problem, and I wish all participants good luck.

See you on the 10th November !

I see. It does feel a bit of a weird way to go about it. Anyways I know this is a learning process for the competitions, but the whole different test set file (with a different format) and this last round of submissions where I haven't really been sure about what I should be doing has been confusing. I have had fun though and learned quite a bit. Cheers

Hi @mmkamani7 , @abhishek ,

I have a pending submission with id 4c1978f6-519b-419c-bd9f-09adc5f5418a,
could you please help me? I tried to concatenate results from public and private test-sets

done now.

Thanks! Is it appropriate to submit the concatenation of public and private results?

I have selected one sub with only private data, and another one with both public + private data.

I hope at least one will work !

The deadline for submission has passed, and we no longer accept any submissions. @abhishek maybe we can disable the submission part if possible to avoid confusion.
Thanks for your participation and congrats on achieving a top-three position. Please refer to this post for the final step before the announcement:
https://huggingface.co/spaces/competitions/wyze-rule-recommendation/discussions/10