by OE-Heart - opened

Hi, I've got some questions on the submmision.
I followed the steps of raft-submission and submited my result called 'yiwise' last week, and it said that the evaluation would be run on June 05. But I haven't seen my result on the leaderboard until now(June 06).
Where did I go wrong?

Hi @OE-Heart! Thanks for raising the issue - there's been some changes in our backend that affected the evaluation pipeline. I'll post a fix this week and report back here when it's done. Apologies for the inconvenience!

Hi @OE-Heart, we're just deployed the fix for the evaluation pipeline and your submission has now been evaluated. Apologies for the delay!

lewtun changed discussion status to closed
OE-Heart
edited Jun 13

Hi @lewtun! Thanks for the fix and I have another submission on Jun 11, will it be evaluated on Jun 12 as expected?

Hi @OE-Heart, looking at the logs, it seems there's a problem with one of the classes in the banking77 subset, specifically about the reverted_card_payment? class. Could you please check that your submission passes the validation test with:

python cli.py validate


Thanks!

lewtun changed discussion status to open
OE-Heart
edited Jun 13

@lewtun Thanks for the reminder, you are right. I made a little mistake on the label. May I have another evaluation please? It's important for me. Appreciate.

Just in case someone else encounters this problem in the future, pay attention to one particular label name is reverted_card_payment? but not reverted_card_payment. The question mark at the end is not an error.

@lewtun Thanks for the reminder, you are right. I made a little mistake on the label. May I have another evaluation please? It's important for me. Appreciate.

Sure! I'll run the evaluation pipeline ASAP

Hey @OE-Heart, I've run the evaluation pipeline and there seems to be a problem with the ID column in one of your datasets. Could you confirm whether the validation script passes for your submission?

python cli.py validate

TomyAndi
edited Jun 14

-

Thanks for checking. Last question: can you also make a new submission by running

python cli.py submit


Hey @OE-Heart, I've run the evaluation pipeline and there seems to be a problem with the ID column in one of your datasets. Could you confirm whether the validation script passes for your submission?

python cli.py validate


I just checked again and it looks ok for me.

Thanks for checking. Last question: can you also make a new submission by running

python cli.py submit


I just submitted.

Hi, @lewtun .Is there anything else I can do, please? Sorry for the trouble.

Looks like it worked! Great job on beating the human baseline :D

lewtun changed discussion status to closed
This comment has been hidden