Failing submissions
I've been experimenting with various setups. What I have observed is the submission keeps failing for certain csvs. I have tried identifying for any obvious errors like lack of rows, incorrect format etc., but can't seem to find any. What is the criteria for failing a submission?
Hard to say without seeing the actual example. Perhaps you could share the one that is failing and we could try to debug.
Overall it must be a well-formatted csv similar to example submission that provides some sort of a score for each pair in test set.
Also, keep in mind that the number of submissions per day is limited to 3. It should be enough as the public leaderboard is only a small subset of the full test set - we encourage participants to tune on our suggested dev set or, even better, construct a cross-validation setup from train+dev.