Repository: Cloud-CV/EvalAI
Issue: Errors when submitting .csv and .json files
Root cause: Validation Gap

- the expected behaviour: 

successful submission

- the observed behaviour: 

Failed submission with stderror file output:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type json
LINE 1: ...5.338415+00:00'::timestamptz, 4585, 253967, 3086, '{"rating_...
                                                             ^
DETAIL:  Token "NaN" is invalid.
CONTEXT:  JSON data, line 1: ...14, "rating_identifies_issues_opportunities": NaN...

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 588, in run_submission
    LeaderboardData.objects.bulk_create(leaderboard_data_list)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 474, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1204, in _batched_insert
    ignore_conflicts=ignore_conflicts,
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1186, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/raven/contrib/django/client.py", line 127, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DataError: invalid input syntax for type json
LINE 1: ...5.338415+00:00'::timestamptz, 4585, 253967, 3086, '{"rating_...
                                                             ^
DETAIL:  Token "NaN" is invalid.
CONTEXT:  JSON data, line 1: ...14, "rating_identifies_issues_opportunities": NaN...

- steps to reproduce the problem:

Below are the first few lines of the "submitted file" (.csv upload) output (there are no missing values):

response_id,rating_chooses_appropriate_action,rating_commits_to_action,rating_gathers_information,rating_identifies_issues_opportunities,rating_interprets_information,rating_involves_others,rating_decision_making_final_score
XSEKmbXmmZi,2.41272,2.73903,2.46766,2.59631,2.2085,2.61134,1.68563
OTigqBfxROr,2.60589,3.16797,2.55415,2.44868,2.29867,2.71851,2.69446
lMGidYKEOBv,2.98529,3.37086,2.96131,2.73464,2.4911,2.89826,3.83156
lEmnRmxniLl,2.97665,3.44252,2.90891,2.62953,2.60479,2.8648,4.38432
OhnFgbmEVqG,2.8607,3.27591,2.76341,2.67578,2.6084,2.91514,3.72941

- operating system: 

macOS Ventura 13.2.1 m2 processor

- browser: 

have tried Safari and google chrome

- other:

I have also tried to submit a .json file from the CLI with no luck. In that case, the output of the stderror file was: 

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2895, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'response_id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 500, in run_submission
    submission_metadata=submission_serializer.data,
  File "/tmp/tmphw10j9bw/compute/challenge_data/challenge_1937/main.py", line 26, in evaluate
    missing_ids = set(test_annotation_file['response_id']).difference(user_submission_file['response_id'])
  File "/usr/local/lib/python3.7/site-packages/pandas/core/frame.py", line 2906, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/local/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    raise KeyError(key) from err
KeyError: 'response_id'


The host wanted integers, not floats. Closing this issue as my errors have been resolved.