Spaces:
Running
on
CPU Upgrade
Your submission failed for unknown reason? Post here
We will do our best to provide you logs of the evaluation (without the part possibly revealing the test set), e.g. package missing or OOM errors.
NOTE
Before Posting:
Please make sure to run your solution locally (python script.py). By default it will run your code and compute scores on the validation set. Ifscript.pyruns locally without error, but your submission still fails on the test server please post here.
Why is the space paused after my submission?
@Siromanec
we are updating all previous leaderboard now with a fixed a re-evaluated metrics, you just got unlucky.
The space is expected to go back online in ~1 hour.
Hello, could you please mail me the logs on runs 04d5854d-434a-4cc7-9871-b0971d6fc514, 1ffda7b8-858d-4418-8208-d1733a416fe8, 65758f07-8643-46ce-9a0d-8da12934b4ab? My email: serhii.ivanov@ucu.edu.ua
@Siromanec of course, not. The logs reveal too much information and could potentially leak the info about the test set.
However, I can post here some error traces, which hopefully help with debugging.
It seems that there is something wrong with packages on your 04d5854d submission
2024-05-26 20:00:54.203 | INFO | __main__:generate_submission_file:41 - Generating submission file
Traceback (most recent call last):
File "/tmp/model/script.py", line 34, in <module>
import hoho; hoho.setup() # YOU MUST CALL hoho.setup() BEFORE ANYTHING ELSE
ModuleNotFoundError: No module named 'hoho'
1ffda7 finished succesfully.
65758f07 - same as 04d5854d submission
hoho module now is not preinstalled? It is extremely odd that this error did not occur three times in a row. Especially when considering the fact that I did not tweak any of the imports, when comparing to the baseline (or between the three runs).
And now the run identical to 65758f07 is a success?
@Siromanec
the hoho module is preinstalled. We do not have a control over the huggingface infrastructure, so some weird stuff is possible. It is also possible, that the 65758f07 has been submitted at the times of the Space restart or something like that. We apologize for possible inconveniences, but, as I said, we do not have control over them.
Any information on 396768ac-d2af-4e66-bf80-c2f4860f0953?
Thanks in advance
d6894d8d-e694-4896-a93d-7bbce7a5eb3b
d6894d8d-e694-4896-a93d-7bbce7a5eb3b
Traceback (most recent call last):
File "/tmp/model/script.py", line 252, in <module>
pred_vertices, pred_edges = solution_hc(sample)
File "/tmp/model/script.py", line 222, in solution_hc
triangulated_corners, triangulated_corner_classes = get_triangulated_corners(gestalt_segmented_images, depth_images, Ks, Rs, ts)
File "/tmp/model/script.py", line 64, in get_triangulated_corners
vertices = get_vertices_from_gestalt(gest_seg_np)
File "/tmp/model/utils.py", line 22, in get_vertices_from_gestalt
apex_mask = cv2.inRange(gest_seg_np, apex_color-color_threshold/2, apex_color+color_threshold/2)
cv2.error: OpenCV(4.9.0) :-1: error: (-5:Bad argument) in function 'inRange'
> Overload resolution failed:
> - src data type = |S41430 is not supported
> - Expected Ptr<cv::UMat> for argument 'src'
Hi again! Sorry for the trouble but I guess there is no other way of getting to know what's wrong. Seems to work locally.
Info on 6af34f92-4805-4226-8b00-b2aa71e6b737 please!
It looks like all finished well, but the submission.parquet was not generated
2024-05-29 11:44:19.522 | ERROR | competitions.utils:wrapper:162 - run has failed due to an exception: Traceback (most recent call last):
File "/app/competitions/utils.py", line 159, in wrapper
return func(*args, **kwargs)
File "/app/competitions/evaluate.py", line 122, in run
generate_submission_file(params)
File "/app/competitions/evaluate.py", line 83, in generate_submission_file
api.upload_file(
File "/app/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/app/env/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1208, in _inner
return fn(self, *args, **kwargs)
File "/app/env/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 4243, in upload_file
operation = CommitOperationAdd(
File "<string>", line 5, in __init__
File "/app/env/lib/python3.10/site-packages/huggingface_hub/_commit_api.py", line 164, in __post_init__
raise ValueError(f"Provided path: '{path_or_fileobj}' is not a file on the local file system")
ValueError: Provided path: '/tmp/model/submission.parquet' is not a file on the local file system
Any idea why "/tmp/model/submission.parquet" does not exist? Using this part from the sample submission.
May I know the suggested way to install a python package? I guess my recent submissions failed due to missing packages. Thanks.
@kcml
it is described in the commented out sections in handcrafted solution example.
https://huggingface.co/usm3d/handcrafted_baseline_submission/blob/main/script.py#L41
In short - you add wheels to your repo, and install from them using example in the script.
You also could request some packages here https://huggingface.co/spaces/usm3d/S23DR/discussions/1 and we will consider adding them to the space ourselves.
@kc92 unfortunately, I cannot help with debugging this. I guess, that the script finished prematurely
@dmytromishkin Added code to install local wheels, but no luck. Would you mind help check the submission ID: e788dbe3-f7ac-4486-b79d-888c6e94ca76
It seems you forgot to do from pathlib import Path
Would mind to check locally first, please?
pth = str(Path(folder) / package_name)
NameError: name 'Path' is not defined. Did you mean: 'pth'?
could you please check 1c5655cb-d7ef-4de2-88c0-349abb9f51e0 ?
Thanks!
from handcrafted_solution import predict
^
IndentationError: unindent does not match any outer indentation level
could you please check this c403c2fa-d16e-4c02-8d64-8965ad8fe3d9 ?
Thanks!
c403c2fa-d16e-4c02-8d64-8965ad8fe3d9
It looks like you are trying to install hoho tools. You don't need to, they are already installed on the test server
Could u check this?
70408be2-e21f-4dda-bf7f-dbe0819a853b
70408be2-e21f-4dda-bf7f-dbe0819a853b
looks like you are trying to install packages by downloading them with pip, but the test server can't connect to the internet. Happy to keep grabbing logs, but could you try disconnecting your computer from the internet and running python script.py? If that's working, but test server fails it is much more likely something on our end :-)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fd45d8665c0>: Failed to establish a new connection: [Errno 100] Network is down')':
...
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fd45d867b50>: Failed to establish a new connection: [Errno 100] Network is down')': /simple/torch/
ERROR: Could not find a version that satisfies the requirement torch==1.13.1 (from versions: none)
ERROR: No matching distribution found for torch==1.13.1
Failed to download packages. Error: Command '['/app/env/bin/python', '-m', 'pip', 'download', '--platform', 'manylinux1_x86_64', '--python-version', '38', '--only-binary=:all:', '-d', 'packages/torch', 'torch==1.13.1', 'torchvision==0.14.1', 'torchaudio==0.13.1']' returned non-zero exit status 1.
Looking in links: packages/torch/torch
Requirement already satisfied: torch in /app/env/lib/python3.10/site-packages (2.3.0)
Requirement already satisfied: filelock in /app/env/lib/python3.10/site-packages (from torch) (3.13.1)
...
Requirement already satisfied: pure-eval in /app/env/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.2)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by
...
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda1e8259c0>: Failed to establish a new connection: [Errno 100] Network is down')': /simple/easydict/
ERROR: Could not find a version that satisfies the requirement easydict (from versions: none)
ERROR: No matching distribution found for easydict
Traceback (most recent call last):
File "/tmp/model/script.py", line 184, in <module>
setup_environment()
File "/tmp/model/script.py", line 160, in setup_environment
subprocess.check_call([sys.executable, "-m", "pip", "install", "easydict"])
File "/app/env/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/app/env/bin/python', '-m', 'pip', 'install', 'easydict']' returned non-zero exit status 1.
thanks for your help, could you please check this?
28a66061-d8be-49db-8fb5-69475b08bc11
and this
3841b447-000f-443b-bef2-aed67b00476f
28a66061-d8be-49db-8fb5-69475b08bc11
Requirement already satisfied: urllib3<3,>=1.21.1 in /app/env/lib/python3.10/site-packages (from requests->dash>=2.6.0->open3d) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /app/env/lib/python3.10/site-packages (from requests->dash>=2.6.0->open3d) (2024.2.2)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /app/env/lib/python3.10/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.8.4)
Requirement already satisfied: ptyprocess>=0.5 in /app/env/lib/python3.10/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.7.0)
Requirement already satisfied: wcwidth in /app/env/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.13)
Requirement already satisfied: executing>=1.2.0 in /app/env/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in /app/env/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (2.4.1)
Requirement already satisfied: pure-eval in /app/env/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.2)
Package installed successfully from packages/open3d/open3d
Looking in links: packages/easydict/easydict
WARNING: Location 'packages/easydict/easydict' is ignored: it is either a non-existing path or lacks a specific scheme.
ERROR: Could not find a version that satisfies the requirement easydict (from versions: none)
ERROR: No matching distribution found for easydict
Failed to install package from packages/easydict/easydict. Error: Command '['/app/env/bin/python', '-m', 'pip', 'install', '--no-index', '--find-links', 'packages/easydict/easydict', 'easydict']' returned non-zero exit status 1.
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Traceback (most recent call last):
File "/tmp/model/script.py", line 184, in <module>
setup_environment()
File "/tmp/model/script.py", line 166, in setup_environment
subprocess.check_call([sys.executable, "setup.py", "install"])
File "/app/env/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/app/env/bin/python', 'setup.py', 'install']' returned non-zero exit status 1.
2024-06-03 22:23:03.549 | ERROR | __main__:generate_submission_file:72 - Subprocess didn't terminate successfully
2024-06-03 22:23:03.549 | INFO | __main__:generate_submission_file:76 - contents of submission_dir
2024-06-03 22:23:03.549 | INFO | __main__:generate_submission_file:77 - ['params.json', '.gitattributes', '.gitignore', 'README.md', 'handcrafted_solution.py', 'packages', 'pc_util', 'script.py', 'socket-kit.so', 'data']
3841b447-000f-443b-bef2-aed67b00476f
Requirement already satisfied: pure-eval in /app/env/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=8.0.4->open3d) (0.2.2)
Package installed successfully from packages/open3d
Looking in links: packages/easydict
Processing ./packages/easydict/easydict-1.13-py3-none-any.whl
Installing collected packages: easydict
Successfully installed easydict-1.13
Package installed successfully from packages/easydict
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.
Traceback (most recent call last):
File "/tmp/model/script.py", line 184, in <module>
setup_environment()
File "/tmp/model/script.py", line 166, in setup_environment
subprocess.check_call([sys.executable, "setup.py", "install"])
File "/app/env/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/app/env/bin/python', 'setup.py', 'install']' returned non-zero exit status 1.
2024-06-03 22:42:56.377 | ERROR | __main__:generate_submission_file:72 - Subprocess didn't terminate successfully
could u please check 57421595-8b05-4cfa-8aa6-ceb3d8551dd6?
running install
/app/env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
/app/env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
running bdist_egg
running egg_info
writing pc_util.egg-info/PKG-INFO
writing dependency_links to pc_util.egg-info/dependency_links.txt
writing top-level names to pc_util.egg-info/top_level.txt
/app/env/lib/python3.10/site-packages/torch/utils/cpp_extension.py:499: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'pc_util.egg-info/SOURCES.txt'
writing manifest file 'pc_util.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
/app/env/lib/python3.10/site-packages/torch/utils/cpp_extension.py:428: UserWarning: There are no g++ version bounds defined for CUDA version 12.1
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'pc_util' extension
creating build
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/src
gcc -pthread -B /app/env/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /app/env/include -fPIC -O2 -isystem /app/env/include -fPIC -I/app/env/lib/python3.10/site-packages/torch/include -I/app/env/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/app/env/lib/python3.10/site-packages/torch/include/TH -I/app/env/lib/python3.10/site-packages/torch/include/THC -I/app/env/include -I/app/env/include/python3.10 -c src/ball_query.cpp -o build/temp.linux-x86_64-cpython-310/src/ball_query.o -g -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=pc_util -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
src/ball_query.cpp:4:10: fatal error: cuda.h: No such file or directory
4 | #include <cuda.h>
| ^~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
Traceback (most recent call last):
File "/tmp/model/script.py", line 188, in <module>
setup_environment()
File "/tmp/model/script.py", line 170, in setup_environment
subprocess.check_call([sys.executable, "setup.py", "install"])
File "/app/env/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/app/env/bin/python', 'setup.py', 'install']' returned non-zero exit status 1.
57421595-8b05-4cfa-8aa6-ceb3d8551dd6
running install
/app/env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
/app/env/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************
!!
self.initialize_options()
running bdist_egg
running egg_info
writing pc_util.egg-info/PKG-INFO
writing dependency_links to pc_util.egg-info/dependency_links.txt
writing top-level names to pc_util.egg-info/top_level.txt
/app/env/lib/python3.10/site-packages/torch/utils/cpp_extension.py:499: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'pc_util.egg-info/SOURCES.txt'
writing manifest file 'pc_util.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
/app/env/lib/python3.10/site-packages/torch/utils/cpp_extension.py:428: UserWarning: There are no g++ version bounds defined for CUDA version 12.1
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'pc_util' extension
creating build
creating build/temp.linux-x86_64-cpython-310
creating build/temp.linux-x86_64-cpython-310/src
gcc -pthread -B /app/env/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /app/env/include -fPIC -O2 -isystem /app/env/include -fPIC -I/app/env/lib/python3.10/site-packages/torch/include -I/app/env/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/app/env/lib/python3.10/site-packages/torch/include/TH -I/app/env/lib/python3.10/site-packages/torch/include/THC -I/app/env/include -I/app/env/include/python3.10 -c src/ball_query.cpp -o build/temp.linux-x86_64-cpython-310/src/ball_query.o -g -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=pc_util -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
src/ball_query.cpp:4:10: fatal error: cuda.h: No such file or directory
4 | #include <cuda.h>
| ^~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
Traceback (most recent call last):
File "/tmp/model/script.py", line 188, in <module>
setup_environment()
File "/tmp/model/script.py", line 170, in setup_environment
subprocess.check_call([sys.executable, "setup.py", "install"])
File "/app/env/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/app/env/bin/python', 'setup.py', 'install']' returned non-zero exit status 1.
2024-06-06 14:42:16.150 | ERROR | __main__:generate_submission_file:72 - Subprocess didn't terminate successfully
2024-06-06 14:42:16.150 | INFO | __main__:generate_submission_file:76 - contents of submission_dir
2024-06-06 14:42:16.150 | INFO | __main__:generate_submission_file:77 - ['params.json', '.gitattributes', '.gitignore', 'README.md', 'handcrafted_solution.py', 'packages', 'pc_util', 'script.py', 'socket-kit.so', 'data']
2024-06-06 14:42:16.150 | INFO | __main__:generate_submission_file:81 - Uploading submission.parquet to the repository
2024-06-06 14:42:16.152 | ERROR | competitions.utils:wrapper:162 - run has failed due to an exception: Traceback (most recent call last):
If you want you can also post a list of packages you want in https://huggingface.co/spaces/usm3d/S23DR/discussions/1 and we can add them to the shared requirements.txt.