Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
@@ -1,128 +1,6 @@
|
|
1 |
---
|
2 |
-
title:
|
3 |
app_file: app.py
|
4 |
sdk: gradio
|
5 |
sdk_version: 4.42.0
|
6 |
---
|
7 |
-
# 3D Reconstruction with Spatial Memory
|
8 |
-
|
9 |
-
### [Paper](https://arxiv.org/abs/2408.16061) | [Project Page](https://hengyiwang.github.io/projects/spanner) | [Video](https://hengyiwang.github.io/projects/spanner/videos/spanner_intro.mp4)
|
10 |
-
|
11 |
-
> 3D Reconstruction with Spatial Memory <br />
|
12 |
-
> [Hengyi Wang](https://hengyiwang.github.io/), [Lourdes Agapito](http://www0.cs.ucl.ac.uk/staff/L.Agapito/)<br />
|
13 |
-
> arXiv 2024
|
14 |
-
|
15 |
-
<p align="center">
|
16 |
-
<a href="">
|
17 |
-
<img src="./assets/spann3r_teaser_white.gif" alt="Logo" width="90%">
|
18 |
-
</a>
|
19 |
-
</p>
|
20 |
-
|
21 |
-
## Installation
|
22 |
-
|
23 |
-
1. Clone Spann3R
|
24 |
-
|
25 |
-
```
|
26 |
-
git clone https://github.com/HengyiWang/spann3r.git
|
27 |
-
cd spann3r
|
28 |
-
```
|
29 |
-
|
30 |
-
2. Create conda environment
|
31 |
-
|
32 |
-
```
|
33 |
-
conda create -n spann3r python=3.9 cmake=3.14.0
|
34 |
-
conda install pytorch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 pytorch-cuda=11.8 -c pytorch -c nvidia # use the correct version of cuda for your system
|
35 |
-
|
36 |
-
pip install -r requirements.txt
|
37 |
-
|
38 |
-
# Open3D has a bug from 0.16.0, please use dev version
|
39 |
-
pip install -U -f https://www.open3d.org/docs/latest/getting_started.html open3d
|
40 |
-
```
|
41 |
-
|
42 |
-
3. Compile cuda kernels for RoPE
|
43 |
-
|
44 |
-
```
|
45 |
-
cd croco/models/curope/
|
46 |
-
python setup.py build_ext --inplace
|
47 |
-
cd ../../../
|
48 |
-
```
|
49 |
-
|
50 |
-
4. Download the DUSt3R checkpoint
|
51 |
-
|
52 |
-
```
|
53 |
-
mkdir checkpoints
|
54 |
-
cd checkpoints
|
55 |
-
# Download DUSt3R checkpoints
|
56 |
-
wget https://download.europe.naverlabs.com/ComputerVision/DUSt3R/DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth
|
57 |
-
```
|
58 |
-
|
59 |
-
5. Download our [checkpoint](https://drive.google.com/drive/folders/1bqtcVf8lK4VC8LgG-SIGRBECcrFqM7Wy?usp=sharing) and place it under `./checkpoints`
|
60 |
-
|
61 |
-
## Demo
|
62 |
-
|
63 |
-
1. Download the [example data](https://drive.google.com/drive/folders/1bqtcVf8lK4VC8LgG-SIGRBECcrFqM7Wy?usp=sharing) (2 scenes from [map-free-reloc](https://github.com/nianticlabs/map-free-reloc)) and unzip it as `./examples`
|
64 |
-
|
65 |
-
2. Run demo:
|
66 |
-
|
67 |
-
```
|
68 |
-
python demo.py --demo_path ./examples/s00567 --kf_every 10 --vis
|
69 |
-
```
|
70 |
-
|
71 |
-
For visualization `--vis`, it will give you a window to adjust the rendering view. Once you find the view to render, please click `space key` and close the window. The code will then do the rendering of the incremental reconstruction.
|
72 |
-
|
73 |
-
|
74 |
-
|
75 |
-
## Training and Evaluation
|
76 |
-
|
77 |
-
### Datasets
|
78 |
-
|
79 |
-
We use Habitat, ScanNet++, ScanNet, ArkitScenes, Co3D, and BlendedMVS to train our model. Please refer to [data_preprocess.md](docs/data_preprocess.md).
|
80 |
-
|
81 |
-
### Train
|
82 |
-
|
83 |
-
Please use the following command to train our model:
|
84 |
-
|
85 |
-
```
|
86 |
-
torchrun --nproc_per_node 8 train.py --batch_size 4
|
87 |
-
```
|
88 |
-
|
89 |
-
### Eval
|
90 |
-
|
91 |
-
Please use the following command to evaluate our model:
|
92 |
-
|
93 |
-
```
|
94 |
-
python eval.py
|
95 |
-
```
|
96 |
-
|
97 |
-
|
98 |
-
|
99 |
-
|
100 |
-
## Acknowledgement
|
101 |
-
|
102 |
-
Our code, data preprocessing pipeline, and evaluation scripts are based on several awesome repositories:
|
103 |
-
|
104 |
-
- [DUSt3R](https://github.com/naver/dust3r)
|
105 |
-
- [SplaTAM](https://github.com/spla-tam/SplaTAM)
|
106 |
-
- [NeRFStudio](https://github.com/nerfstudio-project/nerfstudio)
|
107 |
-
- [MVSNet](https://github.com/YoYo000/MVSNet)
|
108 |
-
- [NICE-SLAM](https://github.com/cvg/nice-slam)
|
109 |
-
- [NeuralRGBD](https://github.com/dazinovic/neural-rgbd-surface-reconstruction)
|
110 |
-
- [SimpleRecon](https://github.com/nianticlabs/simplerecon)
|
111 |
-
|
112 |
-
We thank the authors for releasing their code!
|
113 |
-
|
114 |
-
The research presented here has been supported by a sponsored research award from Cisco Research and the UCL Centre for Doctoral Training in Foundational AI under UKRI grant number EP/S021566/1. This project made use of time on Tier 2 HPC facility JADE2, funded by EPSRC (EP/T022205/1).
|
115 |
-
|
116 |
-
## Citation
|
117 |
-
|
118 |
-
If you find our code or paper useful for your research, please consider citing:
|
119 |
-
|
120 |
-
```
|
121 |
-
@article{wang20243d,
|
122 |
-
title={3D Reconstruction with Spatial Memory},
|
123 |
-
author={Wang, Hengyi and Agapito, Lourdes},
|
124 |
-
journal={arXiv preprint arXiv:2408.16061},
|
125 |
-
year={2024}
|
126 |
-
}
|
127 |
-
```
|
128 |
-
|
|
|
1 |
---
|
2 |
+
title: StableRecon
|
3 |
app_file: app.py
|
4 |
sdk: gradio
|
5 |
sdk_version: 4.42.0
|
6 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|