Stanislaw Szymanowicz commited on
Commit
15f992e
1 Parent(s): 11ffd30

Add examples

Browse files
app.py CHANGED
@@ -95,22 +95,23 @@ def main():
95
 
96
  return ply_out_path
97
 
98
- with gr.Blocks() as demo:
 
 
 
 
 
 
 
99
  gr.Markdown(
100
  """
 
101
 
102
- # Splatter Image Demo
103
- [Splatter Image](https://github.com/szymanowiczs/splatter-image) (CVPR 2024) is a fast, super cheap to train method for object 3D reconstruction from a single image.
104
  The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
105
- On NVIDIA V100 GPU, reconstruction can be done at 38FPS and rendering at 588FPS.
106
- Upload an image of an object to see how the Splatter Image does.
107
-
108
- **Comments:**
109
- 1. The first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
110
- 2. The model does not work well on photos of humans.
111
- 3. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. Artefacts might show - see video for more faithful results.
112
- 4. Best results are achieved on the datasets described in the [repository](https://github.com/szymanowiczs/splatter-image) using that code. This demo is experimental.
113
- 5. Our model might not be better than some state-of-the-art methods, but it is of comparable quality and is **much** cheaper to train and run.
114
  """
115
  )
116
  with gr.Row(variant="panel"):
@@ -131,6 +132,26 @@ def main():
131
  )
132
  with gr.Row():
133
  submit = gr.Button("Generate", elem_id="generate", variant="primary")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
  with gr.Column():
135
  with gr.Row():
136
  with gr.Tab("Reconstruction"):
@@ -140,6 +161,31 @@ def main():
140
  interactive=False
141
  )
142
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
143
  submit.click(fn=check_input_image, inputs=[input_image]).success(
144
  fn=preprocess,
145
  inputs=[input_image, preprocess_background],
 
95
 
96
  return ply_out_path
97
 
98
+ css = """
99
+ h1 {
100
+ text-align: center;
101
+ display:block;
102
+ }
103
+ """
104
+
105
+ with gr.Blocks(css=css) as demo:
106
  gr.Markdown(
107
  """
108
+ # Splatter Image
109
 
110
+ **Splatter Image (CVPR 2024)** [[code](https://github.com/szymanowiczs/splatter-image), [project page](https://szymanowiczs.github.io/splatter-image)] is a fast, super cheap-to-train method for object 3D reconstruction from a single image.
 
111
  The model used in the demo was trained on **Objaverse-LVIS on 2 A6000 GPUs for 3.5 days**.
112
+ Locally, on an NVIDIA V100 GPU, reconstruction (forward pass of the network) can be done at 38FPS and rendering (with Gaussian Splatting) at 588FPS.
113
+ Upload an image of an object or click on one of the provided examples to see how the Splatter Image does.
114
+ For best results clone the [main repository](https://github.com/szymanowiczs/splatter-image) and run the demo locally.
 
 
 
 
 
 
115
  """
116
  )
117
  with gr.Row(variant="panel"):
 
132
  )
133
  with gr.Row():
134
  submit = gr.Button("Generate", elem_id="generate", variant="primary")
135
+
136
+ with gr.Row(variant="panel"):
137
+ gr.Examples(
138
+ examples=[
139
+ './demo_examples/01_bigmac.png',
140
+ './demo_examples/02_hydrant.jpg',
141
+ './demo_examples/03_spyro.png',
142
+ './demo_examples/04_lysol.png',
143
+ './demo_examples/05_pinapple_bottle.png',
144
+ './demo_examples/06_unsplash_broccoli.png',
145
+ './demo_examples/07_objaverse_backpack.png',
146
+ './demo_examples/08_unsplash_chocolatecake.png',
147
+ './demo_examples/09_realfusion_cherry.png',
148
+ './demo_examples/10_triposr_teapot.png'
149
+ ],
150
+ inputs=[input_image],
151
+ cache_examples=False,
152
+ label="Examples",
153
+ examples_per_page=20,
154
+ )
155
  with gr.Column():
156
  with gr.Row():
157
  with gr.Tab("Reconstruction"):
 
161
  interactive=False
162
  )
163
 
164
+ gr.Markdown(
165
+ """
166
+ ## Comments:
167
+ 1. If you run the demo online, the first example you upload should take about 4.5 seconds (with preprocessing, saving and overhead), the following take about 1.5s.
168
+ 2. The 3D viewer shows a .ply mesh extracted from a mix of 3D Gaussians. This is only an approximations and artefacts might show.
169
+ 3. Known limitations include:
170
+ - sphere-like artefacts on the object and white halo around it: this is due to how the .ply mesh is extracted and limitations of the Gradio viewer
171
+ - see-through parts of objects, especially on the back: this is due to the model performing less well on more complicated shapes
172
+ - back of objects are blurry: this is a model limiation due to it being deterministic
173
+ 4. Our model is of comparable quality to state-of-the-art methods, and is **much** cheaper to train and run.
174
+
175
+ ## How does it work?
176
+
177
+ Splatter Image formulates 3D reconstruction as an image-to-image translation task. It maps the input image to another image,
178
+ in which every pixel represents one 3D Gaussian and the channels of the output represent parameters of these Gaussians, including their shapes, colours and locations.
179
+ The resulting image thus represents a set of Gaussians (almost like a point cloud) which reconstruct the shape and colour of the object.
180
+ The method is very cheap: the reconstruction amounts to a single forward pass of a neural network with only 2D operators (2D convolutions and attention).
181
+ The rendering is also very fast, due to using Gaussian Splatting.
182
+ Combined, this results in very cheap training and high-quality results.
183
+ For more results see the [project page](https://szymanowiczs.github.io/splatter-image) and the [CVPR article](https://arxiv.org/abs/2312.13150).
184
+ """
185
+ )
186
+
187
+
188
+
189
  submit.click(fn=check_input_image, inputs=[input_image]).success(
190
  fn=preprocess,
191
  inputs=[input_image, preprocess_background],
demo_examples/01_bigmac.png ADDED
demo_examples/02_hydrant.jpg ADDED
demo_examples/03_spyro.png ADDED
demo_examples/04_lysol.png ADDED
demo_examples/05_pinapple_bottle.png ADDED
demo_examples/06_unsplash_broccoli.png ADDED
demo_examples/07_objaverse_backpack.png ADDED
demo_examples/08_unsplash_chocolatecake.png ADDED
demo_examples/09_realfusion_cherry.png ADDED
demo_examples/10_triposr_teapot.png ADDED