codestella commited on
Commit
31baa93
1 Parent(s): 40c15ce

change README

Browse files
Files changed (1) hide show
  1. README.md +25 -22
README.md CHANGED
@@ -1,14 +1,34 @@
1
  # Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
 
2
  [![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
3
 
4
  <p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
5
 
6
- Welcome to Putting NeRF on a Diet Project!
7
- This project is the Pytorch, JAX/Flax based code implementation of this paper [Putting NeRF on a Diet : Ajay Jain, Matthew Tancik, Pieter Abbeel, Arxiv : https://arxiv.org/abs/2104.00677]
8
- The model generates the novel view synthesis redering (NeRF: Neural Radiances Field) with Fewshot learning scheme.
9
- The semantic loss use the pre-trained CLIP Vision Transformer embedding. This information can give a 2D supervision for 3D.
10
- The Diet NeRF result outperforms the original NeRF in 3D reconstruction and neural rendering with only few images.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
 
 
 
12
 
13
  ## 🤗 Hugging Face Hub Repo URL:
14
  We will also upload our project on the Hugging Face Hub Repository Also.
@@ -49,23 +69,6 @@ Our JAX/Flax implementation currently supports:
49
  </tbody>
50
  </table>
51
 
52
- ## 🤩 Demo
53
-
54
- - Streamlit Space Demo
55
-
56
- You can check our Streamlit Space demo on following site !
57
- With any input camera pose, we can render the novel view synthesis.
58
- [https://huggingface.co/spaces/flax-community/DietNerf-Demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
59
-
60
- - Colab Demo
61
-
62
- Moreover, we prapare the colab ipython notebook for you.
63
- You need colab pro account for running our model on the colab(For memory issue)
64
- [https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
65
- ](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
66
- )
67
-
68
-
69
  ## 💻 Installation
70
 
71
  ```bash
 
1
  # Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
2
+
3
  [![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
4
 
5
  <p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
6
 
7
+ This project attempted to implement the paper **[Putting NeRF on a Diet](https://arxiv.org/abs/2104.00677)** (DietNeRF) in JAX/Flax.
8
+ DietNeRF is designed for rendering quality novel views in few-shot learning scheme, a task that vanilla NeRF (Neural Radiance Field) struggles.
9
+ To achieve this, the author coins **Semantic Consistency Loss** to supervise DietNeRF by prior knowledge from CLIP Vision Transformer. Such supervision enables DietNeRF to learn 3D scene reconstruction with CLIP's prior knowledge on 2D views.
10
+
11
+ Besides this repo, you can check our write-up and demo here:
12
+ - ✍️ **[Write-up in Notion](https://steep-cycle-f6b.notion.site/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**: more details of DietNeRF and our experiments
13
+ - ✨ **[Demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)**: showcase our trained DietNeRFs by Streamlit
14
+
15
+ ## 🤩 Demo
16
+ 1. You can check out [our demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
17
+ 2. Or you can set up our Streamlit demo locally (model checkpoints will be fetched automatically upon startup)
18
+ ```shell
19
+ pip install -r requirements_demo.txt
20
+ streamlit run app.py
21
+ ```
22
+
23
+ <p align="center"><img width="600" height="400" alt="Streamlit Demo" src="assets/space_demo.png"></p>
24
+
25
+ ## ✨ Implementation
26
+
27
+ Our code is written in JAX/ Flax and mainly based upon [jaxnerf](https://github.com/google-research/google-research/tree/master/jaxnerf) from Google Research. The base code is highly optimized in GPU & TPU. For semantic consistency loss, we utilize pretrained CLIP Vision Transformer from [transformers](https://github.com/huggingface/transformers) library.
28
 
29
+ To learn more about DietNeRF, our experiments and implementation, you are highly recommended to check out our very detailed **[Notion write-up](https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**!
30
+
31
+ <p align="center"><img width="500" height="600" alt="스크린샷 2021-07-04 오후 4 11 51" src="assets/report_thumbnail.png"></p>
32
 
33
  ## 🤗 Hugging Face Hub Repo URL:
34
  We will also upload our project on the Hugging Face Hub Repository Also.
 
69
  </tbody>
70
  </table>
71
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
  ## 💻 Installation
73
 
74
  ```bash