hassiahk commited on
Commit
0a7dc31
1 Parent(s): 60e41ba

Fix typos and update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -30
README.md CHANGED
@@ -1,18 +1,17 @@
1
  # Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
 
2
  [![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
3
 
4
  <p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
5
 
6
  Welcome to Putting NeRF on a Diet Project!
7
- This project is the Pytorch, JAX/Flax based code implementation of this paper [Putting NeRF on a Diet : Ajay Jain, Matthew Tancik, Pieter Abbeel, Arxiv : https://arxiv.org/abs/2104.00677]
8
- The model generates the novel view synthesis redering (NeRF: Neural Radiances Field) with Fewshot learning scheme.
9
- The semantic loss use the pre-trained CLIP Vision Transformer embedding. This information can give a 2D supervision for 3D.
10
- The Diet NeRF result outperforms the original NeRF in 3D reconstruction and neural rendering with only few images.
11
 
12
 
13
- ## 🤗 Hugging Face Hub Repo URL:
14
- We will also upload our project on the Hugging Face Hub Repository Also.
15
- [https://huggingface.co/flax-community/putting-nerf-on-a-diet/](https://huggingface.co/flax-community/putting-nerf-on-a-diet/)
16
 
17
  Our JAX/Flax implementation currently supports:
18
 
@@ -53,36 +52,31 @@ Our JAX/Flax implementation currently supports:
53
 
54
  - Streamlit Space Demo
55
 
56
- You can check our Streamlit Space demo on following site !
57
  With any input camera pose, we can render the novel view synthesis.
58
- [https://huggingface.co/spaces/flax-community/DietNerf-Demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
59
 
60
  - Colab Demo
61
 
62
- Moreover, we prapare the colab ipython notebook for you.
63
- You need colab pro account for running our model on the colab(For memory issue)
64
- [https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
65
- ](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
66
- )
67
-
68
 
69
  ## 💻 Installation
70
 
71
  ```bash
72
  # Clone the repo
73
- svn export https://github.com/google-research/google-research/trunk/jaxnerf
74
  # Create a conda environment, note you can use python 3.6-3.8 as
75
  # one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.
76
  conda create --name jaxnerf python=3.6.12; conda activate jaxnerf
77
  # Prepare pip
78
  conda install pip; pip install --upgrade pip
79
  # Install requirements
80
- pip install -r jaxnerf/requirements.txt
81
  # [Optional] Install GPU and TPU support for Jax
82
  # Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.
83
- pip install --upgrade jax jaxlib==0.1.57+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html
84
  # install flax and flax-transformer
85
- pip install flax transformer[flax]
86
  ```
87
 
88
  ## ⚽ Dataset
@@ -93,8 +87,7 @@ in the place you like. Let's assume they are placed under `/tmp/jaxnerf/data/`.
93
 
94
  ## 💖 Methods
95
 
96
- You can check more detail explaination about DietNeRF on following **Notion Report**
97
- * 👉👉 VEEEERY Detail DietNeRF Explaination Docs : https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745
98
 
99
  <p align="center"><img width="400" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/124376591-b312b780-dce2-11eb-80ad-9129d6f5eedb.png"></p>
100
 
@@ -120,7 +113,7 @@ python -m train \
120
  ```
121
  You can toggle the semantic loss by “use_semantic_loss” in configuration files.
122
 
123
- ## 💎 Expriment Result
124
 
125
  ### ❗ Rendered Rendering images by 8-shot learned Diet-NeRF
126
 
@@ -133,9 +126,9 @@ DietNeRF has a strong capacity to generalise on novel and challenging views with
133
 
134
  ### ❗ Rendered GIF by occluded 14-shot learned NeRF and Diet-NeRF
135
 
136
- We made aritificial occulusion on the right side of image (Only picked left side training poses).
137
  The reconstruction quality can be compared with this experiment.
138
- Diet NeRF shows better quailty than Original NeRF when It is occulused.
139
 
140
  #### Training poses
141
  <img width="1400" src="https://user-images.githubusercontent.com/26036843/126111980-4f332c87-a7f0-42e0-a355-8e77621bbca4.png">
@@ -163,9 +156,9 @@ Diet NeRF shows better quailty than Original NeRF when It is occulused.
163
  | Project Managing | [Stella Yang](https://github.com/codestella) To Watch Our Project Progress, Please Check [Our Project Notion](https://www.notion.so/Putting-NeRF-on-a-Diet-e0caecea0c2b40c3996c83205baf870d) |
164
  | NeRF Team | [Stella Yang](https://github.com/codestella), [Alex Lau](https://github.com/riven314), [Seunghyun Lee](https://github.com/sseung0703), [Hyunkyu Kim](https://github.com/minus31), [Haswanth Aekula](https://github.com/hassiahk), [JaeYoung Chung](https://github.com/robot0321) |
165
  | CLIP Team | [Seunghyun Lee](https://github.com/sseung0703), [Sasikanth Kotti](https://github.com/ksasi), [Khali Sifullah](https://github.com/khalidsaifullaah) , [Sunghyun Kim](https://github.com/MrBananaHuman) |
166
- | Cloud TPU Team | [Alex Lau](https://github.com/riven314), [Aswin Pyakurel](https://github.com/masapasa) , [JaeYoung Chung](https://github.com/robot0321), [Sunghyun Kim](https://github.com/MrBananaHuman) |
167
 
168
- * Extremely Don't Sleep Contributors 🤣 : [Seunghyun Lee](https://github.com/sseung0703), [Alex Lau](https://github.com/riven314), [Stella Yang](https://github.com/codestella), [Haswanth Aekula](https://github.com/hassiahk)
169
 
170
  ## 😎 What we improved from original JAX-NeRF : Innovation
171
  - Neural rendering with fewshot images
@@ -218,16 +211,14 @@ This project is based on “Putting NeRF on a Diet”.
218
 
219
  ## ❤️ Special Thanks
220
 
221
-
222
- Our Project is started in the HuggingFace X GoogleAI (JAX) Community Week Event.
223
- https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104
224
 
225
  Thank you for our mentor Suraj and organizers in JAX/Flax Community Week!
226
  Our team grows up with this community learning experience. It was wonderful time!
227
 
228
  <img width="250" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126369170-5664076c-ac99-4157-bc53-b91dfb7ed7e1.jpeg">
229
 
230
- Common Computer AI(https://comcom.ai/ko/) sponsored the multiple V100 GPUs for our project!
231
  Thank you so much for your support!
232
  <img width="250" alt="스크린샷" src="https://user-images.githubusercontent.com/77657524/126914984-d959be06-19f4-4228-8d3a-a855396b2c3f.jpeg">
233
 
 
1
  # Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
2
+
3
  [![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
4
 
5
  <p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
6
 
7
  Welcome to Putting NeRF on a Diet Project!
8
+ This project is a JAX/Flax based code implementation of the paper [Putting NeRF on a Diet](https://arxiv.org/abs/2104.00677). The model generates the novel view synthesis rendering (NeRF: Neural Radiances Field) with Few-shot learning scheme.
9
+ The semantic loss uses the pre-trained CLIP Vision Transformer embedding. This information can give a 2D supervision for 3D.
10
+ The Diet NeRF result outperforms the original NeRF in 3D reconstruction and neural rendering with only few training images.
 
11
 
12
 
13
+ ## 🤗 Hugging Face Model Hub Repo
14
+ You can also find our project on the [Hugging Face Model Hub Repository](https://huggingface.co/flax-community/putting-nerf-on-a-diet/).
 
15
 
16
  Our JAX/Flax implementation currently supports:
17
 
 
52
 
53
  - Streamlit Space Demo
54
 
55
+ You can check our [Streamlit Space demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo).
56
  With any input camera pose, we can render the novel view synthesis.
 
57
 
58
  - Colab Demo
59
 
60
+ Moreover, we have also prapared a [Colab notebook](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing).
61
+ You need colab pro account for running our model on the colab (To avoid out-of-memory issue)
 
 
 
 
62
 
63
  ## 💻 Installation
64
 
65
  ```bash
66
  # Clone the repo
67
+ git clone https://github.com/codestella/putting-nerf-on-a-diet
68
  # Create a conda environment, note you can use python 3.6-3.8 as
69
  # one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.
70
  conda create --name jaxnerf python=3.6.12; conda activate jaxnerf
71
  # Prepare pip
72
  conda install pip; pip install --upgrade pip
73
  # Install requirements
74
+ pip install -r requirements.txt
75
  # [Optional] Install GPU and TPU support for Jax
76
  # Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.
77
+ !pip install --upgrade jax "jax[cuda110]" -f https://storage.googleapis.com/jax-releases/jax_releases.html
78
  # install flax and flax-transformer
79
+ pip install flax transformers[flax]
80
  ```
81
 
82
  ## ⚽ Dataset
 
87
 
88
  ## 💖 Methods
89
 
90
+ * 👉👉 You can check VEEEERY detailed explanation about our project on [Notion Report](https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)
 
91
 
92
  <p align="center"><img width="400" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/124376591-b312b780-dce2-11eb-80ad-9129d6f5eedb.png"></p>
93
 
 
113
  ```
114
  You can toggle the semantic loss by “use_semantic_loss” in configuration files.
115
 
116
+ ## 💎 Experimental Results
117
 
118
  ### ❗ Rendered Rendering images by 8-shot learned Diet-NeRF
119
 
 
126
 
127
  ### ❗ Rendered GIF by occluded 14-shot learned NeRF and Diet-NeRF
128
 
129
+ We made artificial occlusion on the right side of image (Only picked left side training poses).
130
  The reconstruction quality can be compared with this experiment.
131
+ DietNeRF shows better quality than Original NeRF when It is occluded.
132
 
133
  #### Training poses
134
  <img width="1400" src="https://user-images.githubusercontent.com/26036843/126111980-4f332c87-a7f0-42e0-a355-8e77621bbca4.png">
 
156
  | Project Managing | [Stella Yang](https://github.com/codestella) To Watch Our Project Progress, Please Check [Our Project Notion](https://www.notion.so/Putting-NeRF-on-a-Diet-e0caecea0c2b40c3996c83205baf870d) |
157
  | NeRF Team | [Stella Yang](https://github.com/codestella), [Alex Lau](https://github.com/riven314), [Seunghyun Lee](https://github.com/sseung0703), [Hyunkyu Kim](https://github.com/minus31), [Haswanth Aekula](https://github.com/hassiahk), [JaeYoung Chung](https://github.com/robot0321) |
158
  | CLIP Team | [Seunghyun Lee](https://github.com/sseung0703), [Sasikanth Kotti](https://github.com/ksasi), [Khali Sifullah](https://github.com/khalidsaifullaah) , [Sunghyun Kim](https://github.com/MrBananaHuman) |
159
+ | Cloud TPU Team | [Alex Lau](https://github.com/riven314), [Aswin Pyakurel](https://github.com/masapasa), [JaeYoung Chung](https://github.com/robot0321), [Sunghyun Kim](https://github.com/MrBananaHuman) |
160
 
161
+ * Extremely Don't Sleep Contributors 🤣: [Seunghyun Lee](https://github.com/sseung0703), [Alex Lau](https://github.com/riven314), [Stella Yang](https://github.com/codestella), [Haswanth Aekula](https://github.com/hassiahk)
162
 
163
  ## 😎 What we improved from original JAX-NeRF : Innovation
164
  - Neural rendering with fewshot images
 
211
 
212
  ## ❤️ Special Thanks
213
 
214
+ Our Project is started in the [HuggingFace X GoogleAI (JAX) Community Week Event](https://discuss.huggingface.co/t/open-to-the-community-community-week-using-jax-flax-for-nlp-cv/7104).
 
 
215
 
216
  Thank you for our mentor Suraj and organizers in JAX/Flax Community Week!
217
  Our team grows up with this community learning experience. It was wonderful time!
218
 
219
  <img width="250" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126369170-5664076c-ac99-4157-bc53-b91dfb7ed7e1.jpeg">
220
 
221
+ [Common Computer AI](https://comcom.ai/en/) sponsored multiple V100 GPUs for our project!
222
  Thank you so much for your support!
223
  <img width="250" alt="스크린샷" src="https://user-images.githubusercontent.com/77657524/126914984-d959be06-19f4-4228-8d3a-a855396b2c3f.jpeg">
224