initial

README.md

@@ -1,13 +1,122 @@
----
-title: Sat3density
-emoji: 🏆
-colorFrom: green
-colorTo: blue
-sdk: gradio
-sdk_version: 3.41.2
-app_file: app.py
-pinned: false
-license: other
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs
+
+> [Ming Qian](https://qianmingduowan.github.io/), Jincheng Xiong, [Gui-Song Xia](http://www.captain-whu.com/xia_En.html), [Nan Xue](https://xuenan.net)
+>
+> IEEE/CVF International Conference on Computer Vision (ICCV), 2023
+>
+> [Project](https://sat2density.github.io/) | [Paper](https://arxiv.org/abs/2303.14672) | [Data]() | [Install.md](docs/INSTALL.md)
+
+> <p align="center" float="left">
+>    <img src="docs/figures/demo/case1.sat.gif" alt="drawing" width="19%">  
+>    <img src="docs/figures/demo-density/case1.gif" alt="drawing" width="38%">
+>    <img src="docs/figures/demo/case1.render.gif" alt="drawing" width="38%">
+> </p>
+
+> <p align="center" float="left">
+>    <img src="docs/figures/demo/case2.sat.gif" alt="drawing" width="19%">  
+>    <img src="docs/figures/demo-density/case2.gif" alt="drawing" width="38%">
+>    <img src="docs/figures/demo/case2.render.gif" alt="drawing" width="38%">
+> </p>
+
+> <p align="center" float="left">
+>    <img src="docs/figures/demo/case3.sat.gif" alt="drawing" width="19%">  
+>    <img src="docs/figures/demo-density/case3.gif" alt="drawing" width="38%">
+>    <img src="docs/figures/demo/case3.render.gif" alt="drawing" width="38%">
+> </p>
+
+> <p align="center" float="left">
+>    <img src="docs/figures/demo/case4.sat.gif" alt="drawing" width="19%">  
+>    <img src="docs/figures/demo-density/case4.gif" alt="drawing" width="38%">
+>    <img src="docs/figures/demo/case4.render.gif" alt="drawing" width="38%">
+> </p>
+
+## Checkpoints Downloading
+> Two checkpoints for CVACT and CVUSA can be found from [thisurl](https://github.com/sat2density/checkpoints/releases). You can also run the following command to download them.
+```
+bash scripts/download_weights.sh
+```
+
+## QuickStart Demo
+### Video Synthesis
+  #### Example Usage
+  ```
+  python test.py --yaml=sat2density_cvact \
+    --test_ckpt_path=2u87bj8w \
+    --task=test_vid \
+    --demo_img=demo_img/case1/satview-input.png  \
+    --sty_img=demo_img/case1/groundview.image.png  \
+    --save_dir=results/case1
+  ```
+  #### 
+
+### Illumination Interpolation
+<!-- ```
+bash inference/quick_demo_interpolation.sh
+``` -->
+```
+python test.py --task=test_interpolation \
+--yaml=sat2density_cvact \
+--test_ckpt_path=2u87bj8w \
+--sty_img1=demo_img/case9/groundview.image.png \
+--sty_img2=demo_img/case7/groundview.image.png \
+--demo_img=demo_img/case3/satview-input.png \
+--save_dir=results/case2
+```
+
+## Train & Inference
+- *We trained our model using 1 V100 32GB GPU. The training phase will take about 20 hours.*
+- *For data preparation, please check out [data.md](dataset/INSTALL.md).*
+
+
+
+
+### Inference
+
+To test Center Ground-View Synthesis setting
+If you want save results, please add --task=vis_test
+```bash
+# CVACT
+python offline_train_test.py --yaml=sat2density_cvact --test_ckpt_path=2u87bj8w
+# CVUSA
+python offline_train_test.py --yaml=sat2density_cvusa --test_ckpt_path=2cqv8uh4
+```
+
+To test inference with different illumination
+```bash
+# CVACT
+bash inference/single_style_test_cvact.sh
+# CVUSA
+bash inference/single_style_test_cvusa.sh
+```
+
+To test synthesis ground videos
+```bash
+bash inference/synthesis_video.sh
+```
+
+## Training
+
+### Training command
+
+```bash
+# CVACT
+CUDA_VISIBLE_DEVICES=X python train.py --yaml=sat2density_cvact
+# CVUSA
+CUDA_VISIBLE_DEVICES=X python train.py --yaml=sat2density_cvusa
+```
+
+## Citation
+If you use this code for your research, please cite
+
+```
+@inproceedings{qian2021sat2density,
+  title={Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs},
+  author={Qian, Ming and Xiong, Jincheng and Xia, Gui-Song and Xue, Nan},
+  booktitle={ICCV},
+  year={2023}
+}
+```
+
+## License
+This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
+For commercial use, please contact [mingqian@whu.edu.cn].

app.py

@@ -0,0 +1,244 @@
+import gradio as gr
+import numpy as np
+import os
+from PIL import Image
+import torch
+import torchvision.transforms as transforms
+import options
+import test
+import importlib
+from scipy.interpolate import interp1d, splev, splprep
+import cv2
+
+
+def get_single(sat_img, style_img, x_offset, y_offset):
+    name = ''
+    for i in [name for name in os.listdir('demo_img') if 'case' in name]:
+        style = Image.open('demo_img/{}/groundview.image.png'.format(i)).convert('RGB')
+        style =np.array(style)
+        if (style == style_img).all():
+            name = i
+            break
+
+    input_dict = {}
+    trans = transforms.ToTensor()
+    input_dict['sat'] = trans(sat_img)
+    input_dict['pano'] = trans(style_img)
+    input_dict['paths'] = "demo.png"
+    sky = trans(Image.open('demo_img/{}/groundview.sky.png'.format(name)).convert("L"))
+    input_a = input_dict['pano']*sky
+    sky_histc = torch.cat([input_a[i].histc()[10:] for i in reversed(range(3))])
+    input_dict['sky_histc'] = sky_histc
+    input_dict['sky_mask'] = sky
+
+    for key in input_dict.keys():
+        if isinstance(input_dict[key], torch.Tensor):
+            input_dict[key] = input_dict[key].unsqueeze(0)
+
+    args = ["--yaml=sat2density_cvact", "--test_ckpt_path=wandb/run-20230219_141512-2u87bj8w/files/checkpoint/model.pth", "--task=test_vid", "--demo_img=demo_img/case1/satview-input.png",
+            "--sty_img=demo_img/case1/groundview.image.png", "--save_dir=output"]
+    opt_cmd = options.parse_arguments(args=args)
+    opt = options.set(opt_cmd=opt_cmd)
+    opt.isTrain = False
+    opt.name = opt.yaml if opt.name is None else opt.name
+    opt.batch_size = 1
+
+    m = importlib.import_module("model.{}".format(opt.model))
+    model = m.Model(opt)
+
+    # m.load_dataset(opt)
+    model.build_networks(opt)
+    ckpt = torch.load(opt.test_ckpt_path, map_location='cpu')
+    model.netG.load_state_dict(ckpt['netG'])
+    model.netG.eval()
+
+    model.set_input(input_dict)
+    
+    model.style_temp = model.sky_histc
+    opt.origin_H_W = [-(y_offset*256-128)/128, (x_offset*256-128)/128] # TODO: hard code should be removed in the future
+
+    model.forward(opt)
+
+    rgb = model.out_put.pred[0].clamp(min=0,max=1.0).cpu().detach().numpy().transpose((1,2,0))
+    rgb = np.array(rgb*255, dtype=np.uint8)
+    return  rgb
+
+def get_video(sat_img, style_img, positions):
+    name = ''
+    for i in [name for name in os.listdir('demo_img') if 'case' in name]:
+        style = Image.open('demo_img/{}/groundview.image.png'.format(i)).convert('RGB')
+        style =np.array(style)
+        if (style == style_img).all():
+            name = i
+            break
+
+    input_dict = {}
+    trans = transforms.ToTensor()
+    input_dict['sat'] = trans(sat_img)
+    input_dict['pano'] = trans(style_img)
+    input_dict['paths'] = "demo.png"
+    sky = trans(Image.open('demo_img/{}/groundview.sky.png'.format(name)).convert("L"))
+    input_a = input_dict['pano']*sky
+    sky_histc = torch.cat([input_a[i].histc()[10:] for i in reversed(range(3))])
+    input_dict['sky_histc'] = sky_histc
+    input_dict['sky_mask'] = sky
+
+    for key in input_dict.keys():
+        if isinstance(input_dict[key], torch.Tensor):
+            input_dict[key] = input_dict[key].unsqueeze(0)
+
+    args = ["--yaml=sat2density_cvact", "--test_ckpt_path=wandb/run-20230219_141512-2u87bj8w/files/checkpoint/model.pth", "--task=test_vid", "--demo_img=demo_img/case1/satview-input.png",
+            "--sty_img=demo_img/case1/groundview.image.png", "--save_dir=output"]
+    opt_cmd = options.parse_arguments(args=args)
+    opt = options.set(opt_cmd=opt_cmd)
+    opt.isTrain = False
+    opt.name = opt.yaml if opt.name is None else opt.name
+    opt.batch_size = 1
+
+    m = importlib.import_module("model.{}".format(opt.model))
+    model = m.Model(opt)
+
+    # m.load_dataset(opt)
+    model.build_networks(opt)
+    ckpt = torch.load(opt.test_ckpt_path, map_location='cpu')
+    model.netG.load_state_dict(ckpt['netG'])
+    model.netG.eval()
+
+    model.set_input(input_dict)
+    
+    model.style_temp = model.sky_histc
+
+    unique_lst = list(dict.fromkeys(positions))
+    pixels = []
+    for x in positions:
+        if x in unique_lst:
+            if x not in pixels:
+                pixels.append(x)
+    pixels = np.array(pixels)
+    tck, u = splprep(pixels.T, s=25, per=0)
+    u_new = np.linspace(u.min(), u.max(), 80)
+    x_new, y_new = splev(u_new, tck)
+    smooth_path = np.array([x_new,y_new]).T
+
+    rendered_image_list = []
+    rendered_depth_list = []
+
+
+    for i, (x,y) in enumerate(smooth_path):
+        opt.origin_H_W = [(y-128)/128, (x-128)/128] # TODO: hard code should be removed in the future
+        print('Rendering at ({}, {})'.format(x,y))
+        model.forward(opt)
+
+        rgb = model.out_put.pred[0].clamp(min=0,max=1.0).cpu().detach().numpy().transpose((1,2,0))
+        rgb = np.array(rgb*255, dtype=np.uint8)
+        rendered_image_list.append(rgb)
+
+        rendered_depth_list.append(
+            model.out_put.depth[0,0].cpu().detach().numpy()
+        )
+
+    output_video_path = 'output_video.mp4'
+
+    # 设置视频的帧率、宽度和高度
+    frame_rate = 15
+    frame_width = 512
+    frame_height = 128
+
+    # 使用OpenCV创建视频写入对象，选择H.264编码器
+    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
+    out = cv2.VideoWriter(output_video_path, fourcc, frame_rate, (frame_width, frame_height))
+
+    # 遍历图像列表并将它们写入视频
+    for image_np in rendered_image_list:
+        image_np = cv2.cvtColor(image_np, cv2.COLOR_BGR2RGB)
+        out.write(image_np)
+
+    # 释放视频写入对象
+    out.release()
+
+    return "output_video.mp4"
+
+def copy_image(image):
+    return image
+
+def show_image_and_point(image, x, y):
+    x = int(x*image.shape[1])
+    y = image.shape[0]-int(y*image.shape[0])
+    mask = np.zeros(image.shape[:2])
+    radius = min(image.shape[0], image.shape[1])//60
+    for i in range(x-radius-2, x+radius+2):
+        for j in range(y-radius-2, y+radius+2):
+            if (i-x)**2+(j-y)**2<=radius**2:
+                mask[j, i] = 1
+    return (image, [(mask, 'render point')])
+
+def add_select_point(image, evt: gr.SelectData, state1):
+    if state1 == None:
+        state1 = []
+    x, y = evt.index
+    state1.append((x, y))
+    print(state1)
+    radius = min(image.shape[0], image.shape[1])//60
+    for i in range(x-radius-2, x+radius+2):
+        for j in range(y-radius-2, y+radius+2):
+            if (i-x)**2+(j-y)**2<=radius**2:
+                image[j, i, :] = 0
+    return image, state1
+
+def reset_select_points(image):
+    return image, []
+
+
+
+
+
+
+with gr.Blocks() as demo:    
+    gr.Markdown("# Sat2Density Demos")
+    gr.Markdown("### select/upload the satllite image and select the style image")
+    with gr.Row():
+        with gr.Column():
+            sat_img = gr.Image(source='upload', shape=[256, 256], interactive=True)
+            img_examples = gr.Examples(examples=['demo_img/{}/satview-input.png'.format(i) for i in os.listdir('demo_img') if 'case' in i],
+                inputs=sat_img, outputs=None, examples_per_page=20)
+        with gr.Column():
+            style_img = gr.Image()
+            style_examples = gr.Examples(examples=['demo_img/{}/groundview.image.png'.format(i) for i in os.listdir('demo_img') if 'case' in i],
+                inputs=style_img, outputs=None, examples_per_page=20)
+
+
+    gr.Markdown("### select a certain point to generate single groundview image")
+    with gr.Row():
+        with gr.Column():
+            with gr.Row():
+                with gr.Column():
+                    slider_x = gr.Slider(0.2, 0.8, 0.5, label="x-axis position")
+                    slider_y = gr.Slider(0.2, 0.8, 0.5, label="y-axis position")
+                    btn_single = gr.Button(label="demo1")
+                
+                annotation_image = gr.AnnotatedImage()
+    
+        out_single = gr.Image()
+        
+    gr.Markdown("### draw a trajectory on the map to generate video")
+    state_select_points = gr.State()
+    with gr.Row():
+        with gr.Column():
+            draw_img = gr.Image(shape=[256, 256], interactive=True)  
+        with gr.Column():
+            out_video = gr.Video()
+            reset_btn =gr.Button(value="Reset")
+            btn_video = gr.Button(label="demo1")
+
+    sat_img.change(copy_image, inputs = sat_img, outputs=draw_img)
+
+    draw_img.select(add_select_point, [draw_img, state_select_points], [draw_img, state_select_points])
+    sat_img.change(show_image_and_point, inputs = [sat_img, slider_x, slider_y], outputs = annotation_image)
+    slider_x.change(show_image_and_point, inputs = [sat_img, slider_x, slider_y], outputs = annotation_image, show_progress='hidden')
+    slider_y.change(show_image_and_point, inputs = [sat_img, slider_x, slider_y], outputs = annotation_image, show_progress='hidden')
+    btn_single.click(get_single, inputs = [sat_img, style_img, slider_x, slider_y], outputs=out_single)
+    reset_btn.click(reset_select_points, [sat_img], [draw_img, state_select_points])
+    btn_video.click(get_video, inputs=[sat_img, style_img, state_select_points], outputs=out_video) # 触发
+
+
+demo.launch()

data/CVACT_Shi.py

@@ -0,0 +1,119 @@
+import torch,os
+from torch.utils.data.dataset import Dataset
+from PIL import Image
+import scipy.io as sio
+import torchvision.transforms as transforms
+
+def data_list(img_root,mode):
+    exist_aer_list = os.listdir(os.path.join(img_root , 'satview_correct'))
+    exist_grd_list = os.listdir(os.path.join(img_root , 'streetview'))
+    allDataList = os.path.join(img_root, 'ACT_data.mat')
+    anuData = sio.loadmat(allDataList)
+
+    all_data_list = []
+    for i in range(0, len(anuData['panoIds'])):
+        grd_id_align = anuData['panoIds'][i] + '_grdView.png'
+        sat_id_ori = anuData['panoIds'][i] + '_satView_polish.png'
+        all_data_list.append([grd_id_align, sat_id_ori])
+
+    data_list = []
+    
+    if mode=='train':
+        training_inds = anuData['trainSet']['trainInd'][0][0] - 1
+        trainNum = len(training_inds)
+        for k in range(trainNum):
+            data_list.append(all_data_list[training_inds[k][0]])
+    else:
+        val_inds = anuData['valSet']['valInd'][0][0] - 1
+        valNum = len(val_inds)
+        for k in range(valNum):
+            data_list.append(all_data_list[val_inds[k][0]])
+
+
+    pano_list = [img_root + 'streetview/' + item[0] for item in data_list if item[0] in exist_grd_list and item[1] in exist_aer_list]
+
+    return pano_list
+    
+def img_read(img,size=None,datatype='RGB'):
+    img = Image.open(img).convert('RGB' if datatype=='RGB' else "L")
+    if size:
+        if type(size) is int:
+            size = (size,size)
+        img = img.resize(size = size,resample=Image.BICUBIC if datatype=='RGB' else Image.NEAREST)
+    img = transforms.ToTensor()(img)
+    return img
+
+
+class Dataset(Dataset):
+    def __init__(self, opt,split='train',sub=None,sty_img=None):
+        if sty_img:
+            assert sty_img.endswith('grdView.png')
+            demo_img_path = os.path.join(opt.data.root,'streetview',sty_img)
+            self.pano_list = [demo_img_path]
+
+        elif opt.task in  ['test_vid','test_interpolation'] :
+            demo_img_path = os.path.join(opt.data.root,'streetview',opt.demo_img.replace('satView_polish.png','grdView.png'))
+            self.pano_list = [demo_img_path]
+
+        else:
+            self.pano_list = data_list(img_root=opt.data.root,mode=split)
+            if sub:
+                self.pano_list = self.pano_list[:sub]
+        
+        # select some ground images to test the influence of different skys.
+        # different skys guide different illumination intensity, colors, and etc.
+        if opt.task == 'test_sty':
+            demo_name = [
+                'dataset/CVACT/streetview/pPfo7qQ1fP_24rXrJ2Uxog_grdView.png',
+                'dataset/CVACT/streetview/YL81FiK9PucIvAkr1FHkpA_grdView.png',
+                'dataset/CVACT/streetview/Tzis1jBKHjbXiVB2oRYwAQ_grdView.png',
+                'dataset/CVACT/streetview/eqGgeBLGXRhSj6c-0h0KoQ_grdView.png',
+                'dataset/CVACT/streetview/pdZmLHYEhe2PHj_8-WHMhw_grdView.png',
+                'dataset/CVACT/streetview/ehsu9Q3iTin5t52DM-MwyQ_grdView.png',
+                'dataset/CVACT/streetview/agLEcuq3_-qFj7wwGbktVg_grdView.png',
+                'dataset/CVACT/streetview/HwQIDdMI3GfHyPGtCSo6aA_grdView.png',
+                'dataset/CVACT/streetview/hV8svb3ZVXcQ0AtTRFE1dQ_grdView.png',
+                'dataset/CVACT/streetview/fzq2mBfKP3UIczAd9KpMMg_grdView.png',
+                'dataset/CVACT/streetview/acRP98sACUIlwl2ZIsEyiQ_grdView.png',
+                'dataset/CVACT/streetview/WSh9tNVryLdupUlU0ri2tQ_grdView.png',
+                'dataset/CVACT/streetview/FhEuB9NA5o08VJ_TBCbHjw_grdView.png',
+                'dataset/CVACT/streetview/YHfpn2Mgu1lqgT2OUeBpOg_grdView.png',
+                'dataset/CVACT/streetview/vNhv7ZP1dUkJ93UwFXagJw_grdView.png',
+            ]
+            self.pano_list = demo_name
+
+        self.opt = opt
+
+    def __len__(self):
+        return len(self.pano_list)
+
+    def __getitem__(self, index):
+        pano = self.pano_list[index]
+        aer = pano.replace('streetview','satview_correct').replace('_grdView','_satView_polish')
+        if self.opt.data.sky_mask:
+            sky = pano.replace('streetview','pano_sky_mask')
+        name = pano
+        aer = img_read(aer,  size = self.opt.data.sat_size)
+        pano = img_read(pano,size = self.opt.data.pano_size)
+        if self.opt.data.sky_mask:
+            sky = img_read(sky,size=self.opt.data.pano_size,datatype='L')
+
+        input = {}
+        input['sat']=aer
+        input['pano']=pano
+        input['paths']=name
+        if self.opt.data.sky_mask:
+            input['sky_mask']=sky
+            black_ground = torch.zeros_like(pano)
+            if self.opt.data.histo_mode =='grey':
+                input['sky_histc'] = (pano*sky+black_ground*(1-sky)).histc()[10:] 
+            elif self.opt.data.histo_mode in ['rgb','RGB']:
+                input_a  = (pano*sky+black_ground*(1-sky))
+                for idx in range(len(input_a)):
+                    if idx == 0:
+                        sky_histc = input_a[idx].histc()[10:]
+                    else:
+                        sky_histc = torch.cat([input_a[idx].histc()[10:],sky_histc],dim=0)
+                input['sky_histc'] = sky_histc
+        return input
+

data/CVUSA.py

@@ -0,0 +1,86 @@
+import torch,os
+from torch.utils.data.dataset import Dataset
+from PIL import Image
+import torchvision.transforms as transforms
+import re
+from easydict import EasyDict as edict
+
+def data_list(img_root,mode):
+    data_list=[]
+    if mode=='train':
+        split_file=os.path.join(img_root, 'splits/train-19zl.csv')
+        with open(split_file) as f:
+            list = f.readlines()
+            for i in list:
+                aerial_name=re.split(r',', re.split('\n', i)[0])[0]
+                panorama_name = re.split(r',', re.split('\n', i)[0])[1]
+                data_list.append([aerial_name, panorama_name])
+    else:
+        split_file=os.path.join(img_root+'splits/val-19zl.csv')
+        with open(split_file) as f:
+            list = f.readlines()
+            for i in list:
+                aerial_name=re.split(r',', re.split('\n', i)[0])[0]
+                panorama_name = re.split(r',', re.split('\n', i)[0])[1]
+                data_list.append([aerial_name, panorama_name])
+    print('length of dataset is: ', len(data_list))
+    return [os.path.join(img_root, i[1]) for i in data_list]
+    
+def img_read(img,size=None,datatype='RGB'):
+    img = Image.open(img).convert('RGB' if datatype=='RGB' else "L")
+    if size:
+        if type(size) is int:
+            size = (size,size)
+        img = img.resize(size = size,resample=Image.BICUBIC if datatype=='RGB' else Image.NEAREST)
+    img = transforms.ToTensor()(img)
+    return img
+
+
+class Dataset(Dataset):
+    def __init__(self, opt,split='train',sub=None,sty_img=None):
+        self.pano_list = data_list(img_root=opt.data.root,mode=split)
+        if sub:
+            self.pano_list = self.pano_list[:sub]
+        if opt.task == 'test_vid':
+            demo_img_path = os.path.join(opt.data.root, 'streetview/panos', opt.demo_img)
+            self.pano_list = [demo_img_path]
+        if sty_img:
+            assert opt.sty_img.split('.')[-1] == 'jpg'
+            demo_img_path = os.path.join(opt.data.root, 'streetview/panos', opt.sty_img)
+            self.pano_list = [demo_img_path]
+
+        self.opt = opt
+
+    def __len__(self):
+        return len(self.pano_list)
+
+    def __getitem__(self, index):
+        pano = self.pano_list[index]
+        aer = pano.replace('streetview/panos', 'bingmap/19')
+        if self.opt.data.sky_mask:
+            sky = pano.replace('streetview/panos','sky_mask').replace('jpg', 'png')
+        name = pano
+        aer = img_read(aer,  size = self.opt.data.sat_size)
+        pano = img_read(pano,size = self.opt.data.pano_size)
+        if self.opt.data.sky_mask:
+            sky = img_read(sky,size=self.opt.data.pano_size,datatype='L')
+
+        input = {}
+        input['sat']=aer
+        input['pano']=pano
+        input['paths']=name
+        if self.opt.data.sky_mask:
+            input['sky_mask']=sky
+            black_ground = torch.zeros_like(pano)
+            if self.opt.data.histo_mode =='grey':
+                input['sky_histc'] = (pano*sky+black_ground*(1-sky)).histc()[10:] 
+            elif self.opt.data.histo_mode in ['rgb','RGB']:
+                input_a  = (pano*sky+black_ground*(1-sky))
+                for idx in range(len(input_a)):
+                    if idx == 0:
+                        sky_histc = input_a[idx].histc()[10:]
+                    else:
+                        sky_histc = torch.cat([input_a[idx].histc()[10:],sky_histc],dim=0)
+                input['sky_histc'] = sky_histc
+        return input
+

dataset/INSTALL.md

@@ -0,0 +1,32 @@
+For reproduce our paper,
+
+you should first download 4 zip file:
+
+`
+CVACT/satview_correct.zip , 
+CVACT/streetview.zip , 
+CVUSA/bingmap/19.zip ,
+CVUSA/streetview/panos.zip
+`
+ from [here](https://anu365-my.sharepoint.com/:f:/g/personal/u6293587_anu_edu_au/EuOBUDUQNClJvCpQ8bD1hnoBjdRBWxsHOVp946YVahiMGg?e=F4yRAC), the project page is [Sat2StrPanoramaSynthesis](https://github.com/shiyujiao/Sat2StrPanoramaSynthesis).
+
+Then download the sky mask from [here](https://drive.google.com/drive/folders/1pfzwONg4P-Mzvxvzb2HoCpuZFynElPCk?usp=sharing)
+
+Last，the users should organize the dataset just like:
+```
+├dataset
+├── CVACT
+│   ├── streetview
+│   ├── satview_correct
+│   ├── pano_sky_mask
+│   ├── ACT_data.mat
+└── CVUSA
+│   ├── bingmap
+│   │   ├── 19
+│   └── streetview
+│   │   ├── panos
+│   ├── sky_mask
+│   ├── splits
+```
+
+Tip： The sky masks are processed with [Trans4PASS](https://github.com/jamycheung/Trans4PASS).

demo_img/runall.sh

@@ -0,0 +1,30 @@
+# for case in `ls -d demo_img/case*`
+for case_id in 1 2 3 4
+do
+    case=demo_img/case$case_id
+    echo $case
+    python test.py --yaml=sat2density_cvact \
+    --test_ckpt_path=2u87bj8w \
+    --task=test_vid \
+    --demo_img=$case/satview-input.png  \
+    --sty_img=$case/groundview.image.png  \
+    --save_dir=results/$case
+    # ffmpeg -framerate 10 -i results/$case/rendered_images+depths/%5d.png results/$case/render.gif
+    ffmpeg -framerate 10 -i results/$case/rendered_images+depths/%5d.png -vf "palettegen" results/$case-palette.png
+    ffmpeg -framerate 10 -i results/$case/rendered_images+depths/%5d.png -i results/$case-palette.png -filter_complex "paletteuse" results/$case/render.gif
+
+    ffmpeg -framerate 10 -i results/$case/sat_images/%5d.png -vf "palettegen" results/$case-palette.png
+    ffmpeg -framerate 10 -i results/$case/sat_images/%5d.png -i results/$case-palette.png -filter_complex "paletteuse" results/$case/sat.gif
+    # ffmpeg -framerate 10 -i results/$case/sat_images/%5d.png results/$case/sat.gif
+done
+
+# for case in `ls -d demo_img/case*`
+for case_id in 1 2 3 4
+do
+    case=demo_img/case$case_id
+    sat_gif=results/$case/sat.gif
+    render_gif=results/$case/render.gif
+    # echo $sat_gif
+    cp $sat_gif docs/figures/demo/case$case_id.sat.gif
+    cp $render_gif docs/figures/demo/case$case_id.render.gif
+done

imaginaire/__init__.py

@@ -0,0 +1,4 @@
+# Copyright (C) 2021 NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
+#
+# This work is made available under the Nvidia Source Code License-NC.
+# To view a copy of this license, check out LICENSE.md