Original: AO-Net: Efficient Neural Network for Ambient Occlusion
Revised: AO-Net: Efficient Neural Network for Ambient Occlusion

Original: A BSTRACT
Revised: A BSTRACT

Original: Screen space based ambient occlusion is widely applied in real-time 3D applications due to its high efficiency, however, it frequently exhibits artifacts including banding, blurring etc.
Revised: Screen space based ambient occlusion is widely applied in real-time 3D applications due to its high efficiency, however, it frequently exhibits artifacts including banding and blurring.

Original: In this paper, we propose AO-Net, a learning-based method for fast and high-quality ambient occlusion generation.
Revised: In this paper, we propose AO-Net, a learning-based method for fast and high-quality ambient occlusion generation.

Original: Our neural network is built upon kernel prediction-based architecture with careful input screen space feature selection, leading to a light-weight and compact solution.
Revised: Our neural network is built upon kernel prediction-based architecture with careful input screen space feature selection, leading to a light-weight and compact solution.

Original: Experiment results indicate that our approach can achieve visual quality to a level in comparable with ray-traced solutions, meanwhile maintaining real-time performance.
Revised: Experiment results indicate that our approach can achieve visual quality to a level in comparable with ray-traced solutions, meanwhile maintaining real-time performance.

Original: In addition, our method can be easily integrated into existing rendering pipelines and shows robustness for unseen scenes.
Revised: In addition, our method can be easily integrated into existing rendering pipelines and shows robustness for unseen scenes.

Original: 1 I NTRODUCTION
Revised: 1 I NTRODUCTION

Original: Ambient occlusion(AO) plays a crucial role in realistic rendering applications and is a widely-adopted approximation to global illumination.
Revised: Ambient occlusion(AO) plays a crucial role in realistic rendering applications and is a widely-adopted approximation to global illumination.

Original: It can greatly enhance viewers’ 3D perception by darkening locations via adjusting obscurance coefficients [1].
Revised: It can greatly enhance viewers’ 3D perception by darkening locations via adjusting obscurance coefficients [1].

Original: As illustrated in
Revised: As illustrated in

Original: Fig.
Revised: Fig.

Original: 1, the AO value of position p with normal n can be calculated by integrating the visibility function over the hemisphere Ω .
Revised: 1, the AO value of position p with normal n can be calculated by integrating the visibility function over the hemisphere Ω .

Original: One key observation is that the AO values are view independent and only relate to scene geometry.
Revised: One key observation is that the AO values are view independent and only relate to scene geometry.

Original: High-quality AO can be obtained via hardware-assisted ray-traced techniques [2], which are usually time-consuming and difficult to use in real-time applications due to the limited power budget.
Revised: High-quality AO can be obtained via hardware-assisted ray-traced techniques [2], which are usually time-consuming and difficult to use in real-time applications due to the limited power budget.

Original: In order to achieve high performance, screen-based solutions that solely rely on depth or/and normal buffer as input are widely adopted.
Revised: In order to achieve high performance, screen-based solutions that solely rely on depth or/and normal buffer as input are widely adopted.

Original: To name a few, Screen Space Directional Occlusion(SSDO) [3],
Revised: To name a few, Screen Space Directional Occlusion(SSDO) [3],

Original: Ground Truth-based Ambient Occlusion(GTAO)
Revised: Ground Truth-based Ambient Occlusion(GTAO)

Original: [4] and Horizon- based Ambient Occlusion (HBAO) [5] are all approaches of this kind.
Revised: [4] and Horizon- based Ambient Occlusion (HBAO) [5] are all approaches of this kind.

Original: These approximations, however, often lead to low visual quality due to the absence of global information and are prone to exhibiting artifacts such as blurring and banding.
Revised: These approximations, however, often lead to low visual quality due to the absence of global information and are prone to exhibiting artifacts such as blurring and banding.

Original: In recent years, important advances incorporating neural networks into computer graphics have been made, especially in tasks such as Monte Carlo imagery denoising [6,7], super sampling [8], and light field [9].
Revised: In recent years, important advances incorporating neural networks into computer graphics have been made, especially in tasks such as Monte Carlo imagery denoising [6,7], super sampling [8], and light field [9].

Original: Learning-based approaches [10,11] have shown potential to produce AO.
Revised: Learning-based approaches [10,11] have shown potential to produce AO.

Original: However, both quality and efficiency need to be improved to meet the demands of real-time rendering.
Revised: However, both quality and efficiency need to be improved to meet the demands of real-time rendering.

Original: In addition, the generalization and robustness of deep learning-based approaches have not been carefully studied.
Revised: In addition, the generalization and robustness of deep learning-based approaches have not been carefully studied.

Original: In our work, we propose a novel method for AO which aims at obtaining ray-traced quality and maintaining real-time performance.
Revised: In our work, we propose a novel method for AO which aims at obtaining ray-traced quality and maintaining real-time performance.

Original: Our framework generates AO by a deep learning network from screen space buffers, which can be easily obtained in a deferred rendering pipeline.
Revised: Our framework generates AO by a deep learning network from screen space buffers, which can be easily obtained in a deferred rendering pipeline.

Original: First, we investigate the sensitivity of screen space buffers with respect to the quality of AO and exclude redundant features that have little influence to keep the memory bandwidth as small as possible.
Revised: First, we investigate the sensitivity of screen space buffers with respect to the quality of AO and exclude redundant features that have little influence to keep the memory bandwidth as small as possible.

Original: Second, the artifacts, for instance, blurring and banding can be addressed via a kernel-prediction neural network.
Revised: Second, the artifacts, for instance, blurring and banding can be addressed via a kernel-prediction neural network.

Original: In the training phase, we use ray-traced AO as targets.
Revised: In the training phase, we use ray-traced AO as targets.

Original: In addition, we build up a prototype by integrating our trained model into a rasterization real-time rendering pipeline.
Revised: In addition, we build up a prototype by integrating our trained model into a rasterization real-time rendering pipeline.

Original: We validate our approach on various seen and unseen scenes.
Revised: We validate our approach on various seen and unseen scenes.

Original: To summarize, our work has made the following contributions:
Revised: To summarize, our work has made the following contributions:

Original: • A lightweight and compact network built upon kernel prediction-based architecture, which eliminates blurring and banding artifacts, and achieves ray-traced visual quality at real-time frame rates.
Revised: • A lightweight and compact network built upon kernel prediction-based architecture, which eliminates blurring and banding artifacts, and achieves ray-traced visual quality at real-time frame rates.

Original: • A careful input screen space features selection for AO tasks, which reduces redundancy and improves the network efficiency.
Revised: • A careful input screen space features selection for AO tasks, which reduces redundancy and improves the network efficiency.

Original: • An end-to-end AO solution that produces high-quality AO results by combining a deferred rendering pipeline.
Revised: • An end-to-end AO solution that produces high-quality AO results by combining a deferred rendering pipeline.

Original: 2 R ELATED W ORKS
Revised: 2 R ELATED W ORKS

Original: Ray-traced Ambient Occlusion Early methods used ray- tracing to generate AO [12–14] for static scenes, which more accurately accounted for ambient light.
Revised: Ray-traced Ambient Occlusion Early methods used ray- tracing to generate AO [12–14] for static scenes, which more accurately accounted for ambient light.

Original: These methods are used as preprocessing steps in real-time interactive applications, but are limited to static scenes and are slow to compute.
Revised: These methods are used as preprocessing steps in real-time interactive applications, but are limited to static scenes and are slow to compute.

Original: In order to support dynamic scenes, Kontkanen [15] et al. proposed a precomputed
Revised: In order to support dynamic scenes, Kontkanen [15] et al. proposed a precomputed

Original: AO field for rapid calculation of AO, which is further applied in approximating the AO of dynamic objects.
Revised: AO field for rapid calculation of AO, which is further applied in approximating the AO of dynamic objects.

Original: Bunnell [16] proposed a
Revised: Bunnell [16] proposed a

Original: GPU-based deformed surface method to deal with dynamic cases.
Revised: GPU-based deformed surface method to deal with dynamic cases.

Original: Christensen [17] extended this approach and used it to calculate diffuse global illumination.
Revised: Christensen [17] extended this approach and used it to calculate diffuse global illumination.

Original: All these methods are based on vari- ous surface discretization or ray-tracing.
Revised: All these methods are based on vari- ous surface discretization or ray-tracing.

Original: For AO method based on ray-tracing, it is of utmost importance to increase calculation speed.
Revised: For AO method based on ray-tracing, it is of utmost importance to increase calculation speed.

Original: [Table] Ambient Occlusion (SSAO) method was first introduced by Mittring [18] to avoid the huge amount of calculation and additional storage
Revised: Screen Space Ambient Occlusion Screen Space Ambient Occlusion (SSAO) method was first introduced by Mittring [18] to avoid the huge amount of calculation and additional storage

Original: over- head of ray-traced AO methods.
Revised: over- head of ray-traced AO methods.

Original: By sampling from the depth buffer in the view sphere space, SSAO calculates the number of occlusion points within the depth surface, and estimates the occlusion factor.
Revised: By sampling from the depth buffer in the view sphere space, SSAO calculates the number of occlusion points within the depth surface, and estimates the occlusion factor.

Original: It’s widely used in modern 3D applications, but due to the large difference in depth of neighboring pixels in screen space, it sometimes causes artifacts.
Revised: It’s widely used in modern 3D applications, but due to the large difference in depth of neighboring pixels in screen space, it some- times causes artifacts.

Original: Bavoi et al.
Revised: Bavoi et al.

Original: [5] introduced Horizon Based Ambient Occlusion (HBAO).
Revised: [5] introduced Horizon Based Ambient Occlusion (HBAO).

Original: The rays travel along the depth buffer, and HBAO is calculated based on the depth difference.
Revised: The rays travel along the depth buffer, and HBAO is calculated based on the depth difference.

Original: However, this method only considers the visible points in the current tracking perspective, which will result in incorrect or missing shadows.
Revised: However, this method only considers the visible points in the current tracking perspective, which will result in incorrect or missing shadows.

Original: An important strategy was adopted by Vardis et al.
Revised: An important strategy was adopted by Vardis et al.

Original: [19] to identify key areas of sampling through multi-view joint optimization.
Revised: [19] to identify key areas of sampling through multi-view joint optimization.

Original: Jimenez et al.
Revised: Jimenez et al.

Original: [4] propose GTAO, to speed up calculations and improve accuracy through a new form of integration.
Revised: [4] propose GTAO, to speed up calculations and improve accu- racy through a new form of integration.

Original: A common problem with the screen space AO methods described above is that the thresholds or experience sampling parameters must be carefully chosen to avoid artifacts.
Revised: A common problem with the screen space AO methods described above is that the thresholds or experience sampling parameters must be carefully chosen to avoid artifacts.

Original: Deep Learning for Ambient Occlusion In recent years, with the successful application of neural network, it is used to solve rendering problems.
Revised: Deep Learning for Ambient Occlusion In recent years, with the successful application of neural network, it is used to solve rendering problems.

Original: To solve the AO problem, Holden et al.
Revised: To solve the AO problem, Holden et al.

Original: [20] first introduced a neural network called NNAO that uses a neural network to predict an approximation of Ambient occlusion.
Revised: [20] first introduced a neural network called NNAO that uses a neural network to predict an approximation of Ambient occlusion.

Original: NNAO network is merely a 4-layer MLP, so the learning ability of NNAO is poor, and high-quality AO results cannot be produced.
Revised: NNAO network is merely a 4-layer MLP, so the learning ability of NNAO is poor, and high-quality AO results cannot be produced.

Original: Nalbach et al.
Revised: Nalbach et al.

Original: [10] introduced Deepshading, a convolutional neural network, which learns the mapping from the view-space position and normal to ray- traced AO image.
Revised: [10] introduced Deepshading, a convolutional neural network, which learns the mapping from the view-space position and normal to ray- traced AO image.

Original: However, this method has low efficiency and poor generalization performance.
Revised: However, this method has low efficiency and poor generalization performance.

Original: Zhang et al.
Revised: Zhang et al.

Original: [11] proposed DeepAO, which reduces the number of network layers and parameters based on U-Net networks, further improving rendering efficiency.
Revised: [11] proposed DeepAO, which reduces the number of network layers and parameters based on U-Net networks, further improving rendering efficiency.

Original: But it sacrifices rendering quality to a large extent so the rendering quality still needs to be improved.
Revised: But it sacrifices rendering quality to a large extent so the rendering quality still needs to be improved.

Original: Kernel Prediction Kernel Prediction has demonstrated signifi- cant effectiveness in the field of denoising in Monte Carlo rendering to remove noise such as indirect lighting, soft shadows, etc.
Revised: Kernel Prediction Kernel Prediction has demonstrated signifi- cant effectiveness in the field of denoising in Monte Carlo rendering to remove noise such as indirect lighting, soft shadows, etc.

Original: Bako et al.
Revised: Bako et al.

Original: [21] used convolutional neural networks to predict the filter kernel and yielded more significant results in denoising effects.
Revised: [21] used convolutional neural networks to predict the filter kernel and yielded more significant results in denoising effects.

Original: Vo- gels et al.
Revised: Vo- gels et al.

Original: [7] use multiple deep residual networks to predict multiple filter kernels at different scales.
Revised: [7] use multiple deep residual networks to predict multiple filter kernels at different scales.

Original: We first introduce kernel prediction module into the field of AO generation, where the features extracted by U-Net network are used to generate kernel weights for AO.
Revised: We first apply kernel prediction module to the field of AO generation, where the features extracted by U-Net network are used to generate kernel weights for AO.

Original: We also compare our method with original U-Net, which confirms that kernel prediction in AO generation tasks is essential for removing noise from sparsely sampled images, leading to further improvement in rendering effects.
Revised: We also compare our method with original U-Net, which confirms that kernel prediction in AO generation tasks is essential for removing noise from sparsely sampled images, leading to further improvement in rendering effects.

Original: 3 O UR M ETHODS
Revised: 3 O UR M ETHODS

Original: Our approach is illustrated in Fig.4: given G-buffers from a deferred rendering pipeline at a certain view, we seek to generate its AO pass.
Revised: Our approach is illustrated in Fig.4: given G-buffers from a deferred rendering pipeline at a certain view, we seek to generate its AO pass.

Original: Consequently, we first describe the screen space feature selection in Sec.3.1 via a sensitivity study, and then discuss the network architecture in Sec.3.2 and Sec.3.3.
Revised: Consequently, we first describe the screen space feature selection in Sec.3.1 via a sensitivity study, and then discuss the network architecture in Sec.3.2 and Sec.3.3.

Original: Finally, we discuss the results on various scenes in Sec.4.2 before concluding in Sec.5.
Revised: Finally, we discuss the results on various scenes in Sec.4.2 before concluding in Sec.5.

Original: 3.1 Feature Selection
Revised: 3.1 Feature Selection

Original: We use supervised learning to train our neural network.
Revised: We use supervised learning to train our neural network.

Original: The training pairs consist of features generated by rasterization and reference AO generated by ray-tracing.
Revised: The training pairs consist of features generated by rasterization and reference AO generated by ray-tracing.

Original: Previous learning-based AO generation methods [10,11] used view-space normal and depth in G-buffer as input to the network, which is usually time-consuming for network inference and thus difficult for real-time applications.
Revised: Previous learning-based AO generation methods [10,11] used view-space normal and depth in G-buffer as input to the network, which is usually time-consuming for network inference and thus difficult for real-time applications.

Original: As mentioned in Sec.1, the AO values only relate to the neighboring geometry in-
Revised: As mentioned in Sec.1, the AO values only relate to the neighboring geometry in-

Original: formation(normal and depth).
Revised: formation(normal and depth).

Original: On the one hand, it is also interesting to note that normal and depth are indeed correlated:the normal is the cross product of gradients of the depth buffer in smooth regions.
Revised: On the one hand, it is also interesting to note that normal and depth are indeed correlated: normal is the cross product of gradients of the depth buffer in smooth regions.

Original: On the other hand, the visible view-space normal always points to the camera.
Revised: On the other hand, the visible view-space normal always points to the camera.

Original: In conclusion, we believe  the used features can be further reduced.
Revised: So we believe that the used features can be further reduced.

Original: Our feature selection method is inspired by Neural Shadow Map- ping [22], which uses sensitivity analysis to simplify the neural network input by quantifying the importance of each input channel.
Revised: Our feature selection method is inspired by Neural Shadow Map- ping [22], which uses sensitivity analysis to simplify the neural network input by quantifying the importance of each input channel.

Original: Sensitivity analysis is used to evaluate changes in network output caused by small perturbations in the input channel.
Revised: Sensitivity analysis is used to evaluate changes in network output caused by small perturbations in the input channel.

Original: Intuitively, if the channel contributes more to interpreting changes in output, the sensitivity is higher.
Revised: Intuitively, if the channel contributes more to interpreting changes in output, the sensitivity is higher.

Original: The absolute sensitivity S i of the input channel x i is expressed by
Revised: The absolute sensitivity S i of the input channel x i is expressed by

Original: [Equation]
Revised: [Equation]

Original: where φ is our network, and ξ i is randomly generated Gaussian noise with the mean value of 0 and the standard deviation of σ i , which adds random perturbations to the i th channel.
Revised: where φ is our network, and ξ i is randomly generated Gaussian noise with the mean value of 0 and the standard deviation of σ i , which adds random perturbations to the i th channel.

Original: σ i is the standard deviation corresponding to the i th channel, which is estimated empirically by aggregating all pixels in the dataset of each channel.
Revised: σ i is the standard deviation corresponding to the i th channel, which is estimated empirically by aggregating all pixels in the dataset of each channel.

Original: Finally, we calculate the relative sensitivity s i
Revised: Finally, we calculate the relative sensitivity s i

Original: =
Revised: =

Original: S i / Σ
Revised: S i / Σ

Original: i S i to compare the sensitivity of the different input channels.
Revised: i S i to compare the sensitivity of the different input channels.

Original: We systematically evaluate and select all channels including depth and three dimensions of normal by calculating the sensitivity of each channel, as shown in Fig.
Revised: We systematically evaluate and select all channels including depth and three dimensions of normal by calculating the sensitivity of each channel, as shown in Fig.

Original: In order to ensure the consistency of the perturbation, we select normalized depth for the experiment, so that the scale of depth is the same as the scale of normal.
Revised: In order to ensure the consistency of the perturbation, we select normalized depth for the experiment, so that the scale of depth is the same as the scale of normal.

Original: It shows that the perturbation of normal has a great influence on the output results, indicating channels of normal play a vital role in the network.
Revised: It shows that the perturbation of normal has a great influence on the output results, indicating channels of normal play a vital role in the network.

Original: In contrast, the depth channel has little effect on network results, less than 10%.
Revised: In contrast, the depth channel has little effect on network results, less than 10%.

Original: Subsequent ablation experiments demonstrated the effectiveness of our feature selection, as shown in Table 3.
Revised: Subsequent ablation experiments demonstrated the effectiveness of our feature selection, as shown in Table 3.

Original: Moreover, depth varies greatly in different scenes, making it difficult for the network to converge.
Revised: Moreover, depth varies greatly in different scenes, making it difficult for the network to converge.

Original: Therefore, we choose normal as the input to our network while the output is compared against ray-traced AO as the target during training.
Revised: Therefore, we choose normal as the input to our network while the output is compared against ray-traced AO as the target during training.

Original: 3.2 Patch-based U-Net
Revised: 3.2 Patch-based U-Net

Original: Vanilla U-Net yields a network with low performance ( > 20ms), which can not be adapted to the real-time graphics pipeline.
Revised: Vanilla U-Net yields a network with low performance ( > 20ms), which can not be adapted to the real-time graphics pipeline.

Original: There- fore, the primary goal of our approach is to optimize the network structure to improve efficiency with minimal impact on quality.
Revised: Therefore, the primary goal of our approach is to optimize the network structure to improve efficiency with minimal impact on quality.

Original: We first analyze the performance of each layer of the network.
Revised: We first analyze the performance of each layer of the network.

Original: Fig.
Revised: Fig.

Original: 3 shows the time spent at each layer of the network, which takes a total of 20.56ms.
Revised: 3 shows the time spent at each layer of the network, which takes a total of 20.56ms.

Original: We see that the first layer (including encoder-decoder) takes 11.61ms, which takes the longest time and accounts for more than 50% of the time for the entire network.
Revised: We see that the first layer (including encoder-decoder) takes 11.61ms, which takes the longest time and accounts for more than 50% of the time for the entire network.

Original: In addition, we observe that as we move from the inner layer of U-Net to the outer layer, the number of channels is reduced by half and the resolution is doubled.
Revised: In addition, we observe that as we move from the inner layer of U-Net to the outer layer, the number of channels is reduced by half and the resolution is doubled.

Original: In practice, as the resolution increases, the run-time of the layer increases significantly.
Revised: In practice, as the resolution increases, the run-time of the layer increases significantly.

Original: Therefore, we believe that the key solution to reducing network run-time is to optimize the first layer, which is the most time-consuming.
Revised: Therefore, we believe that the key solution to reducing network run-time is to optimize the first layer, which is the most time-consuming.

Original: In addition, reducing the resolution of each layer of the network is critical to improving network performance.
Revised: In addition, reducing the resolution of each layer of the network is critical to improving network performance.

Original: A common approach is to replace the first layer with a simple downsampler and upsampler in the U-Net.
Revised: A common approach is to replace the first layer with a simple downsampler and upsampler in the U-Net.

Original: However, it can cause some crucial information be lost in the input and can produce less clear output or even artifacts.
Revised: However, it can cause some crucial information be lost in the input and can produce less clear output or even artifacts.

Original: Therefore, a patch-based network is proposed to optimize net- work performance.
Revised: Therefore, a patch-based network is proposed to optimize net- work performance.

Original: We optimize the first layer (including encoderdecoder) which is the most time-consuming.
Revised: We optimize the first layer (including encoderdecoder) which is the most time-consuming.

Original: On the encoder side, we use a patch partition to flatten each input 4x4 square of pixels as a patch to 16 separate channels and send the (h/4 × w/4) patches directly to the second layer.
Revised: On the encoder side, we use a patch partition to flatten each input 4x4 square of pixels as a patch to 16 separate channels and send the (h/4 × w/4) patches directly to the second layer.

Original: Therefore, the input buffer size is converted from (h × w × c) to (h/4 × w/4 × 16c).
Revised: Therefore, the input buffer size is converted from (h × w × c) to (h/4 × w/4 × 16c).

Original: We do the opposite on the decoder side: rearranging the 16 output channels to 4x4 pixels through patch merging.
Revised: We do the opposite on the decoder side: rearranging the 16 output channels to 4x4 pixels through patch merging.

Original: Patch-based networks replace the most time- consuming first layer, which greatly improves network speed.
Revised: Patch-based networks replace the most time- consuming first layer, which greatly improves network speed.

Original: After our optimization, the run-time of the first layer network is reduced from 11.64 ms to 0.32 ms.
Revised: After our optimization, the run-time of the first layer network is reduced from 11.64 ms to 0.32 ms.

Original: The patch-based method optimizes the network performance to 3.60 ms, which is only 18% of the initial time.
Revised: The patch-based method optimizes the network performance to 3.60 ms, which is only 18% of the initial time.

Original: Fig.
Revised: Fig.

Original: 5 shows the speed and quality comparison before and after network optimization.
Revised: 5 shows the speed and quality comparison before and after network optimization.

Original: It shows that our network has a significant increase in network speed with minimal impact on quality.
Revised: It shows that our network has a significant increase in network speed with minimal impact on quality.

Original: We optimized the network structure so that the network has high performance ( < 4ms, resolution: 720 ×1280), which allows the network to be easily integrated into the rendering pipeline.
Revised: We optimized the network structure so that the network has high performance ( < 4ms, resolution: 720 ×1280) and requires little memory ( < 1.5MBs), allowing the network to be easily integrated into the rendering pipeline.

Original: However, the large compression of time also leads to a slight loss of quality, reducing the network’s ability to preserve details, so we introduce a kernel prediction module to improve the quality and detail of the generated AO.
Revised: However, the large compression of time also leads to a slight loss of quality, reducing the network’s ability to preserve details, so we incorporate a kernel prediction module to improve the quality and detail of the generated AO.

Original: 3.3 Kernel Prediction Module
Revised: 3.3 Kernel Prediction Module

Original: Furtherly, we introduce the kernel prediction module, which has demonstrated significant effectiveness in the field of denoising [7, 21], to alleviate the above-mentioned problems and enable our AO- Net generating fine-level details.
Revised: Furtherly, we incorporate the kernel prediction module, which has demonstrated significant effectiveness in the field of denoising [7, 21], to alleviate the above-mentioned problems and enable our AO- Net generating fine-level details.

Original: Kernel prediction uses neural networks to estimate the locally weighted kernel of each pixel from neighboring pixels.
Revised: Kernel prediction uses neural networks to estimate the locally weighted kernel of each pixel from neighboring pixels.

Original: By concatenating U-Net in series with kernel prediction module, the features extracted by our patch-based U-Net are directly used to generate weighted kernels.
Revised: By concatenating U-Net in series with kernel prediction module, the features extracted by our patch-based U-Net are directly used to generate weighted kernels.

Original: We use the predicted kernel weight to locally weigh the input normal, and obtain the output AO.
Revised: We use the predicted kernel weight to locally weigh the input normal, and obtain the output AO.

Original: The output of kernel prediction has K 2 N channels, K is the size of linear filters at each pixel and N is the number of input channels per pixel.
Revised: The output of kernel prediction has K 2 N channels, K is the size of linear filters at each pixel and N is the number of input channels per pixel.

Original: The value Y p at each pixel p can be expressed as
Revised: The value Y p at each pixel p can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where V p ( X i ) is the K × K neighborhood of pixel p in image X of the i -channel, and f is its corresponding kernel.
Revised: where V p ( X i ) is the K × K neighborhood of pixel p in image X of the i -channel, and f is its corresponding kernel.

Original: <,> means a dot product then calculate average.
Revised: <,> means a dot product then calculate average.

Original: Our approach combines the advantages of U-Net and kernel pre- diction.
Revised: Our approach combines the advantages of U-Net and kernel pre- diction.

Original: It preserves more local details meanwhile achieving a large receptive field, resulting in a output with sharper contour.
Revised: It preserves more local details meanwhile achieving a large receptive field, resulting in a output with sharper contour.

Original: In addi- tion, it ensures that the final output estimate always lies within a convex hull of the respective neighborhoods of the input image.
Revised: In addi- tion, it ensures that the final output estimate always lies within a convex hull of the respective neighborhoods of the input image.

Original: This greatly reduces the search space of the output value, thus avoiding the artifacts generated by U-Net direct prediction.
Revised: This greatly reduces the search space of the output value, thus avoiding the artifacts generated by U-Net direct prediction.

Original: (e.g., color shifts).
Revised: (e.g., color shifts).

Original: Similar to Bako et al.
Revised: Similar to Bako et al.

Original: [21], each filter uses the decoder block of
Revised: [21], each filter uses the decoder block of

Original: U-Net to predict a 4 × 4 kernel that is applied to the input image.
Revised: U-Net to predict a 4 × 4 kernel that is applied to the input image.

Original: We compare our kernel prediction based method with U-Net direct prediction which is shown in Fig.
Revised: We compare our kernel prediction based method with U-Net direct prediction which is shown in Fig.

Original: The results show that the AO directly predicted by U-Net retains the blurring and artifacts.
Revised: The results show that the AO directly predicted by U-Net retains the blurring and artifacts.

Original: After the introduction of kernel prediction, the network preserves more local details, which further improves the rendering quality.
Revised: After the introduction of kernel prediction, the network preserves more local details, which further improves the rendering quality.

Original: 3.4 Loss Function
Revised: 3.4 Loss Function

Original: In this section, we detail loss functions that are employed in the training process as below.
Revised: In this section, we detail loss functions that are employed in the training process as below.

Original: Content loss In previous AO generation methods, SSIM [23] loss function is commonly used to minimize the error between the predicted AO and the ground truth.
Revised: Content loss In previous AO generation methods, SSIM [23] loss function is commonly used to minimize the error between the predicted AO and the ground truth.

Original: Although large SSIM weights can speed up the rate of convergence, they also cause excessive smoothness of rendering results.
Revised: Although large SSIM weights can speed up the rate of convergence, they also cause excessive smoothness of rendering results.

Original: To avoid this, we set 0.5 as its weight:
Revised: To avoid this, we set 0.5 as its weight:

Original: [Equation]
Revised: [Equation]

Original: Perceptual loss Content loss compares differences in spatial similarity, but it is less sensitive to detail.
Revised: Perceptual loss Content loss compares differences in spatial similarity, but it is less sensitive to detail.

Original: We use perceptual loss in
Revised: We use perceptual loss in

Original: Equation 4 to produce sharper edges and improve the image detail of the rendering result.
Revised: Equation 4 to produce sharper edges and improve the image detail of the rendering result.

Original: We introduce VGG-19 [24] perceptual loss function to minimize the distance between the actual output and generated value in the feature space,  we use the first several layers feature map of VGG-19 network to obtain better visual quality.
Revised: VGG-19 [24] perceptual loss function can minimize the distance between the actual output and generated value in the feature space, so we use the first several layers feature map of VGG-19 network to obtain better visual quality.

Original: The perceptual loss is defined as
Revised: The perceptual loss is defined as

Original: [Equation]
Revised: [Equation]

Original: where Φ denotes activation of the i -th layer in the VGG-19 network.
Revised: where Φ denotes activation of the i -th layer in the VGG-19 network.

Original: In summary, the overall loss of AO-Net is defined as
Revised: In summary, the overall loss of AO-Net is defined as

Original: [Equation]
Revised: [Equation]

Original: 4 E XPERIMENT 4.1 Experimental Settings
Revised: 4 E XPERIMENT 4.1 Experimental Settings

Original: Datasets a) Our dataset: consists of 3996 pairs of deferred shading G-buffer data and corresponding reference images in a resolution of 720 × 1280 px .
Revised: Datasets a) Our dataset: consists of 3996 pairs of deferred shading G-buffer data and corresponding reference images in a resolution of 720 × 1280 px .

Original: To create the dataset, 37 scenes are utilized to generate the pair annotated data.
Revised: To create the dataset, 37 scenes are utilized to generate the pair annotated data, the scenes include food, animal, furniture, building, etc.

Original: In these 3996 pairs, we use 3200 pairs to train the network, 398 for validation, and the remaining 398 for testing.
Revised: In these 3996 pairs, we use 3200 pairs to train the network, 398 for validation, and the remainingfor testing.

Original: The training and validation images share the same set of 29 scenes, while the test images come from 8 different scenes not used for training or validation.
Revised: The training and validation images share the same set of 29 scenes, while the test images come from 8 different scenes not used for training or validation.

Original: b) DeepAO dataset: provided by Zhang et al.
Revised: b) DeepAO dataset: provided by Zhang et al.

Original: [11], which includes 27 scenes and 105,000 pairs of deferred shading G-buffer data, with corresponding reference AO results.
Revised: [11], which includes 17 scenes and 42,000 pairs of deferred shading G-buffer data, with corresponding reference AO results.

Original: 90,000 pairs of images are used to train the network while 10,000 pairs are used for validation, and the remaining are used for testing.
Revised: 32,000 pairs of images are used to train the network while 5,000 pairs are used for validation, and the remaining are used for testing.

Revised: The training and validation images share the same set of 14 scenes, while the test images come from the remaining 3 scenes.

Revised: The training and test scenes are disjoint.

Original: Baselines Using ray-traced AO as a reference, we compare the rendering effect and runtimes with the current state-of-the-art methods, which proves the effectiveness of AO-Net.
Revised: Baselines Using ray-traced AO as a reference, we compare the rendering effect and runtimes with the current state-of-the-art methods, which proves the effectiveness of AO-Net.

Original: We compare against a) state-of-the-art neural network AO methods: Deepshading [10] and DeepAO [11].
Revised: We compare against a) state-of-the-art neural network AO methods: Deepshading [10] and DeepAO [11].

Original: b) classic screen space AO methods: GTAO [4] generated with eevee in Blender.
Revised: b) classic screen space AO methods: GTAO [4] generated with eevee in Blender.

Original: Implementation details We use Pytorch framework in imple- mentation and employ one NVIDIA Tesla v100 GPU for both training and inference.
Revised: Implementation details We use Pytorch framework in imple- mentation and employ one NVIDIA Tesla v100 GPU for both training and inference.

Original: We train our model using Adam optimizer with an initial learning rate of 1e-3.
Revised: We train our model using Adam optimizer with an initial learning rate of 1e-3.

Original: During U-Net upsampling, we use bi-linear interpolation instead of expensive transpose convolutions to further speed up inference.
Revised: During U-Net upsampling, we use bi-linear interpolation instead of expensive transpose convolutions to further speed up inference.

Original: In addition, we use Average-pool instead of Max-pool to improve temporal stability.
Revised: In addition, we use Average-pool instead of Max-pool to improve temporal stability.

Original: The loss weights in Equation 5 are set to λ 1 = 0 .
Revised: The loss weights in Equation 5 are set to λ 1 = 0 .

Original: 06, λ 2 = 0 .
Revised: 06, λ 2 = 0 .

Original: The mini-batch size for all scenes is set at 16 in training.
Revised: The mini-batch size for all scenes is set at 16 in training.

Original: 4.2 Result
Revised: 4.2 Result

Original: Quantitative Comparisons
Revised: Quantitative Comparisons

Original: The statistical results of our exper- iment are shown in Table 1.
Revised: The statistical results of our exper- iment are shown in Table 1.

Original: We use SSIM and PSNR to evaluate the rendering quality and mark the best results in bold in the table. 
Revised: We use SSIM and PSNR to evaluate the rendering quality and mark the best results in bold in the table. ” [Table]

Revised: ” means that the running time of GTAO has not been measured, because it is difficult to measure the running time of this process in Blender.

Original: It can be seen that our method outperforms other comparison methods on quality, and is superior to all state-of-the-art neural network AO methods on speed.
Revised: It can be seen that our method outperforms other comparison methods on quality, and is superior to all state-of-the-art neural network AO methods on speed.

Original: Qualitative Comparisons We randomly selected several scenes to compare the visualization results of various algorithms, as shown in Fig.
Revised: Qualitative Comparisons We randomly selected several scenes to compare the visualization results of various algorithms, as shown in Fig.

Original: We can observe that our method is superior to other advanced algorithms in overall visual effect and predicts AO results  closest to the ray-traced level.
Revised: We can observe that our method is superior to other advanced algorithms in overall visual effect and predicts AO results [Table] closest to the ray-traced level.

Original: In addition, our method performs best in occlusion details along object edges and at corners, as shown in
Revised: In addition, our method performs best in occlusion details along object edges and at corners, as shown in

Original: Fig.
Revised: Fig.

Original: We demonstrate that our approach can lead to more accurate generation and better image detail.
Revised: We demonstrate that our approach can lead to more accurate generation and better image detail.

Original: 4.2.1 Run times at different resolutions
Revised: 4.2.1 Run times at different resolutions

Original: In addition, we tested the average run time at different resolutions.
Revised: In addition, we tested the average run time at different resolutions.

Original: We mark the best performance in Table 2 in bold.
Revised: We mark the best performance in Table 2 in bold.

Original: It can be seen that our method is superior to other methods on speed, especially at high-resolution input.
Revised: It can be seen that our method is superior to other methods on speed, especially at high-resolution input.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4.2.2 Ablation Study
Revised: 4.2.2 Ablation Study

Original: We first analyze the validity of our feature selection.
Revised: We first analyze the validity of our feature selection.

Original: Next, we ablate the impact of patch-based optimization and kernel predict module on reference time and quality.
Revised: Next, we ablate the impact of patch-based optimization and kernel predict module on reference time and quality.

Original: Ablation study on input features To validate the effectiveness of our feature selection, we compare three different kinds of input features and report metrics averaged over our dataset in Table 3.
Revised: Ablation study on input features To validate the effectiveness of our feature selection, we compare three different kinds of input features and report metrics averaged over our dataset in Table 3.

Original: We observe that choosing depth as input feature has little impact on the quality of the rendered image, but the quality can be degraded if the depth  is not normalized or is not normalized correctly.
Revised: We observed that with normal input alone we can get high-quality AO while the addition of depth has little impact on the quality of the rendered image. As mentioned in Sec.3.1, normal is the cross product of the gradient of the depth buffer in the smooth region, so normal can be seen as knowledge distilled from depth, and is sufficient to express the geometric information required for AO generation. More importantly, because the depth of different scenes varies greatly, the quality can degrade greatly if depth is not normalized or is not normalized correctly, thus we consider the instability of depth information to play a negative role in AO generation tasks.

Original: In addition, only normal input improves the network efficiency and accelerates the network convergence speed in the experiment.
Revised: In addition, only normal input improves the network efficiency and accelerates the network convergence speed in the experiment.

Revised: [Table]

Original: Ablation study on Network Architecture We designed experi- ments to verify the validity of our proposed patch-based optimization scheme and kernel prediction module.
Revised: Ablation study on Network Architecture We designed experi- ments to verify the validity of our proposed patch-based optimization scheme and kernel prediction module.

Original: As shown in Table 4, Base works as a baseline which use original U-Net to predict rendering results.
Revised: As shown in Table [Table] Base works as a baseline which use original U-Net to predict rendering results.

Original: We train the model Base+Patch that includes the patchbased optimization above the model Base .
Revised: We train the model Base+Patch that includes the patch- based optimization above the model Base .

Original: Compared with Base ,[Table] Base+Patch reduce the time consumption to 18%, only 3.60ms.
Revised: Compared with Base , Base+Patch reduce the time consumption to 18%, only 3.60ms.

Original: In addition, kernel prediction module can further optimize the rendering quality without incurring a greater loss of efficiency.
Revised: In addition, kernel prediction module can further optimize the ren- dering quality without incurring a greater loss of efficiency.

Original: More importantly, it avoids blurring and artifacts from direct predictions, as shown in Fig.
Revised: More importantly, it avoids blurring and artifacts from direct predictions, as shown in Fig.

Original: Base+Kernel+patch combines kernel prediction with patch-based optimization to generate high-quality renderings results at a fast speed and achieve optimal results. 
Revised: Base+Kernel+patch combines kernel prediction with patch-based optimization to generate high-quality renderings results at a fast speed and achieve optimal results. [Table]

Original: 5 C ONCLUSION
Revised: 5 C ONCLUSION

Original: We present a learning-based method for AO generation.
Revised: We present a learning-based method for AO generation.

Original: The com- pact input of our solution is designed via an investigation of different screen space information’s influences on the AO quality.
Revised: The com- pact input of our solution is designed via an investigation of different screen space information’s influences on the AO quality.

Original: Further- more, the use of kernel prediction-based architecture further improves the visual quality to ray-traced level.
Revised: Further- more, the use of kernel prediction-based architecture further improves the visual quality to ray-traced level.

Original: We demonstrate that our method can reach real-time frame rates by integrating with a deferred rendering pipeline.
Revised: We demonstrate that our method can reach real-time frame rates by integrating with a deferred rendering pipeline.

Original: Various examples indicate that our methods are robust to unseen scenes.
Revised: Various examples indicate that our methods are robust to unseen scenes.

Revised: However, our work has the same limitations as all screen space solutions.

Revised: G-buffer does not contain complete information about the scene, which may lead to defects on screen boundaries or special perspectives.

Original: In the future, we would like to explore using global information[Table] to further improve the quality and other scenarios such as stereo rendering. 
Revised: In the future, we would like to explore using global information to further improve the quality and other scenarios such as stereo rendering. R