CUDA-Optimized real-time rendering of a Foveated Visual System
Abstract
The spatially-varying ﬁeld of the human visual system has recently received a resurgence of interest with the development of virtual reality (VR) and neural networks.
The computational demands of high resolution rendering desired for VR can be offset by savings in the periphery [16], while neural networks trained with foveated input have shown perceptual gains in i.i.d and o.o.d generalization [27, 6].
In this paper, we present a technique that exploits the CUDA GPU architecture to efﬁciently generate Gaussian-based foveated images at high deﬁnition (1920 px × 1080 px ) in real-time (165 Hz ) , with a larger number of pooling regions than previous Gaussian-based foveation algorithms by several orders of magnitude [10, 27], producing a smoothly foveated image that requires no further blending or stitching, and that can be well ﬁt for any contrast sensitivity function.
The approach described can be adapted from Gaussian blurring to any eccentricity-dependent image processing and our algorithm can meet demand for experimentation to evaluate the role of spatially-varying processing across biological and artiﬁcial agents, so that foveation can be added easily on top of existing systems rather than forcing their redesign (“emulated foveated renderer” [24]).
Altogether, this paper demonstrates how a GPU, with a CUDA block-wise architecture, can be employed for radially-variant rendering, with opportunities for more complex post-processing to ensure a metameric foveation scheme [35]. Our code can be found here [21].
Introduction
Understanding the computational and representational goal of the foveated nature of the human visual ﬁeld is a theme of ongoing research [25, 19, 11, 29, 5, 12].
Over the last decade, several works have suggested foveation as trying to achieve a computational goal tailored towards efﬁciency.
These works support the previous hypothesis by proposing simulated experiments where a foveated perceptual system can achieve the same asympototic performance as a non-foveated perceptual system without any beneﬁcial implications for a standard perceptual test such as object recognition (Akbas & Eckstein [2]).
On the other hand, another current of thought has recently tried to ﬁnd support for the representational goal of foveation (Rosenholtz [30]).
Works of this ﬂavor generally have shown that under speciﬁc perceptual tests, a foveated perceptual system can achieve an advantage over non-foveated perceptual systems by learning distinct representational signatures leading to robustness to scene occlusion (Deza & Konkle [6]), or reduction of false alarm rates in object detection (Pramod et al.
However, all previous works have approached this problem by studying a stationary visual agent that does not interact with the environment – a critical component suggested in developmental and perceptual psychology [13, 14, 34] that has slowly permeated into robotic systems [9, 32, 15, 18].
[Figure]
Thus anticipating the advent of robotic agents that may require a foveated visual system to explore and interact with the environment (Orhan et al.
[23]), – it will be critical to develop real-time efﬁcient foveated image transforms if we are to study the perceptual implications of such a set of transforms (Deza et al. [7], Fridman et al.
[8], Kaplanyan et al.
Without such rendering speed, it will be computationally intractable to train reinforcement learning (RL) agents that require millions of trials – thus image frames – for training; or to deploy spatially-varying transformations on actual robotic agents such as drones that rely on portable GPUs.
In this paper we will address this issue by developing a neuro-physiologically inspired CUDA optimized foveated visual transform – that is based on adaptive gaussian blurring [10], which emulates convergence of retinal ganglion cells (RGC) as a function of retinal eccentricity in the visual ﬁeld of a model system (Baden et al.
Thus as we move one step forward in the direction of high-speed foveated rendering, the two-fold goal of this algorithm is: 1) To enable computational experiments that evaluate the role of spatially-varying processing via gaussian-like computation [27] in the periphery (RGC convergence) across biological and artiﬁcial agents [26, 3, 23, 33]; 2) To aid in the computational rendering tractability of next-generation Augmented/Virtual Reality (AR/VR) systems and robots ( i.e. drones) that require adaptive low-energy powered systems [28, 37].
Proposed Approach: Block-wise Foveation
Block-wise foveation exploits the CUDA GPU architecture to efﬁciently generate foveated images with no subsampling and independently of the number of pooling regions.
Our approach can be summarized in Figure 1, where the ﬁnal image is assembled from square fragments, each of which has been blurred according to the distance from its midpoint to the point of ﬁxation (that can be displaced anywhere in the visual ﬁeld).
To achieve this, each fragment in the ﬁnal image is produced by a single CUDA block, with each block capable of using a different Gaussian ﬁlter to produce its uniquely assigned set of pixels.
Given that the complexity is independent of the ( n ) -number of
[Figure]
Distribution
[Figure]
The fragment patchwork has a total ofpooling regions.
The CUDA kernel executes in 1.65 ms on an NVIDIA GTX 1060 GPU.
pooling regions; n can be much larger than previous foveation algorithms that are Gaussian-based by several orders of magnitude [10, 27], producing a smoothly foveated image that requires no further blending or stitching, and that can be ﬁt for any contrast sensitivity function.
By eliminating the need to subsample and interpolate we produce a higher quality adaptive blur transform with virtually no artifacts.
Foveating in blocks takes advantage of the small amount of shared memory available and speciﬁc to each CUDA block.
Blurring is performed via separable convolution which is done in two stages - horizontal and vertical (See Appendix 5.1 – as a similar strategy is employed in the Gaussian Pyramid [4] and
Feature Congestion [31]).
The output of the ﬁrst stage – the intermediate result – can be stored within shared memory, where only the threads within the block can access it.
In doing so, the blocks work independently of each other to produce blurred fragments, making it simple to distribute blurring strengths across blocks.
Figure 2 shows the wide range of Gaussian ﬁlters used to mimic the spatial frequency cut-off rates of the human retina – our algorithm concurrently employs all ﬁlters to process the image in a single pass (Algorithm 1).
Algorithm
Experiments
To quantify the efﬁciency and precision of our foveated rendering scheme we devised a set ofexperiments that assess both the rendering precision in addition to the theoretical-practical compute time along several hyper-parameters that are systematically varied.
3.1 Localized Image Quality Assessment
The Structural Similarity Index Measure (SSIM) is typically used to evaluate the quality of a compressed image with reference to the original [38].
We employ this metric to assess the accuracy of our implementation, using ‘perfect’ (loss-less) pixel-wise foveation as reference.
Block-wise foveation scores lowest (0.971) near the ﬁxation point, as can be seen in Figure 3.
The spatial frequency function is steepest at the fovea (See Appendix 5.2.1), so relatively large differences arise at the edges of fragments, where the image is either over-blurred or under-blurred (blurring strength is set appropriate to the center of each fragment).
To mitigate this effect, we shift the tiling of fragments to center a single fragment on the ﬁxation point.
This ensures that the untouched foveal region is as small as possible, minimizing the very ﬁrst blurring step where the spatial frequency function is at its
The ring-like pattern of under-blurring and over-blurring can be seen to continue out into the periphery with decreasing disparity as the function gradient becomes shallower.
Although the edges of the fragments, where the steps occur, are imperceptible on still frames (Figure 1), they can become faintly visible with strong foveation and a large fragment size, as can be seen in the accompanying video [20].
Reducing the size of the fragments eliminates these artifacts.
Moreover, it may be the case that these artifacts would not be perceptible to a user looking at the ﬁxation point, though this would need to be evaluated through psychophysical experimentation.
3.2 Computation Time
The time to compute a foveated image depends on the size of the fragments used.
Smaller fragments are desirable because they result in more circular boundaries between pooling regions (Figure 4).
Moreover, the smaller the fragments, the more pooling regions can be ﬁt into the frame of the image.
This leads to a ﬁner discretization of the foveation curve and hence a better approximation of smooth retinal blurring.
Smaller fragments are, however, slower to compute.
Since it is necessary to perform work in the padding around a fragment during separable convolution, larger fragments “distribute” this extra work across more output pixels (Figure 4).
ORIGINAL
PYRAMID
BLOCKWISE
Foveation Strength (degree eccentricity of image corner)
E x e c u t i o n T i m e ( m s )
Block-wise vs Pyramid Foveation
Block-wise vs Pyramid Foveation 512x512blockwise
512x512pyramid 1024x1024blockwise 1024x1024pyramid 2048x2048blockwise 2048x2048pyramid
80 Foveation Strength (degree eccentricity of image corner)
E x e c u t i o n T i m e ( m s )
Fragment Size Comparison (Constant 1024x1024 Image Size) 8x8 Fragments 32x32 Fragments Pyramid 16x16 Fragments        80         fixation Image Size Comparison (Constant 32x32 Fragments)
In Figure 5, we compare the kernel execution time of our algorithm with the execution time of
Gaussian pyramid foveation implemented in OpenGL [17]. The Gaussian pyramid approach generates a mipmap which is then used to sample or interpolate output pixels (Appendix 5.1). Interpolation tends to produce blocky results, as compared to the smooth blur of our convolutional block-wise approach. The execution time of our algorithm rises with increasing foveation strength, which corresponds to larger Gaussian ﬁlters and hence more operations during convolution.
The performance of the Gaussian pyramid approach only varies with image size, as for a given image size mipmap generation and subsequent sampling require the same number of operations across all foveation strengths. While Figure 5 demonstrates timings for center-frame ﬁxation, execution time also varies with the location of the gaze in the image. Fixating on the edge of the frame introduces larger ﬁlters on the opposite edge, when compared to center ﬁxation, therefore increasing computation time.
On top of kernel execution, data transfer between the CPU and the GPU can signiﬁcantly affect the total time to generate a foveated image. Although the 1920 × 1080 px image in Figure 1 took just 1.65 ms to foveate, the time spent in transfer was 4.4 ms (using pinned memory), leading to a foveation rate of 165 Hz. In general, several steps can be taken to further reduce the total foveation time, if more complicated foveation structures are required (See Discussion 4). Execution time can be accelerated by using half-precision ﬂoating-point arithmetic, as well as more powerful GPUs.
The impact of data transfer on real-time performance can be mitigated by overlapping kernel execution with transfers to and from the GPU.
Figure 6 showcases other CUDA-Driven optimization strategies that are discussed more in depth in Appendix 5.5.
[Figure] [Figure]
The ﬂexibility of our block-wise algorithm means that it can be extended to non-radially-symmetric functions with no additional computational overhead.
Any image of the shape f ( x, y ) :
R × R → R can be used to describe blurring strength throughout the rendering frame, such that the resultant foveation can represent any desired retinal structure.
Figure 7 shows an example of how our rendering scheme can be adapted to a variety of animal retinas given their retinal ganglion cell (RGC) convergence maps.
These images allow us to probe the use of different retinal structures through reverse-engineering of perceptual architectures, where we can train different machine vision systems with asymmetric image processing schemes to evaluate how well they may learn a task such as object or scene classiﬁcation.
Naturally, each RGC map is usually coupled with its own set of image statistics such as colors, vantage point and structure.
Altogether, rendering stimuli as viewed from a biological agent may aid us in understanding the development of their visual system.
In return, given that our framework is real-time, it opens the door to test new theories of visual perception via Reinforcement Learning (RL) agents that may require millions of visual training frames that need not be spatially uniform, and that can later be deployed in robots (Figure 8).
Finally, the rise of several AR/VR systems that have human users (who have foveated visual systems) may beneﬁt from accelerated rendering systems as well [16]. Whether it is for purely entertainment purposes (video games), HumanMachine Interfaces such as tele-operation, or VR-driven psychophysics, the high demand of real-time foveated rendering procedures may become useful in the near future. Acknowledgments
Authors would like to thank MIT’s Center for Brains, Minds and Machines (CBMM), the National Science Foundation (NSF), MIT Quest for Intelligence and Lockheed Martin. Authors would also like to thank Ronald Alvarez for permissions to use the upper portion of Figure and members of the Poggio Lab for insightful comments during lab discussions
Appendix 5.1 Previous Work on Image Foveation
Many techniques have been proposed in the domain of computer graphics to introduce foveation into the rendering pipeline in order to reduce computational cost.
Most similar in design to our algorithm is Variable Rate Shading (VRS) - an NVIDIA Turing GPU feature that tiles the image frame and adapts shading rate from tile to tile to increase rendering performance versus individual pixel shading [22].
This work differs from ours by targeting the shading step; our algorithm is proposed as a post-process and applied to the existing full-resolution image to enable experimentation with foveated visual systems under the requirement of real-time performance.
Previous work on image foveation as a post-process has employed Gaussian pyramids a la Burt &
Adelson [4] to generate foveated images at reasonable time scales [10].
This technique convolves an image with a small Gaussian ﬁlter, followed by subsampling of the pixels, reducing the size of the image by a factor of four while avoiding aliasing issues thanks to the preceding low-pass convolution.
Repeating these steps yields a series of sequentially smaller images that can be imagined to make up a pyramid.
Upscaling each image back to its original size produces increasingly blurrier versions of the original, which can then be combined into a single foveated image.
Mipmaps are used in computer graphics and employ the same technique to render scaled-down textures without aliasing.
The rendered pixels are interpolated from the mipmap as a fractional level of the pyramid, i.e. from between two pyramid levels.
The foveation quality of the Gaussian pyramid approach suffers from its reliance on interpolation, which produces a blocky image.
Bilinear interpolation is frequently used as a middle ground between the pixelated results of nearest neighbor sampling and the high computational cost of bicubic interpolation, however, it cannot provide the same smoothness as a purely convolutional blur scheme.
5.2 Methods
[Figure]
for the vertical step by several pixels, leading to a net decrease in the number of operations.
To render foveated images, it is necessary to designate a ﬁxation point and perform increasing blur into the periphery.
Blur can be achieved by convolving the image with a ﬁlter – a small matrix of values that is centered on every pixel in the image, followed by an element-wise multiplication and sum.
Although several ﬁlters exist for achieving a blur effect, we employ the Gaussian ﬁlter, owing to its smooth blur and lack of ringing effects, as well as to its separable property that reduces computational complexity.
The computational demands of the convolution operation can render it prohibitive for time-sensitive tasks.
Gaussian blurring of an image of width w and height h with a ﬁlter of dimensions N × N has a complexity of O ( whN 2 ) .
Consequently, high quality foveated rendering, with large peripheral averaging ﬁlters and high resolution images, presents a challenge by virtue of the number of operations it requires.
A key step in lowering convolution complexity is taking advantage of the separable property of the Gaussian ﬁlter, reducing it from an N 2 square ﬁlter into two N -length one-dimensional (1 D ) ﬁlters [4].
The convolution of the two-dimensional square ﬁlter is equivalent to ﬁrst a horizontal convolution over the image with the ﬁrst ﬁlter, and then a vertical convolution with the second over the result of the horizontal pass.
In the case of a Gaussian ﬁlter both 1 D ﬁlters are identical, hence only one needs to be stored for both the horizontal and vertical step.
Separable convolution is signiﬁcantly faster by reducing the number of operations such that the computation can be performed in O ( whN ) time.
The Gaussian ﬁlter is deﬁned by its standard deviation σ , with a larger σ producing a larger spread of values, and hence more averaging and a stronger blur. [Equation]
[Equation]
Although the Gaussian function decays out to inﬁnity, we must determine a cutoff point for our ﬁlter to use during convolution.
Values beyond 3 σ from the center of the ﬁlter are small enough to be considered negligible, hence each ﬁlter is dimensioned as (cid:100) 6 σ (cid:101) ×
(cid:100) 6 σ (cid:101) .
The ﬁlter matrix is populated with the output of the Gaussian function at each discrete matrix coordinate (center pixel - (0,0)), then normalized to prevent the convolution operation from altering the luminance of the image.
5.2.1 Choosing the Standard Deviation of the Gaussian Filter
The standard deviation of the Gaussian ﬁlter used at a given eccentricity is found through the method used by Geisler and Perry [10], starting with the contrast threshold formula, which gives the contrast required for detection of a patch of sinusoidal grating of a given spatial frequency f (cycles per degree) at retinal eccentricity e (degrees):
[Equation]
where CT 0 is the minimum contrast threshold, α is the spatial frequency decay constant, and e 2 is the half-resolution eccentricity.
We set these values to be the same as used by Geisler and Perry [10], such that α = 0 .
106 , e 2 = 2 . 3 , and CT 0 = 164 .
Setting the left-hand-side of the equation to 1.0, which denotes maximum contrast, and rearranging for f gives:
[Equation]
where f c _ deg is the highest spatial frequency (cycles per degree) that can be resolved at eccentricity e , no matter the contrast.
This equation therefore gives the spatial frequency cutoff at a given retinal eccentricity.
In a metameric foveation scheme the viewing distance and monitor dot-pitch are taken into account to calculate retinal eccentricity and create foveated images indistinguishable from non-foveated images under the set conditions (e.g. Pramod et al.
Instead, we use a retinal mapping approach, where the blurring factor scales proportionally to the falloff of visual acuity of the human retina.
The maximum spatial frequency in an image is a single cycle occupying two pixels, i.e. a black and white pixel in succession, which equates to 0.5 cycles per pixel.
We use the following equation to derive the spatial frequency in cycles per pixel from the spatial frequency in cycles per degree:
[Equation]
where f max is the maximum resolvable spatial frequency across the retina, corresponding to highest visual acuity at the fovea.
The degree eccentricity of a pixel is calculated by ﬁrst assigning the image center to 0 ◦ , and the corner of the image to a desired retinal eccentricity.
By interpreting the ﬂat image as a spherical retina, we create a linear relationship between Euclidean pixel distance and retinal eccentricity:
[Equation]
where d p is the pixel distance from the ﬁxation point, d corner is the half-diagonal image distance, and e corner is the degree retinal eccentricity assigned to the image corner.
In this way, we produce foveated images designed to represent the human visual experience relative to the resolution of the original image, rather than metameric foveation under any speciﬁc conditions.
The algorithm is simple to adapt to achieve the latter.
Finally, we ﬁnd the standard deviation σ of the Gaussian ﬁlter at the chosen eccentricity by treating f c _ pix as the standard deviation of the ﬁlter in the frequency domain, σ f : [Equation]
[Equation]
This signiﬁes that frequencies at σ f and above are considered cut off, or sufﬁciently attenuated by the
Gaussian ﬁlter.
This threshold can be shifted higher for stronger blurring.
5.3 Localized Image Quality Assessment dataset
[Figure]
thread (1, 0) thread (2, 0) thread (3, 0) thread (0, 1) thread (1, 1) thread (2, 1) thread (3, 1) thread (0, 2) thread (1, 2) thread (2, 2) thread (3, 2)
[Figure]
CUDA is the programming interface through which custom programs (kernels) can be executed on an NVIDIA graphics processing unit (GPU).
The advantage of employing a GPU for foveation is that a GPU is able to perform parallel computation of the same kind on large datasets, hence is an ideal candidate for a convolution operation involving millions of pixels.
The GPU is used in tandem with the CPU to accelerate highly parallel operations.
The CPU (host) handles the transfer of data to the
GPU (device), which executes the kernel code written with CUDA.
Figure 11 shows the basic CUDA architecture, which consists of three levels – the threads, the blocks, and the grid.
Each thread performs one of the parallelized computations, such as a single write of a blurred pixel to an output array.
The CUDA threads are organized into blocks, which reﬂect the physical design of the GPU, where the workload is distributed across Streaming Multiprocessors.
The threads within the same block have the useful property of being able to communicate with each other via a fast form of memory (shared memory), which is crucial to the optimization of CUDA code.
The blocks are organized into the grid, which ties together the whole structure of the CUDA kernel.
The grid’s purpose is to deﬁne how many blocks a kernel is launched with, and how these blocks are arranged.
Both the grid and the blocks can have 3 dimensions, which helps organize the programmer’s solution [1].
5.4.1 The CUDA memory model
Optimization of device kernels requires understanding of the different types and properties of memory offered by CUDA.
Global memory is the most basic type of memory available on the GPU – it is the memory typically accessed by the host when data is transferred to the device.
Global memory offers the largest storage size on the GPU, on the order of several gigabytes, but has the drawback of being slow to read from and write to (it is located off-chip, away from the streaming multiprocessors).
Constant memory is a faster type of memory.
It is termed “constant” because it can only be written to by host code, hence is useful when the kernel requires access to read-only data.
The size of this memory is far smaller than that of Global memory - only 64KB [1].
Although it is still off-chip, it is nonetheless much faster to access than global memory because it is aggressively cached to on-chip memory.
Each thread block is designated a small amount of shared memory (48-163 KB) which can be written to and read from by that block alone.
Shared memory is located on-chip and is therefore much faster to access, making it useful for storing intermediate values within a kernel, or data that must be accessed repeatedly.
The GPU architecture also features registers and local memory, which are speciﬁc to individual threads.
Registers offer storage for any variables or arrays declared by a thread within the kernel [Figure] and are the fastest to access, but are limited in quantity.
The CUDA compiler determines which data is placed within a register [Figure] and which data spills out into local memory – termed local not because of its location, but because of its scope.
Local memory is off-chip but accessible only by the speciﬁc thread, hence it is local to each thread but slow to access.
As a result, register spilling is highly undesirable.
5.5 CUDA-Driven Optimization
[Figure]
Our algorithm demonstrates the suitability of CUDA’s block architecture to space-variant image processing.
We detail below the considerations of the kernel design such that the algorithm can be adapted to other foveation schemes while maintaining high performance.
5.5.1 Workload of the Block
Four arrays are placed on the GPU: the input image to be foveated, the blur grid describing the blurring strengths of each fragment, the array of one-dimensional Gaussian ﬁlters a g , and an accompanying array of cumulative ﬁlter sizes a c .
The threads within the block identify their position on the CUDA grid, which corresponds to a value in the blur grid .
This value is an index into a c , which allows the thread to extract the appropriate
Gaussian ﬁlter from a g .
If the block is assigned the foveal fragment, the pixels from the input image are simply written directly to the output array - no blurring is done.
Otherwise, the block performs
Gaussian blurring on its assigned pixels using the extracted Gaussian ﬁlter.
The pixels corresponding to the fragment are padded according to the size of the Gaussian ﬁlter used.
This tile of pixels is extracted, and horizontal convolution is performed.
The result of this operation is the half-blurred data, termed the intermediate result .
Convolution is performed a second time, with the same Gaussian ﬁlter, but this time in a vertical manner, producing the ﬁnal blurred fragment.
The result is written to the output array.
5.5.2 Shared Memory
Shared memory is faster than global memory because it is located on-chip, hence we can achieve higher performance by using it to hold frequently accessed data.
Since the convolution operation accesses the same pixels multiple times, we can ﬁrst load the tile of pixels from global memory into shared memory before the horizontal pass.
This extra step leads to a performance boost by reducing latency throughout the convolution.
We take care to retain the unsigned 8-bit integer image datatype until convolution, which minimizes the data transfer time and the storage footprint in shared memory.
The intermediate result is likewise placed into shared memory, as it is speciﬁc to the workload of the individual block and each thread must access the results of horizontal convolution generated by vertically neighboring threads.
Therefore, the same beneﬁts of shared memory are applied to the vertical convolution pass.
5.5.3 Constant Memory
The storage size of constant memory is too small to store the image to be foveated, but is sufﬁcient to hold the Gaussian ﬁlters used for convolution.
These ﬁlters remain unchanged throughout the kernel execution, but are frequently accessed, typically millions of times during foveation.
Constant memory is therefore an ideal location because it offers low latency in exchange for read-only designation.
The two 1 D ﬁlters produced by the separation of the 2 D Gaussian ﬁlter contain identical valuestheir only difference is the direction in which they are applied.
We therefore only store one instance of an N -length ﬁlter and apply it in a horizontal or vertical fashion as appropriate.
All Gaussian ﬁlters are transferred together into array a g in constant memory, and their cumulative sizes are likewise placed into constant memory as a second array a c .
Cumulative ﬁlter sizes allow the kernel to locate the start and end point of any given Gaussian ﬁlter within a g .
5.5.4 Optimizing for Cancelled Threads
Although image processing typically involves two-dimensional thread blocks to map individual threads to image pixels, this approach is inefﬁcient for space-varying algorithms.
There are three main stages in the execution of the kernel - pixel transfer into shared memory as a padded tile , followed by the horizontal and vertical convolution passes.
These three steps each require a different arrangement of threads and vary with Gaussian ﬁlter size which determines the amount of padding around a fragment.
Therefore, a two-dimensional thread block is impractical, as a ﬁxed thread arrangement across the block workload would lead to unused threads at several points in the algorithm.
Moreover, by launching blocks with ﬁxed dimensions to accommodate the largest ﬁlters in the periphery, we would lose performance on smaller tiles closer to the ﬁxation point.
Optimal performance for every task throughout the workload can be achieved by launching each block as a one-dimensional ( one-D ) line of threads.
Within the kernel, the dimensions of active positions within the tile are speciﬁed at every step, so that the line of threads can be organized to best ﬁt the task at hand.
Figure 13 shows how a one-dimensional block of threads can be reshaped.
The number of active positions in the tile is usually greater than the number of invoked threads, so blocks typically iterate through the tile.
The number of iterations is reduced for one-dimensional blocks as compared to ﬁxed two-dimensional blocks as threads populate only active positions (with the exception of the last iteration where overﬂow might occur).
5.5.5 Data Reuse
An optimization discussed by Volkov [36] involves reusing data accessed within the kernel through registers, which reduces accesses of slower forms of memory.
This technique can be applied during the convolution step of the algorithm.
During convolution, the same Gaussian ﬁlter is used in an overlapping fashion for many neighboring pixels.
If the convolution operation at every pixel is assigned a separate thread, neighboring threads access the same image and ﬁlter data several times.
Performance can be improved by increasing the work done by a single thread, such that memory accesses and data transfers are reduced.
By having a single thread perform four convolutions at once, we achieve over 2x speedup (Figure 12).
The exact factor by which to increase per-thread workload can be determined through experimental evaluation.
Note that since there are typically many more pixel positions in the tile (at which convolution is to be performed) than there are invoked threads, increasing per-thread workload does not lead to inactive threads but rather reduces the number of iterations through the tile.
Data reuse involves registers - on-chip memory that holds variables and arrays declared by a thread.
Registers are the fastest form of memory to access but are limited in quantity, therefore they can be used for temporary storage of values fetched from less efﬁcient memory locations.
Figure 14 shows the ﬁrst two steps of horizontal convolution performed by a thread, where each step is concerned with a different Gaussian ﬁlter value.
In Step 1, the ﬁrst ﬁlter value is accessed.
This value will be used as the ﬁrst step of convolution for all four highlighted pixels, so it is placed in a register and applied immediately at the appropriate positions - four separate convolution sums are maintained.
Reusing the ﬁlter value is faster than extracting it four times from constant memory.
Step 2 uses three of the four image pixel values from the previous step, so these values are likewise reused, rather than being fetched anew from shared memory.
The reuse pattern seen in Step 2 continues for the remaining steps.
5.5.6 Algorithm Limitations
The optimized algorithm is limited by the sizes of constant and shared memory on the GPU.
On an
NVIDIA compute capability 6.1 device, the maximum amount of shared memory per thread block is
48 KB, and the constant memory size is 64 KB [1].
These pooling regions and largest ﬁlter size can be used by the algorithm.
Shared memory is used for image data and the intermediate result produced by horizontal convolution.
The image data is in the form of a tile of width tileW and height tileH , which holds fragment data of dimensions fragmentW and fragmentH , as well as the surrounding padding.
The tile is stored in shared memory with unsigned 8-bit integer datatype - the same as the input image.
The array that holds the intermediate result has dimensions fragmentW ∗ tileH with ﬂoat32 datatype, having been produced through multiplication with ﬂoating point Gaussian ﬁlter values.
Since ﬂoat32 values are four times the size of unsigned 8-bit integers , the intermediate result , despite holding fewer values, can take up more shared memory than image data.
Shared memory usage can be calculated in bits as follows:
[Equation]
Larger Gaussian ﬁlters require more padding around a fragment leading to a larger tileW and tileH .
For fragment dimensions of 32 × 32 , 16 × 16 , and 8 × 8 pixels, the maximum ﬁlter size is 133, 174, and 196 pixels respectively.
Gaussian ﬁlters and their cumulative sizes are placed into arrays a g and a c in constant memory.
The number of bits placed into constant memory is a function of the total number N g of ﬂoat32 values in array a g (total length of all ﬁlters used), as well as the number N c of int32 values in a c (number of ﬁlters used, i.e. number of pooling regions):
[Equation]
N g is determined by the lengths of the Gaussian ﬁlters which correspond to blurring strength deﬁned by speciﬁc foveation parameters.
The maximum number of pooling regions, each of which is deﬁned by a unique ﬁlter, must therefore be determined experimentally.
The limiting factors of shared and constant memory can be removed through the use of global memory in exchange for lower performance.
The impact of these compromises on execution time is shown in
Figure 12.