T he importance of video surveillance techniques [3] , [9] , [25] has increased considerably since the latest terrorist incidents. Safety and security have become critical in many public areas, and there is a specific need to enable human operators to remotely monitor activity across large environments such as: a) transport systems (railway transportation, airports, urban and motorway road networks, and maritime transportation), b) banks, shopping malls, car parks, and public buildings, c) industrial environments, and d) government establishments (military bases, prisons, strategic infrastructures, radar centers, and hospitals). Modern video-based surveillance systems (see classifications described in [9] ) employ real-time image analysis techniques for efficient image transmission, color image analysis, event-based attention focusing, and model-based sequence understanding. Moreover, cheaper and faster computing hardware combined with efficient and versatile sensors create complex system architectures; this is a contributing factor to the increasingly widespread deployment of multicamera systems. These multicamera systems can provide surveillance coverage across a wide area, ensuring object visibility over a large range of depths. They can also be employed to disambiguate occlusions. Techniques that address handover between cameras (in configurations with shared or disjoint views) are therefore becoming increasingly more important. Events of interest (identified as moving objects and people) must be then coordinated in the multiview system, and events deemed of special interest must be tracked throughout the scene (see Figure 1 ). Wherever possible, tracked events should be classified and their dynamics (sometimes called behavior) analyzed to alert an operator or authority of a potential danger. For security awareness based on multiscale spatio-temporal tracking, see Hampampur et al. [10]. In the development of advanced visual-based surveillance systems, a number of key issues critical to successful operation must be addressed. The necessity of working with complex scenes characterized by high variability requires the use of specific and sophisticated algorithms for video acquisition, camera calibration, noise filtering, and motion detection that are able to learn and adapt to changing scene, lighting, and weather conditions. Working with scenes characterized by poor structure requires the use of robust pattern recognition and statistical methods. The use of clusters of fixed cameras, usually grouped in areas of interest but also scattered across the entire scene, requires automatic methods of compensating for chromatic range differences, synchronization of acquired data (for overlapping and nonoverlapping views), estimation of correspondences between and among overlapping views, and registration with local Cartesian reference frames. This article describes the low-level image and video processing techniques needed to implement a modern visual-based surveillance system. In particular, change detection methods for both fixed and mobile cameras (pan and tilt) will be introduced and the registration methods for multicamera systems with overlapping and nonoverlapping views will be discussed. A visual-surveillance system is comprised of a network of sensors (typically conventional closed circuit (CCTV) cameras), some with overlapping fields of view, providing continuous (24/7) online operation. Each visual surveillance network has its own specific architecture. For fixed cameras, the architecture is very much datadriven and its data-to-information flow is bottom-up. As mobile cameras are employed in more sophisticated networks, one might envisage a number of feedback controls to tune camera parameters (for instance, to adapt to weather or illumination conditions) or to track events of interest. Figure 1 illustrates the main processing tasks for a visual system: ■ camera calibration with respect to an extrinsic Cartesian reference frame ■ scene acquisition ■ adaptive modeling of background ■ change detection for foreground regions/blobs identification ■ multicamera registration. All of the steps are intertwined: camera calibration and registration can be learned from observation data, and the processes used to achieve this automatically require basic image processing, such as the modeling of background views and the detection of foreground events. Change detection and camera registration have been chosen as the two basic steps, and they will be described in detail in the next two sections. A variety of change detection methods have been developed for fixed cameras (see Figure 2) . Exhaustive reviews can be found in [9] . Here, the main methods used in visual-based surveillance systems are detailed. The simple difference (SD) method (see Figure 3 ) computes for each time instant t the absolute difference D t (x, y) between the pixel intensities of the input image pair; it then applies a threshold (Th) to obtain a binary image B (x, y The SD method is the simplest and fastest, but it is very sensitive to noise and illumination changes; this directly affects the gray level recorded in the scene, which could be incorrectly interpreted as structural changes. To overcome the problem of noise sensitivity, the derivative model (DM) method considers n × n pixel regions in the two input images and computes a likelihood ratio L ij by using the means and the variances of the two regions R i and R j . The output binary image is obtained as where L Th is the threshold. The shading method (SM) models the intensity at a given point I p (x, y) as the product of the illumination I i (x, y) and a shading coefficient S p which is calculated for each point according to Phong's illumination model. It can be easily proved that, to establish whether a change has taken place in a given region R i over two consecutive frames, I t−1 (x, y) and I t (x, y), it is sufficient to calculate the variance σ i of the intensity ratios I t /(I t−1 ) in that region. If σ i is close to zero, no change has taken place. The DM and SM methods yield similar results. The noise level affecting the output image is lower than that generated by the SD method; however, the accuracy of the detected blobs, in terms of shape, position, and size, is lower. In particular, object contours are significantly altered and the original object shape is partially lost. The LIG method [22] is based on the assumption that pixels at locations having a high gray-level gradient form a part of an object and that nearly all pixels with similar gray levels will be also part of the same object. The intensity gradient is computed as Thereafter, the G(x, y) image is divided into m × m subimages in order to limit the effects of illumination change on the computation of local means and deviations. The regional means and deviations are first smoothed using the neighboring regions and then interpolated to refill a m × m region. Finally, a threshold procedure is applied to isolate object pixels from the background. The LIG method gives satisfactory results, even if is not sufficient to completely discriminate the object from the background. To minimize errors in the background change detection process due to noise effects, illumination, and/or environmental changes, advanced visual-based surveillance systems apply background updating procedures [8] . A classical approach is based on the Kalman filter (see Figure 4) . Background updating procedures attempt to determine significant changes using an estimate of the background scene (see an example in Figure 5 ). In Figure 4 , the filter is applied to each pixel (x, y) of the input image to adaptively predict an estimate of the related background pixel at each time instant. Figure 3 illustrates the data flow diagram for a Kalman-based background updating module [8] . The dynamic system model is represented by where S t (x, y) represents the gray level of the background image point (x, y) at the instant time t; S t+1 (x, y) represents an estimate of the same quantity at t + 1 and µ t is an estimate of the system model error. Such an error takes into account the system model approximation and is formed by two components (i.e., µ t = β t + ν t , where β t represents a slow variation (with non-zero mean and a temporal range comparable to the filter response time) and ν t represents a white noise with zero mean). The model for β t is a random walk model, [FIG3] General scheme of the Kalman-based background updating module.  Background Updating Current Image β t+1 = β t + ε t with initial condition β 0 = 0, where ε t is white noise with zero mean. The measurement model is represented by where I t (x, y) represents the gray-level of the current image point (x, y) and η t (x, y) is the measurement noise (an estimate of the noise affecting the input image). This noise is assumed to be Gaussian with zero mean and variance σ 2 . The recursive equations of the Kalman filter are applied to obtain the optimum linear estimate of the background image S t (x, y) based on the observations (system measures) {I t (x, y) : 0 t k} and initial conditions S 0 (x, y) = I 0 (x, y). The updating module uses the gray-level D t (x, y) of the difference image point (x, y) and the estimate η t (x, y) of the noise affecting the input image to update the filter gain g t (x, y). Then, it computes an estimate of the system model error ε t on the basis of the filter gain g t (x, y) and the D t (x, y) value (i.e., , y) ). The prediction module computes an estimate of the gray-level S t (x, y) of the background image point (x, y) at the next frame. The image storage block simulates the delay between two successive image acquisitions. A multilayered background model has been used in the CMU surveillance system [5] , which employs a combination of temporal differences and template matching to perform object tracking. Each time an object enters the scene and remains stationary for a predefined amount of time, it is considered a new layer and is added to the background (i.e., a parked car). This adopted multilayer solution was effective in solving typical problems of background-based change detection approaches, such as "holes'' or "ghosts'' created by objects in the background that began to move (such as a car leaving the area). None of the methods described above address the problem of multimodal backgrounds that are typical of outdoor scenes, where the pixels can switch state (e.g., due to trees waving in the wind). For these conditions, a single Gaussian model per pixel cannot be assumed. Stauffer and Grimson [34] modeled the recent history of each pixel, {x 1 , . . . , x t }, as a mixture of k Gaussian distributions. The probability of observing the current pixel value is given by where k is the number of distributions, ω i,t , µ i,t , i,t are an estimate of the weight, the mean, and the covariance matrix of the ith Gaussian of the mixture at time t, and * is the convolution operator. η is a Gaussian probability density function: The parameter k is, in part, determined by the available memory and computational power; a value in the range of 3-5 is suggested by Stauffer and Grimson [34]. [FIG5] A flow chart with the main tasks involved in the computation of the image of differences in the context of a mobile camera. It is worth noting how the motion compensation is employed to negate the camera motion.
Foresti et al. REF used a background subtraction technique to detect the moving object and then remove the shadow in the subsequent phase.
Active video-based surveillance system: the low-level image and video processing techniques needed for implementation
IEEE Signal Processing Magazine
IEEE Signal Processing Magazine
Computer Science