Abstract-We consider modeling the visibility of individual and multiple packet losses in H.264 videos. We propose a model for predicting the visibility of multiple packet losses and demonstrate its performance on dual losses (two nearby packet losses). We extract the factors affecting visibility using a reduced-reference method. We predict the probability that a loss is visible using a generalized linear model. We achieve MSE values (between actual and predicted probabilities) of 0.0253 and 0.0398 for individual and dual losses respectively. We also examine the effect of various factors on visibility. Index Terms: Video codecs, Quality Control I. INTRODUCTION Compressed video streams transmitted over heterogeneous networks experience visual quality impairments due to various factors such as delay, jitter, packet loss, drift (due to scalability) and compression artifacts. Being able to quantify the perceptual quality degradation due to these factors is important for a network quality monitor. Objective quality measures such as PSNR do not correlate well with subjective results. The current work concentrates on modeling the quality degradation in H.264 video due to packet losses by predicting the packet loss visibility. Predicting the visibility of a packet loss is useful for several reasons. Packets which are perceptually important can be given more error protection (unequal error protection). If one assigns perceptual importance levels to each of the packets at the encoder, the packets with lower importance level can be discarded, if the buffer in a network node overflows. Thirdly, packet loss visibility can be very useful for accurate, real-time network quality monitoring. In our previous work [1], we looked at the problem of predicting the visibility of individual packet losses in MPEG-2 bitstreams. However, video transmission over internet or wireless links is typically characterized by bursty losses. Stuhlmuller et al. [2] analyzed the distortion (MSE) of isolated packet losses. They also used a linear additive model to quantify the distortion of multiple packet losses. In [3] , the authors compare bursty losses with isolated losses of equal combined length. They conclude that: (a) the loss pattern has significant impact on distortion, (b) bursty loss produces larger distortion than an equal number of isolated losses. Chakareski et al. [4] proposed a scheme to predict the distortion incurred by bursty losses. In this paper, we extend our previous work [1] in two ways. First, we model the visibility of isolated packet losses in H.264 instead of MPEG-2 bitstreams. We use motioncompensated error concealment (MCEC) to conceal the losses, instead of the zero error concealment (ZEC). Second, we model the visibility of multiple packet losses for H.264. We define a multiple loss as a set of N individual losses occurring in close temporal proximity. We introduce a model framework to predict the visibility of a multiple packet loss and we examine the performance of this approach when N = 2 (dual loss). We compare the importance of factors between the isolated and dual loss cases. Section II describes the design of the subjective experiment. Section III discusses the factors we use to predict the visibility of a loss. Section IV describes our modeling approach, while Section V provides results. We conducted subjective tests to obtain ground truth on the visibility of packet losses. The viewers' task is to indicate when they see an artifact or abnormality. Our tests are single stimulus tests, so viewers are only shown videos with packet losses, not original videos. We assume a packet loss entails the loss of a single slice, namely a horizontal row of macroblocks (MBs)'. The initial error induced by a packet loss depends on the decoder's error concealment strategy. We use motion-compensated error concealment (MCEC). The initial error incurred with this method is less than that of the zero-error concealment method used in our previous work [1] . The video sequences in the subjective test are muted travel documentaries at SIF resolution (352 x 240) and 30 fps. They are encoded and decoded using the extended profile of H.264 JM Version 9.1 Codec. Our encoding structure is I B P B P B...P B with a GOP size of 20 frames. For P frames, we use two reference frames for motion compensation: one long-term and one short-term. The long-term reference frame is always the I frame of the current GOP. We follow the usual convention for the short-term reference frame to be the previous P frame. B frames use the future P frame and either reference frame for bidirectional prediction. We use fine quantization (28) without rate control, so that the only artifacts in the lossy videos are due to packet loss. The decoder conceals the lost slices using MCEC where the concealment motion vector is estimated as the median of the motion vectors of surrounding blocks. We are interested in the visibility of individual and multiple packet losses. In practice, packet losses tend to occur together. Different losses within a multiple packet loss interact with each other; the overall effect is not the sum of individual effects. This interaction can be either physical or perceptual. Physical interaction is caused by inter-frame prediction. Perceptual interaction is caused by the close proximity (spatial or temporal) of the losses. Here, we study the overall effect of two packet losses (dual loss) occurring together. A dual loss is characterized by the spatial separation D between the two losses in MB units (i.e., D = 1 implies the two packet losses affect adjacent slices) and by the temporal separation T between the two losses in number of frames. In our case, D varies from 0 to 14 (15 slices in each frame). T varies from 0 to 5 frames (maximum separation of 1/6 of a second). We choose six videos of 6 minutes each, divided into 4-second intervals called slots, producing 90 slots per video and 540 slots in total. A loss (individual/dual) is introduced in the first three seconds of each slot. The last second is reserved to create a guard interval, which prevents interaction between losses across slots and provides the viewer time to respond to the current loss before the next loss occurs. We design 4 individual losses for each slot and use all 6 combinations of individual loss pairs to get 6 dual losses. Within each time slot, either exactly one of these 4 individual losses will appear, or exactly two of them will appear (dual loss). The individual losses are designed such that the set of dual losses are approximately distributed uniformly over D and T. Since we have 10 different losses (4 individual and 6 dual) that can be introduced in a slot, 10 different lossy versions are created from each source video. Each lossy video has both individual and dual losses, but in different slots. In total, we introduced 2160 individual losses and 3240 dual losses. During the subjective test, a viewer is shown only one set of 6 lossy videos. Each set of lossy videos was evaluated by 12 viewers, for a total of 120 viewers. To help viewers understand their task, we show them a 1-minute pilot training video before the actual test. Viewers are told that they will watch videos which are affected by packet losses. Whenever they see a visible artifact or a glitch, they should press the space bar. After the subjective test, we generate a table of the viewers' boolean responses, corresponding to whether they saw a loss or not. This defines the ground truth for the probability of visibility for each loss. In this paper, we use a Reduced-Reference (RR) method for predicting the visibility of packet losses. A RR method has access to the decoder's reconstructed video (with losses) and factors extracted from the encoded video. We classify the factors that determine packet-loss visibility into ContentIndependent and Content-Dependent Factors. Content-Independent Factors depend only on the location of the loss, not on its actual content. We consider two content-independent factors. HGT [1] is defined as the height of the lost slice in a given frame, where slices are numbered from top to bottom. FRAMETYPE is the type of frame (B/P/I) affected by the packet loss and is treated as a categorical factor. FRAMETYPE ML is the counterpart of FRAMETYPE for the dual (multiple) loss case. Between the two frames affected, FRAMETYPE ML represents the type that belongs to a higher category. Category I is higher than category P, which is higher than category B. Content-dependent Factors, on the other hand, depend on the actual video content at the location of the loss, such as Motion, Contrast, etc. We use the following: 1) Initial Mean Squared Error (IMSE) is the MSE between the error-free reconstructed MB and the lossy concealed MB. Factors AVGIMSE and MAXIMSE are the average and maximum IMSE of all MBs in a given slice. 2) Residual Energy is the energy (sum of squares of all DCT coefficients) of the residual after motion compensation. If a slice is lost, then even if MCEC does a perfect job of estimating the lost motion vectors, the resultant slice still differs from the original. The residual energy, calculated on a MB basis, is one way to assess the magnitude of this difference. AVGRSENGY and MAXRSENGY are the
However, there is also the issue of PLV which in REF 's subjective tests of H.264 video was found to be dependent on: the initial mean square error between the error-free and concealed MBs in a slice; and the maximum number of partitions of a block; and the frame-type, though other factors were also significant.
Predicting H.264 Packet Loss Visibility using a Generalized Linear Model
2006 International Conference on Image Processing
2006 International Conference on Image Processing
Engineering Computer Science