We show how to build the components of a privacy-aware, live video analytics ecosystem from the bottom up, starting with OpenFace, our new open-source face recognition system that approaches state-of-the-art accuracy. Integrating OpenFace with interframe tracking, we build RTFace, a mechanism for denaturing video streams that selectively blurs faces according to specified policies at full frame rates. This enables privacy management for live video analytics while providing a secure approach for handling retrospective policy exceptions. Finally, we present a scalable, privacy-aware architecture for large camera networks using RTFace and show how it can be an enabler for a vibrant ecosystem and marketplace of privacy-aware video streams and analytics services. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.  Video cameras are ubiquitous today. A recent survey in the United Kingdom estimated one surveillance camera in a public space for every 11 people [11] . Despite privacy concerns, there was a 62% positive response on the Debate.org public opinion website to the question "Are video surveillance cameras in public places a good idea?" [21] . If privacy concerns can be satisfactorily addressed, enabling real-time analytics on video streams from public spaces can be valuable. Examples abound. In a shopping mall, an alert triggered by detection of a liquid spill or broken glass can be promptly addressed. An "Amber Alert" for a lost or abducted child can trigger real-time face recognition in parallel on many video streams from public spaces and help to rapidly locate the child. Real-time activity inferencing on outdoor camera feeds can detect pedestrians slipping on an icy sidewalk and trigger corrective measures. Real-time video analytics can also offer business value, e.g., prompt detection and remedy of lengthening checkout lines to improve customer satisfaction. Prompt discovery that shoppers are not lingering at a new window display may suggest that it is not as appealing as expected and needs to be changed soon. Early detection of icy sidewalks directly benefits insurance companies, who underwrite the risk. Live video offers several advantages relative to other sensing modalities. Most important are its flexibility and open-endedness: new image and video processing algorithms can be developed to enhance the information extracted from an existing video stream. Additionally, video offers high resolution, wide coverage, and low cost relative to other sensing modalities. The passive nature of video sensing is especially attractive for public spaces. A participant does not have to wear a special device, install an app, or do anything special. He or she merely has to be visible to a camera. In this article, we describe the components of an ecosystem around privacy-aware live video analytics from the bottom up. Starting with the core algorithms for privacy, we build up to a scalable system for privacy-aware analytics. Following the lead of Simoens et al. [48], we use the term denaturing to refer to the process of making video "safe" from a privacy viewpoint. A denatured video stream is one whose content has been analyzed to identify potential privacy leaks and has been modified to eliminate those leaks. Section 2.1 discusses the denaturing concept in more detail. A denatured video stream may be safely released to untrusted video analytics software. Our prototype focuses on face recognition as the basis of denaturing for two reasons. First, it leads to a compelling proof of concept because accurate real-time face recognition is a difficult challenge even today. Second, recognition of specific individuals enables many important realworld policies. Figure 1 illustrates the overall architecture of our system. We use edge computing rather than cloud computing to perform denaturing and video analytics. Edge computing is a nascent model of computing in which small multitenant data centers called cloudlets are located close to IoT devices such as video cameras. Davies et al. [20] have explained why denaturing on a cloudlet offers greater confidence in privacy enforcement than denaturing in the cloud. They refer to the software module performing denaturing as a privacy mediator. Scalability is also improved by performing denaturing and video analytics on cloudlets: by avoiding transmission of video into the cloud, ingress bandwidth demand per camera is greatly reduced. This article is an extended version of our prior conference paper [54] , where we first introduced our new face recognition system OpenFace and showed how it can be combined with interframe tracking to produce RTFace, a system for real-time privacy preservation with very low rates of privacy leaks. We showed how RTFace can be used in a scalable architecture for privacy-aware analytics. This extended version further analyzes the costs of deploying privacy-preserving camera systems in an enterprise setting and shows how an ecosystem of camera streams and analytics services can be implemented. A video demo of RTFace is at https://youtu.be/gQa8oScFS94.
Recently proposed RTFace REF not only achieves high recognition accuracy but also operates at the full frame rates of videos.
Enabling Live Video Analytics with a Scalable and Privacy-Aware Framework
TOMM
Computer Science