Original: S ELF -S UPERVISED L EARNING OF A PPLIANCE U SAGE
Revised: S ELF -S UPERVISED L EARNING OF A PPLIANCE U SAGE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning home appliance usage is important for understanding people’s activities and optimizing energy consumption.
Revised: Learning home appliance usage is important for understanding people’s activities and optimizing energy consumption.

Original: The problem is modeled as an event detection task, where the objective is to learn when a user turns an appliance on, and which appliance it is (microwave, hair dryer, etc.).
Revised: The problem is modeled as an event detection task, where the objective is to learn when a user turns an appliance on, and which appliance it is (microwave, hair dryer, etc.).

Original: Ideally, we would like to solve the problem in an unsupervised way so that the method can be applied to new homes and new appliances without any labels.
Revised: Ideally, we would like to solve the problem in an unsupervised way so that the method can be applied to new homes and new appliances without any labels.

Original: To this end, we introduce a new deep learning model that takes input from two home sensors: 1) a smart electricity meter that outputs the total energy consumed by the home as a function of time, and 2) a motion sensor that outputs the locations of the residents over time.
Revised: To this end, we introduce a new deep learning model that takes input from two home sensors: 1) a smart electricity meter that outputs the total energy consumed by the home as a function of time, and 2) a motion sensor that outputs the locations of the residents over time.

Original: The model learns the distribution of the residents’ locations conditioned on the home energy signal.
Revised: The model learns the distribution of the residents’ locations conditioned on the home energy signal.

Original: We show that this cross-modal prediction task allows us to detect when a particular appliance is used, and the location of the appliance in the home, all in a self-supervised manner, without any labeled data.
Revised: We show that this cross-modal prediction task allows us to detect when a particular appliance is used, and the location of the appliance in the home, all in a self-supervised manner, without any labeled data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning home appliance usage patterns is useful for understanding user habits and optimizing electricity consumption.
Revised: Learning home appliance usage patterns is useful for understanding user habits and optimizing electricity consumption.

Original: For example, knowing when a person uses their microwave, fridge, oven, coffee machine or dish washer provides information about their eating patterns.
Revised: For example, knowing when a person uses their microwave, fridge, oven, coffee machine or dish washer provides information about their eating patterns.

Original: Similarly, understanding when they use their TV, air-conditioner, or washing machine provides knowledge of their behavior and habits.
Revised: Similarly, understanding when they use their TV, air-conditioner, or washing machine provides knowledge of their behavior and habits.

Original: Such information may be used to encourage energy saving by optimizing appliance usage, to track the wellbeing of elderly living alone (Armel et al., 2013; Donini et al., 2013; Debes et al., 2016), or to provide users with behavioral analytics (Zhou & Yang, 2016; Zipperer et al., 2013).
Revised: Such information may be used to encourage energy saving by optimizing appliance usage, to track the wellbeing of elderly living alone (Armel et al., 2013; Donini et al., 2013; Debes et al., 2016), or to provide users with behavioral analytics (Zhou & Yang, 2016; Zipperer et al., 2013).

Revised: This data is also useful for various businesses such as home insurance companies interested in assessing accident risks and utility companies interested in optimizing energy efﬁciency (Armel et al., 2013).

Original: The problem can be modeled as event detection – i.e., given the total energy consumed by the house as a function of time, we want to detect when various appliances are turned on.
Revised: The problem can be modeled as event detection – i.e., given the total energy consumed by the house as a function of time, we want to detect when various appliances are turned on.

Original: Past work has looked at analyzing the energy signal from the home utility meter to detect when certain appliances are on.
Revised: Past work has looked at analyzing the energy signal from the home utility meter to detect when certain appliances are on.

Original: Most solutions, however, assume that the energy pattern for each appliance is unique and known, and use this knowledge to create labeled data for their supervised models.
Revised: Most solutions, however, assume that the energy pattern for each appliance is unique and known, and use this knowledge to create labeled data for their supervised models.

Original: (Kolter et al., 2010; Zhong et al., 2014; 2015; Kelly & Knottenbelt, 2015; Zhang et al., 2018; Bonﬁgli et al., 2018).
Revised: (Kolter et al., 2010; Zhong et al., 2014; 2015; Kelly & Knottenbelt, 2015; Zhang et al., 2018; Bonﬁgli et al., 2018).

Original: Unfortunately, such solutions do not generalize well because the energy pattern of an appliance depends on its brand and can differ from one home to another (Kelly & Knottenbelt, 2015; Bonﬁgli et al., 2018).
Revised: Unfortunately, such solutions do not generalize well because the energy pattern of an appliance depends on its brand and can differ from one home to another (Kelly & Knottenbelt, 2015; Bonﬁgli et al., 2018).

Original: The literature also contains some unsupervised methods, but they typically have limited accuracy (Kim et al., 2011; Kolter & Jaakkola, 2012; Johnson & Willsky, 2013;
Revised: The literature also contains some unsupervised methods, but they typically have limited accuracy (Kim et al., 2011; Kolter & Jaakkola, 2012; Johnson & Willsky, 2013;

Original: Parson et al., 2014; Wytock & Kolter, 2014;
Revised: Parson et al., 2014; Wytock & Kolter, 2014;

Original: Zhao et al., 2016; Lange & Berges, 2018).
Revised: Zhao et al., 2016; Lange & Berges, 2018).

Original: Unsupervised event detection in a data stream is intrinsically challenging because we do not know what patterns to look for.
Revised: Unsupervised event detection in a data stream is intrinsically challenging because we do not know what patterns to look for.

Original: In our task, not only may appliance energy patterns be unknown, but also the energy signal may include many background events unrelated to appliance activation, such as the fridge or HVAC power cycling events.
Revised: In our task, not only may appliance energy patterns be unknown, but also the energy signal may include many background events unrelated to appliance activation, such as the fridge or HVAC power cycling events.

Original: One way to address this challenge is to consider the self-supervised paradigm.
Revised: One way to address this challenge is to consider the self-supervised paradigm.

Original: If a different stream of data also observes the events of interest, we can use this second modality to provide self-supervising signals for event detection.
Revised: If a different stream of data also observes the events of interest, we can use this second modality to provide self-supervising signals for event detection.

Original: To that end, we leverage the availability of new ﬁne-resolution motion sensors which track the locations of people at home (Adib et al., 2015; Joshi et al., 2015;
Revised: To that end, we leverage the availability of new ﬁne-resolution motion sensors which track the locations of people at home (Adib et al., 2015; Joshi et al., 2015;

Original: Li et al., 2016; Ghourchian et al., 2017; Hsu et al., 2017).
Revised: Li et al., 2016; Ghourchian et al., 2017; Hsu et al., 2017).

Original: Such sensors operate as a consumer radar, providing decimeter-level location accuracy.
Revised: Such sensors operate as a consumer radar, providing decimeter-level location accuracy.

Original: They do not require people to wear sensors on their bodies, can operate through walls, and track people’s locations in different rooms.
Revised: They do not require people to wear sensors on their bodies, can operate through walls, and track people’s locations in different rooms.

Original: These location sensors indirectly observe the events of interest.
Revised: These location sensors indirectly observe the events of interest.

Original: Speciﬁcally, they capture the change in user locations as they reach out to an appliance to set it up or turn it on (put food in a microwave and turn it on).
Revised: Speciﬁcally, they capture the change in user locations as they reach out to an appliance to set it up or turn it on (put food in a microwave and turn it on).

Original: Hence, the output of such sensors can provide a 
Revised: Hence, the output of such sensors can provide a second modality for self-supervision.

Original: But how should one design the model?
Revised: But how should one design the model?

Original: We cannot directly use location as a label for appliance activation events.
Revised: We cannot directly use location as a label for appliance activation events.

Original: People can be next to an appliance but neither activate it nor interact with it.
Revised: People can be next to an appliance but neither activate it nor interact with it.

Original: We also cannot use the two modalities to learn a joint representation of the event in a shared space.
Revised: We also cannot use the two modalities to learn a joint representation of the event in a shared space.

Original: This is because location and energy are unrelated most of the time and become related only when the event of interest occurs.
Revised: This is because location and energy are unrelated most of the time and become related only when the event of interest occurs.

Original: Furthermore, there are typically multiple residents in the home, making it hard to tell which of them interacted with the appliance.
Revised: Furthermore, there are typically multiple residents in the home, making it hard to tell which of them interacted with the appliance.

Original: Our model is based on cross-modal prediction.
Revised: Our model is based on cross-modal prediction.

Original: We train a neural network that, given the home energy at a particular time, predicts the location of the home residents.
Revised: We train a neural network that, given the home energy at a particular time, predicts the location of the home residents.

Original: Our intuition is that appliance activation events have highly predictable locations, typically the location of the appliance.
Revised: Our intuition is that appliance activation events have highly predictable locations, typically the location of the appliance.

Original: In contrast, background energy events (power cycling of the fridge) do not lead to predictable locations.
Revised: In contrast, background energy events (power cycling of the fridge) do not lead to predictable locations.

Original: Thus, our model uses this learned predictability along with the associated location and energy representation to cluster the events in the energy stream.
Revised: Thus, our model uses this learned predictability along with the associated location and energy representation to cluster the events in the energy stream.

Original: In addition, we formulate a mixture distribution to disentangle irrelevant location information from other residents in the home.
Revised: In addition, we formulate a mixture distribution to disentangle irrelevant location information from other residents in the home.

Original: Interestingly, our model not only learns when each appliance is activated but also discovers the location of that appliance in the home, all without any labeled data.
Revised: Interestingly, our model not only learns when each appliance is activated but also discovers the location of that appliance in the home, all without any labeled data.

Original: We summarize the contributions of this paper as follows:
Revised: We summarize the contributions of this paper as follows:

Original: • The paper introduces a new method for self-supervised event detection from weakly related data streams.
Revised: • The paper introduces a new method for self-supervised event detection from weakly related data streams.

Original: The method combines neural cross-modal prediction with custom clustering based on the learned predictability and representation.
Revised: The method combines neural cross-modal prediction with custom clustering based on the learned predictability and representation.

Original: We apply it to the task of detecting appliance usage events using unlabeled data from two sensors in the home: the energy meter, and a location sensor.
Revised: We apply it to the task of detecting appliance usage events using unlabeled data from two sensors in the home: the energy meter, and a location sensor.

Original: • To evaluate our design, we have created the ﬁrst dataset with concurrent streams of home energy and location data, collected from 4 homes over a period of 7 months.
Revised: • To evaluate our design, we have created the ﬁrst dataset with concurrent streams of home energy and location data, collected from 4 homes over a period of 7 months.

Original: For each home, data was collected for 2 to 4 months.
Revised: For each home, data was collected for 2 to 4 months.

Original: Ground truth measurements are provided via smart plugs connected directly to each appliance.
Revised: Ground truth measurements are provided via smart plugs connected directly to each appliance.

Original: • Compared against past work on unsupervised learning of appliance usage and a new baseline that leverages the two modalities, our method achieves signiﬁcant improvements of 66.7% and 48.8% respectively for the average detection F1 score.
Revised: • Compared against past work on unsupervised learning of appliance usage and a new baseline that leverages the two modalities, our method achieves signiﬁcant improvements of 66.7% and 48.8% respectively for the average detection F1 score.

Original: We will release our code and dataset to encourage future work on multi-modal models for understanding appliance usage patterns and the underlying user behavior.
Revised: We will release our code and dataset to encourage future work on multi-modal models for understanding appliance usage patterns and the underlying user behavior.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Energy disaggregation Our work is related to past work on energy disaggregation, which refers to the problem of separating appliance-level energy from a home’s total (or aggregate) energy signal.
Revised: Energy disaggregation Our work is related to past work on energy disaggregation, which refers to the problem of separating appliance-level energy from a home’s total (or aggregate) energy signal.

Original: Past work in this domain can be broadly classiﬁed into two categories: supervised and unsupervised.
Revised: Past work in this domain can be broadly classiﬁed into two categories: supervised and unsupervised.

Original: Supervised methods assume that the power signatures of individual appliances are available.
Revised: Supervised methods assume that the power signatures of individual appliances are available.

Original: They use data from individual appliances to obtain models for each appliance power signature, and then use those models to detect appliance events from the aggregate energy signal.
Revised: They use data from individual appliances to obtain models for each appliance power signature, and then use those models to detect appliance events from the aggregate energy signal.

Original: Early work learns sparse codes for different appliances (Kolter et al., 2010) or uses a Factorial HMM (FHMM) (Ghahramani & Jordan, 1996) to model each appliance as an HMM (Zhong et al., 2014; 2015).
Revised: Early work learns sparse codes for different appliances (Kolter et al., 2010) or uses a Factorial HMM (FHMM) (Ghahramani & Jordan, 1996) to model each appliance as an HMM (Zhong et al., 2014; 2015).

Original: More recently, neural networks have been used to model appliances (Kelly & Knottenbelt, 2015; Zhang et al., 2018;
Revised: More recently, neural networks have been used to model appliances (Kelly & Knottenbelt, 2015; Zhang et al., 2018;

Original: Bonﬁgli et al., 2018), where extracting appliance-level energy is formulated as a de-noising problem.
Revised: Bonﬁgli et al., 2018), where extracting appliance-level energy is formulated as a de-noising problem.

Original: However, supervised solutions do not generalize well to new homes (Kelly & Knottenbelt, 2015;
Revised: However, supervised solutions do not generalize well to new homes (Kelly & Knottenbelt, 2015;

Original: Bonﬁgli et al., 2018).
Revised: Bonﬁgli et al., 2018).

Original: This is because two appliances of the same type (e.g. coffee machine) in different homes are often manufactured by different brands, and thus have different power signatures.
Revised: This is because two appliances of the same type (e.g. coffee machine) in different homes are often manufactured by different brands, and thus have different power signatures.

Original: Unsupervised methods do not assume prior knowledge of appliance signatures; they attempt to learn those signatures from the aggregate energy signal.
Revised: Unsupervised methods do not assume prior knowledge of appliance signatures; they attempt to learn those signatures from the aggregate energy signal.

Original: Early approaches use variants of FHMM, and learn appliance HMMs with Expectation-Maximization (Kim et al., 2011), approximate footprint extraction procedures (Kolter & Jaakkola, 2012), or use expert knowledge to conﬁgure prior parameters (Johnson & Willsky, 2013; Parson et al., 2014).
Revised: Early approaches use variants of FHMM, and learn appliance HMMs with Expectation-Maximization (Kim et al., 2011), approximate footprint extraction procedures (Kolter & Jaakkola, 2012), or use expert knowledge to conﬁgure prior parameters (Johnson & Willsky, 2013; Parson et al., 2014).

Original: Some papers propose using contextual information (such as temperature, hour of the day, and day of the week)
Revised: Some papers propose using contextual information (such as temperature, hour of the day, and day of the week)

Original: (Wytock & Kolter, 2014), or use event-based signal processing methods to cluster appliances (Zhao et al., 2016).
Revised: (Wytock & Kolter, 2014), or use event-based signal processing methods to cluster appliances (Zhao et al., 2016).

Original: More recently, Lange & Berges (2018) proposed using a recurrent neural network as the variational distribution in learning the FHMM. 
Revised: More recently, Lange & Berges (2018) proposed using a recurrent neural network as the variational distribution in learning the FHMM. In [Figure]

Original: In contrast, our work leverages people’s location data as a self-supervising signal.
Revised: contrast, our work leverages people’s location data as a self-supervising signal.

Original: We cluster appliance events through learning the relation between energy events and people’s locations, and also learn appliance locations as a by-product.
Revised: We cluster appliance events through learning the relation between energy events and people’s locations, and also learn appliance locations as a by-product.

Original: [Figure] Passive location sensing Motivated by new in-home applications and continuous health monitoring, recent years have witnessed an increasing number of indoor location sensing systems (Adib et al., 2015; Joshi et al., 2015; Li et al., 2016; Ghourchian et al., 2017).
Revised: Passive location sensing Motivated by new in-home applications and continuous health monitoring, recent years have witnessed an increasing number of indoor location sensing systems (Adib et al., 2015; Joshi et al., 2015; Li et al., 2016; Ghourchian et al., 2017).

Original: They infer people’s locations passively by analyzing how people change the surrounding radio signals (e
Revised: They infer people’s locations passively by analyzing how people change the surrounding radio signals (e

Original: [Figure]
Revised: [Figure]

Original: g. WiFi) and do not require people to wear any sensors.
Revised: g. WiFi) and do not require people to wear any sensors.

Original: These sensors have been used for various applications including activity recognition (Wang et al., 2014; 2015), sleep monitoring (Zhao et al., 2017), mobility tracking (Hsu et al., 2017)
Revised: These sensors have been used for various applications including activity recognition (Wang et al., 2014; 2015), sleep monitoring (Zhao et al., 2017), mobility tracking (Hsu et al., 2017)

Original: [Figure]
Revised: [Figure]

Original: and health monitoring (Kaltiokallio et al., 2012).
Revised: and health monitoring (Kaltiokallio et al., 2012).

Original: In our work, we leverage the availability of such sensors to introduce location data as an additional data modality for learning appliance usage patterns.
Revised: In our work, we leverage the availability of such sensors to introduce location data as an additional data modality for learning appliance usage patterns.

Original: Self-supervised multi-modal learning Our work is related to a growing body of work on multimodal learning.
Revised: Self-supervised multi-modal learning Our work is related to a growing body of work on multimodal learning.

Original: Most approaches learn to encode the multi-modal data into a shared space (Harwath et al., 2018; Owens & Efros, 2018; Zhao et al., 2018).
Revised: Most approaches learn to encode the multi-modal data into a shared space (Harwath et al., 2018; Owens & Efros, 2018; Zhao et al., 2018).

Original: In contrast, since our two modalities are mostly unrelated and become related only when an activation event happens, we learn to predict one modality conditioned on the other.
Revised: In contrast, since our two modalities are mostly unrelated and become related only when an activation event happens, we learn to predict one modality conditioned on the other.

Original: Our work is also related to cross-modal prediction (Krishna et al., 2017; Owens et al., 2016; Zhang et al., 2017) but differs from it in an essential way.
Revised: Our work is also related to cross-modal prediction (Krishna et al., 2017; Owens et al., 2016; Zhang et al., 2017) but differs from it in an essential way.

Original: Past work on cross-modal prediction typically uses the prediction as the target outcome (e.g. output text for video captioning).
Revised: Past work on cross-modal prediction typically uses the prediction as the target outcome (e.g. output text for video captioning).

Original: In contrast, our objective is to discover the hidden appliance activation events.
Revised: In contrast, our objective is to discover the hidden appliance activation events.

Original: Thus, we design our method to leverage the learned predictability and cross-modal mapping for clustering activation events.
Revised: Thus, we design our method to leverage the learned predictability and cross-modal mapping for clustering activation events.

Original: Furthermore, we introduce a mixture prediction design to disentangle unrelated information in our predicted modality (location measurements unrelated to energy events).
Revised: Furthermore, we introduce a mixture prediction design to disentangle unrelated information in our predicted modality (location measurements unrelated to energy events).

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: Our goal is to learn appliance activation events in an unsupervised way, using two input streams: home aggregate energy and residents’ location data.
Revised: Our goal is to learn appliance activation events in an unsupervised way, using two input streams: home aggregate energy and residents’ location data.

Original: Figure 1 shows the two data modalities.
Revised: Figure 1 shows the two data modalities.

Original: We describe each of them formally and deﬁne appliance “events” below.
Revised: We describe each of them formally and deﬁne appliance “events” below.

Original: Aggregate energy signal A household’s total energy consumption is measured by a utility meter regularly.
Revised: Aggregate energy signal A household’s total energy consumption is measured by a utility meter regularly.

Original: This measures the sum of energy consumption from all appliances at each point in time.
Revised: This measures the sum of energy consumption from all appliances at each point in time.

Original: We denote the aggregate energy signal by y =
Revised: We denote the aggregate energy signal by y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y T ) , where y t ∈ R + .
Revised: , y T ) , where y t ∈ R + .

Original: Suppose there are a total of K appliances in a home, and each appliance’s energy signal is denoted by x k = ( x 1 ,k , x 2 ,k , . . .
Revised: Suppose there are a total of K appliances in a home, and each appliance’s energy signal is denoted by x k = ( x 1 ,k , x 2 ,k , . . .

Original: , x T,k ) , where x t,k ∈ R + .
Revised: , x T,k ) , where x t,k ∈ R + .

Original: Only the aggregate energy signal is observed, y t
Revised: Only the aggregate energy signal is observed, y t

Original: = (cid:80)K k
Revised: = (cid:80)

Original:  =1 x t,k + (cid:15) t , where
Revised: Kk =1 x t,k + (cid:15) t , where

Original: (cid:15) t ∼ N (0 , σ 2 ) is the background noise.
Revised: (cid:15) t ∼ N (0 , σ 2 ) is the background noise.

Revised: 𝐸 𝑦 #:#%& ’ (𝑦 #:#%& ’ 𝑧 #,+,-# 𝑧 #,+.

Revised: # 𝐿𝐿𝒫(𝑙|𝑧 #,+.# ) 𝒫(𝑙|𝑐)

Revised: 𝑐 𝐷 ⨁ 𝒫(𝑙|𝑦 #:#%& ’ ,𝑐) ℒ ;0+ ℒ <,+ 𝑙 #:#%& = 𝛼 (1 − 𝛼 ) (energy signal) (context) (location)

Revised: [Figure]

Original: Figure 1a shows one day of an aggregate energy signal.
Revised: Figure 1a shows one day of an aggregate energy signal.

Original: The base power level shifts constantly throughout a day, depending on the background load (e.g. ceiling lights).
Revised: The base power level shifts constantly throughout a day, depending on the background load (e.g. ceiling lights).

Original: Added on top of the base level are the various appliance events.
Revised: Added on top of the base level are the various appliance events.

Original: Figure 1b zooms in around 20:30, and shows examples of those events.
Revised: Figure 1b zooms in around 20:30, and shows examples of those events.

Original: The stove was turned on around 20:28, and its power continues to cycle between a few levels.
Revised: The stove was turned on around 20:28, and its power continues to cycle between a few levels.

Original: While the stove was on, the microwave was also turned on and ran for a few minutes, and the garbage disposer was turned on shortly.
Revised: While the stove was on, the microwave was also turned on and ran for a few minutes, and the garbage disposer was turned on shortly.

Original: 𝐸 𝑦 #:#%& ’ (𝑦 #:#%& ’ 𝑧 #,+,-# 𝑧 #,+.

Original: # 𝐿𝐿𝒫(𝑙|𝑧 #,+.# ) 𝒫(𝑙|𝑐)

Original: 𝑐 𝐷 ⨁ 𝒫(𝑙|𝑦 #:#%& ’ ,𝑐) ℒ ;0+ ℒ <,+ 𝑙 #:#%& = 𝛼 (1 − 𝛼 ) (energy signal) (context) (location)

Original: [Figure]

Original: [Figure]

Original: Indoor location data We use a single location sensor similar to that in Hsu et al.
Revised: Indoor location data We use a single location sensor similar to that in Hsu et al.

Original: (2017) to measure people’s indoor locations passively.
Revised: (2017) to measure people’s indoor locations passively.

Original: The sensor sends out radio signals and analyzes the reﬂections to localize multiple people.
Revised: The sensor sends out radio signals and analyzes the reﬂections to localize multiple people.

Original: Similarly to a regular WiFi router, the sensor has a limited coverage area of up to 40 feet.
Revised: Similarly to a regular WiFi router, the sensor has a limited coverage area of up to 40 feet.

Original: Suppose there are P t people in the coverage area at time t .
Revised: Suppose there are P t people in the coverage area at time t .

Original: The location data is denoted by l t =
Revised: The location data is denoted by l t =

Original: ( l t, 1 , l t, 2 , . . .
Revised: ( l t, 1 , l t, 2 , . . .

Original: , l t,P t ) , where l t,p ∈ R 2 is the x-y location at time t of person p .
Revised: , l t,P t ) , where l t,p ∈ R 2 is the x-y location at time t of person p .

Original: We can represent the location data over multiple time frames as l 1: T =
Revised: We can represent the location data over multiple time frames as l 1: T =

Original: ( l 1 , l 2 , . . .
Revised: ( l 1 , l 2 , . . .

Original: , l T ) .
Revised: , l T ) .

Original: Figure 1c shows one minute of location data from two people, and Figure 1d shows the data from a top-down view.
Revised: Figure 1c shows one minute of location data from two people, and Figure 1d shows the data from a top-down view.

Original: Appliance activation events When an appliance is turned on, it causes a jump in energy consumption, i.e. a leading edge in the energy signal, as shown in Figure 1b.
Revised: Appliance activation events When an appliance is turned on, it causes a jump in energy consumption, i.e. a leading edge in the energy signal, as shown in Figure 1b.

Original: We call such a pattern an appliance activation event.
Revised: We call such a pattern an appliance activation event.

Original: On the other hand, when an appliance changes its internal state, it can also cause a change in the energy signal as shown in the same ﬁgure.
Revised: On the other hand, when an appliance changes its internal state, it can also cause a change in the energy signal as shown in the same ﬁgure.

Original: We call such a pattern a background event.
Revised: We call such a pattern a background event.

Original: We are interested in discovering activation events to learn appliance usage patterns.
Revised: We are interested in discovering activation events to learn appliance usage patterns.

Original: Thus, for each jump in the aggregate signal, we take a time window (default is 25 seconds) centered around that jump, and analyze it to detect whether it is an activation event and which appliance it corresponds to.
Revised: Thus, for each jump in the aggregate signal, we take a time window (default is 25 seconds) centered around that jump, and analyze it to detect whether it is an activation event and which appliance it corresponds to.

Original:  M ODEL
Revised:  M ODEL

Original: Our model operates on time windows (25 seconds) centered around jumps in aggregate energy signal, and the corresponding time windows of location data.
Revised: Our model operates on time windows (25 seconds) centered around jumps in aggregate energy signal, and the corresponding time windows of location data.

Original: The model aims to detect appliance activation events by ﬁnding windows with highly predictable user locations conditioned on the energy signal.
Revised: The model aims to detect appliance activation events by ﬁnding windows with highly predictable user locations conditioned on the energy signal.

Original: Figure 2 shows our model.
Revised: Figure 2 shows our model.

Original: The idea underlying our model is to ﬁrst learn a representation of appliance event windows that separate the appliance type, z t,cat , from the shape of the energy signal, z t,cont in the ﬁgure.
Revised: The idea underlying our model is to ﬁrst learn a representation of appliance event windows that separate the appliance type, z t,cat , from the shape of the energy signal, z t,cont in the ﬁgure.

Original: This is achieved through the appliance energy encoder E .
Revised: This is achieved through the appliance energy encoder E .

Original: We can then use the appliance type to predict the location data through the location predictor L e , which is conditioned on z t,cat .
Revised: We can then use the appliance type to predict the location data through the location predictor L e , which is conditioned on z t,cat .

Original: Since people’s locations have information unrelated to appliance events, the total location predictor is a mixture of L e and a second module L g which captures event-independent location information.
Revised: Since people’s locations have information unrelated to appliance events, the total location predictor is a mixture of L e and a second module L g which captures event-independent location information.

Original: Below, we describe these modules in detail.
Revised: Below, we describe these modules in detail.

Original: Appliance Energy Encoder Given a window of aggregate energy signal y t : t + w= ( y t , y t +1 , . . .
Revised: Appliance Energy Encoder Given a window of aggregate energy signal y t : t + w= ( y t , y t +1 , . . .

Original: , y t + w 1 ) , 3 the encoder E encodes the series into an event vector z t .
Revised: , y t + w 1 ) , 3 the encoder E encodes the series into an event vector z t .

Original: We break the event vector into two parts: a categorical vector z t,cat and a continuous vector z t,cont .
Revised: We break the event vector into two parts: a categorical vector z t,cat and a continuous vector z t,cont .

Original: We aim to capture the appliance type with z t,cat (e.g. microwave vs. dishwasher), and use z t,cont to capture the variability within an appliance.
Revised: We aim to capture the appliance type with z t,cat (e.g. microwave vs. dishwasher), and use z t,cont to capture the variability within an appliance.

Original: A softmax layer is applied to z t,cat to ensure that it is a valid distribution over appliance types.
Revised: A softmax layer is applied to z t,cat to ensure that it is a valid distribution over appliance types.

Original: E is parametrized using convolution layers, with one fully connected layer to produce z t,cont and another for z t,cat .
Revised: E is parametrized using convolution layers, with one fully connected layer to produce z t,cont and another for z t,cat .

Original: We denote by θ E the parameters of the encoder.
Revised: We denote by θ E the parameters of the encoder.

Original: Location predictors We try to predict the location data conditioned on the appliance event, i.e. we predict a window of locations l = l t : t + w 2 = ( l t , l t +1 , . . .
Revised: Location predictors We try to predict the location data conditioned on the appliance event, i.e. we predict a window of locations l = l t : t + w 2 = ( l t , l t +1 , . . .

Original: , l t + w 2 ) centered around the appliance event.
Revised: , l t + w 2 ) centered around the appliance event.

Original: We handle multiple people’s locations with a mixture model.
Revised: We handle multiple people’s locations with a mixture model.

Original: Speciﬁcally, we use L e to predict locations related to energy events and L g to handle other locations.
Revised: Speciﬁcally, we use L e to predict locations related to energy events and L g to handle other locations.

Original: The ﬁnal prediction is a mixture of predictions from L e and L g :
Revised: The ﬁnal prediction is a mixture of predictions from L e and L g :

Original: [Equation]
Revised: [Equation]

Original: where p θ Le ( · ) is parametrized by L e with parameters θ L e , p θ Lg ( · ) is parametrized by L g with parameters θ L g , θ L = (cid:8) θ L e , θ L g (cid:9) , and c includes context features.
Revised: where p θ Le ( · ) is parametrized by L e with parameters θ L e , p θ Lg ( · ) is parametrized by L g with parameters θ L g , θ L = (cid:8) θ L e , θ L g (cid:9) , and c includes context features.

Original: We use the number of people in the window (reported by the location sensor), the time of day, and the day of the week as the context features.
Revised: We use the number of people in the window (reported by the location sensor), the time of day, and the day of the week as the context features.

Original: The weight α depends on the number of people in the current window α = 1 /P t .
Revised: The weight α depends on the number of people in the current window α = 1 /P t .

Original: To represent location data, we blur each location measurement with a Gaussian kernel on the x-y plane to create an image, and process the window of locations l t : t + w 2 into frames of images.
Revised: To represent location data, we blur each location measurement with a Gaussian kernel on the x-y plane to create an image, and process the window of locations l t : t + w 2 into frames of images.

Original: We reuse the notation l to represent frames of location images ∈ R | X |×| Y |×| T | , where | X | , | Y | , | T | are the number of discretized points on the x-y and time dimensions.
Revised: We reuse the notation l to represent frames of location images ∈ R | X |×| Y |×| T | , where | X | , | Y | , | T | are the number of discretized points on the x-y and time dimensions.

Original: By presenting location data as images, we also remove the variable P t while handling a variable number of people in each frame.
Revised: By presenting location data as images, we also remove the variable P t while handling a variable number of people in each frame.

Original: We choose p θ Le ( l | z t,cat ) to be a multivariate Gaussian with a diagonal covariance structure: p θ Le
Revised: We choose p θ Le ( l | z t,cat ) to be a multivariate Gaussian with a diagonal covariance structure: p θ Le

Original: ( l | z t,cat ) =
Revised: ( l | z t,cat ) =

Original: N ( l ; µ e , Σ ) =
Revised: N ( l ; µ e , Σ ) =

Original: (cid:81)
Revised: (cid:81)

Original: x,y,t N ( l x,y,t ; µ x,y,t , σ 2 x,y,t ) where µ e =
Revised: x,y,t N ( l x,y,t ; µ x,y,t , σ 2 x,y,t ) where µ e =

Original: L e ( z t,cat ; θ L e ) and choose σ x,y,t to be a constant.
Revised: L e ( z t,cat ; θ L e ) and choose σ x,y,t to be a constant.

Original: We use 3D deconvolution networks to model L e , which takes z t,cat as input and outputs the means of the location distributions.
Revised: We use 3D deconvolution networks to model L e , which takes z t,cat as input and outputs the means of the location distributions.

Original: We model p θ Lg ( · ) and L g in a similar way.
Revised: We model p θ Lg ( · ) and L g in a similar way.

Revised: The network implementation details are discussed in Appendix 8.4.

Original: During training, given a window of data ( l ( i ) , y ( i ) t : t + w 1 , c ( i ) ) , we minimize the negative log likelihood of the mixture distribution in predicting the locations:
Revised: During training, given a window of data ( l ( i ) , y ( i ) t : t + w 1 , c ( i ) ) , we minimize the negative log likelihood of the mixture distribution in predicting the locations:

Original: [Equation]
Revised: [Equation]

Original: Note that as the gradient ﬂows through z t,cat , the likelihood is a function of both θ E and θ L .
Revised: Note that as the gradient ﬂows through z t,cat , the likelihood is a function of both θ E and θ L .

Original: Hence, the encoder E also learns to encode the energy series based on the concurrent location data.
Revised: Hence, the encoder E also learns to encode the energy series based on the concurrent location data.

Original: Energy Decoder
Revised: Energy Decoder

Original: The decoder D takes both z t,cat and z t,cont and learns to reconstruct the original input energy series by predicting ˆ y t : t + w 1 .
Revised: The decoder D takes both z t,cat and z t,cont and learns to reconstruct the original input energy series by predicting ˆ y t : t + w 1 .

Original: We minimize the reconstruction loss during training:
Revised: We minimize the reconstruction loss during training:

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss encourages the encoder E to produce good initial vectors for L e to predict locations.
Revised: The reconstruction loss encourages the encoder E to produce good initial vectors for L e to predict locations.

Original: At the same time, it serves as a regularizer to prevent encoder E from generating meaningless vectors by overﬁtting location predictions.
Revised: At the same time, it serves as a regularizer to prevent encoder E from generating meaningless vectors by overﬁtting location predictions.

Original: Training We train all components to jointly optimize the location predictions and energy reconstruction.
Revised: Training We train all components to jointly optimize the location predictions and energy reconstruction.

Original: We minimize the total loss:
Revised: We minimize the total loss:

Original: L total =
Revised: L total =

Original: L loc + λ ∗ L rec where λ is a parameter to balance the two terms.
Revised: L loc + λ ∗ L rec where λ is a parameter to balance the two terms.

Original: The network implementation and training details are discussed in Appendix 8.4.
Revised: The network implementation and training details are discussed in Appendix 8.4.

Original: 4. C LUSTERING A PPLIANCE E VENTS WITH C ROSS -M ODAL P REDICTIONS
Revised: 4. C LUSTERING A PPLIANCE E VENTS WITH C ROSS -M ODAL P REDICTIONS

Original: Once the model is trained, we obtain for each window of energy data
Revised: Once the model is trained, we obtain for each window of energy data

Original: its appliance event vector
Revised: its appliance event vector

Original: z t,cat and its cross-modal location prediction p θ Le ( ·| z t,cat ) .
Revised: z t,cat and its cross-modal location prediction p θ Le ( ·| z t,cat ) .

Original: Next, we use these two vectors for clustering.
Revised: Next, we use these two vectors for clustering.

Original: We design a density-based clustering algorithm leveraging the cross-modal relation we learned.
Revised: We design a density-based clustering algorithm leveraging the cross-modal relation we learned.

Original: Our intuition is that activation events for the same appliance will cluster together since they have the same appliance type and the same location.
Revised: Our intuition is that activation events for the same appliance will cluster together since they have the same appliance type and the same location.

Original: We omit the cat notation below for brevity.
Revised: We omit the cat notation below for brevity.

Original: It is typically difﬁcult to cluster in a space learned by a neural encoder because the transformation is highly non-linear and the distance metric is not well-deﬁned.
Revised: It is typically difﬁcult to cluster in a space learned by a neural encoder because the transformation is highly non-linear and the distance metric is not well-deﬁned.

Original: We circumvent this problem by associating the encoded space with a Euclidean space, in which we can easily measure distance.
Revised: We circumvent this problem by associating the encoded space with a Euclidean space, in which we can easily measure distance.

Original: Speciﬁcally, for two event vectors z 1 and z 2 , we can measure their distance in the location space using p θ Le ( ·| z 1 ) and p θ Le ( ·| z 2 ) .
Revised: Speciﬁcally, for two event vectors z 1 and z 2 , we can measure their distance in the location space using p θ Le ( ·| z 1 ) and p θ Le ( ·| z 2 ) .

Original: The location prediction p θ Le ( ·| z i ) represents the likelihood of observing any location l x,y,t around the time of the appliance event.
Revised: The location prediction p θ Le ( ·| z i ) represents the likelihood of observing any location l x,y,t around the time of the appliance event.

Original: We found that for events related to human activities (e.g., turning on a kettle or microwave), p θ Le ( ·| z i ) shows a peak value at the location of the appliance in the x-y space at the time when a person interacted with the appliance.
Revised: We found that for events related to human activities (e.g., turning on a kettle or microwave), p θ Le ( ·| z i ) shows a peak value at the location of the appliance in the x-y space at the time when a person interacted with the appliance.

Original: For events not related to human activities (e.g. fridge cycles or random background events), p θ Le ( ·| z i ) has low values and is diffused.
Revised: For events not related to human activities (e.g. fridge cycles or random background events), p θ Le ( ·| z i ) has low values and is diffused.

Original: We deﬁne the location predictability score (or the conﬁdence of location prediction) as s ( z i ) =
Revised: We deﬁne the location predictability score (or the conﬁdence of location prediction) as s ( z i ) =

Original: max x,y,t p θ Le ( l x,y,t | z i ) , and the location distance D loc between two events as: D loc ( z 1 , z 2 ) =
Revised: max x,y,t p θ Le ( l x,y,t | z i ) , and the location distance D loc between two events as: D loc ( z 1 , z 2 ) =

Original: || ( x ∗ 1 − x ∗ 2 , y ∗ 1 − y ∗ 2 )
Revised: || ( x ∗ 1 − x ∗ 2 , y ∗ 1 − y ∗ 2 )

Original: || 2 , where ( x ∗ i , y ∗ i , t ∗ i ) =
Revised: || 2 , where ( x ∗ i , y ∗ i , t ∗ i ) =

Original: arg max x,y,t p θ Le ( l x,y,t | z i ) .
Revised: arg max x,y,t p θ Le ( l x,y,t | z i ) .

Original: Similarly, the neighborhood distance D nb between two events is deﬁned as D nb ( z 1 , z 2 ) =
Revised: Similarly, the neighborhood distance D nb between two events is deﬁned as D nb ( z 1 , z 2 ) =

Original: ||
Revised: ||

Original: z 1 − z
Revised: z 1 − z

Original: || 2 .
Revised: || 2 .

Original: Our clustering algorithm starts with a z i with high predictability score
Revised: Our clustering algorithm starts with a z i with high predictability score

Original: s ( z i ) .
Revised: s ( z i ) .

Original: It expands the cluster around z
Revised: It expands the cluster around z

Original: i ’s local neighborhood in the z space.
Revised: i ’s local neighborhood in the z space.

Original: It stops expanding if a neighbor’s location distance
Revised: It stops expanding if a neighbor’s location distance

Original: D loc is too far from the cluster center.
Revised: D loc is too far from the cluster center.

Original: If all neighbors of the current cluster are visited and none has a small enough D loc , we start a new cluster from another event with high predictability score.
Revised: If all neighbors of the current cluster are visited and none has a small enough D loc , we start a new cluster from another event with high predictability score.

Original: The algorithm is described formally in Algorithm 1.
Revised: The algorithm is described formally in Algorithm 1.

Original: We discuss the choice of parameters in Appendix 8.5.
Revised: We discuss the choice of parameters in Appendix 8.

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original:  D ATASET
Revised:  D ATASET

Original: We collected concurrent streams of aggregate energy signal and location data from 4 homes overmonths.
Revised: We collected concurrent streams of aggregate energy signal and location data from 4 homes overmonths.

Original: We use this dataset for our evaluation.
Revised: We use this dataset for our evaluation.

Original: To obtain ground truth labels of appliance events, we deployed programmable smart plugs on the power outlet associated with each appliance.
Revised: To obtain ground truth labels of appliance events, we deployed programmable smart plugs on the power outlet associated with each appliance.

Original: Since not all appliances can be measured by a smart plug (some appliances do not connect to a power outlet), we also developed a labeling tool for manual labeling.
Revised: Since not all appliances can be measured by a smart plug (some appliances do not connect to a power outlet), we also developed a labeling tool for manual labeling.

Original: The tool allows labelers to label appliance events from the aggregate energy signal, with the help of smart plug data and information collected from the home residents.
Revised: The tool allows labelers to label appliance events from the aggregate energy signal, with the help of smart plug data and information collected from the home residents.

Original: The choice of sensors and their sampling rates are detailed in Appendix 8.1.
Revised: The choice of sensors and their sampling rates are detailed in Appendix 8.1.

Original:  R ESULTS
Revised:  R ESULTS

Original: We evaluate our model and clustering algorithm on unsupervised appliance activation event detection and their learned appliance locations.
Revised: We evaluate our model and clustering algorithm on unsupervised appliance activation event detection and their learned appliance locations.

Original: 6. U NSUPERVISED A PPLIANCE E VENT D ETECTION
Revised: 6. U NSUPERVISED A PPLIANCE E VENT D ETECTION

Original: For appliance event detection, we compare with four baselines.
Revised: For appliance event detection, we compare with four baselines.

Original: Our method and two baselines have access to location information.
Revised: Our method and two baselines have access to location information.

Original: EL-Kmeans takes both energy and location data as input and directlyclusters them using k-means (Arthur & Vassilvitskii, 2007). E-only-Kmeans clusters only theenergy signal with k-means.
Revised: EL-Kmeans takes both energy and location data as input and directly only the signal with k-means.

Original: Methods with location information pre-ﬁlter the events and discard events without any location data, as they are unlikely to be activation events.
Revised: Methods with location information pre-ﬁlter the events and discard events without any location data, as they are unlikely to be activation events.

Original: The other two baselines only take the total energy signal as input: AFAMAP (Kolter & Jaakkola, 2012) uses factorial HMM, and
Revised: The other two baselines only take the total energy signal as input: AFAMAP (Kolter & Jaakkola, 2012) uses factorial HMM, and

Original: VarBOLT (Lange & Berges, 2018) uses a recurrent neural network to model aggregate appliance signals.
Revised: VarBOLT (Lange & Berges, 2018) uses a recurrent neural network to model aggregate appliance signals.

Original: We use publicly available implementations for these methods (implementation, a;b).
Revised: We use publicly available implementations for these methods (implementation, a;b).

Original: We use the same hyper-parameters for the network architecture, training, and clustering algorithm across all homes.
Revised: We use the same hyper-parameters for the network architecture, training, and clustering algorithm across all homes.

Original: As our clustering algorithm is non-parametric, we choose the same number of clusters that it discovers for other methods if possible.
Revised: As our clustering algorithm is non-parametric, we choose the same number of clusters that it discovers for other methods if possible.

Original: For VarBOLT, we report results usingclusters, since the training time grows exponentially with the number of clusters and training with more clusters is prohibitively slow.
Revised: For VarBOLT, we report results usingclusters, since the training time grows exponentially with the number of clusters and training with more clusters is prohibitively slow.

Original: As in past unsupervised work, we report the detection F1 scores based on the best cluster assignments with the ground truth appliances.
Revised: As in past unsupervised work, we report the detection F1 scores based on the best cluster assignments with the ground truth appliances.

Original: Table 1 shows that our method has an average detection F1 score of 73.2%, outperforming other baselines ranging from 4.1% to 24.4%.
Revised: Table 1 shows that our method has an average detection F1 score of 73.2%, outperforming other baselines ranging from 4.1% to 24.4%.

Original: As reported by Bonﬁgli et al.
Revised: As reported by Bonﬁgli et al.

Original: (2018), AFAMAP performs better when appliance-level data is available for training the HMMs.
Revised: (2018), AFAMAP performs better when appliance-level data is available for training the HMMs.

Original: In the unsupervised setting, however, its footprint extraction procedure does not always produce meaningful HMMs for individual appliances (Bonﬁgli et al., 2018;
Revised: In the unsupervised setting, however, its footprint extraction procedure does not always produce meaningful HMMs for individual appliances (Bonﬁgli et al., 2018;

Original: Beckel et al., 2014), causing degraded performance.
Revised: Beckel et al., 2014), causing degraded performance.

Original: VarBOLT’s training objective focuses on explaining the total amount of energy in a home.
Revised: VarBOLT’s training objective focuses on explaining the total amount of energy in a home.

Original: Thus, it often uses multiple components (clusters) to model appliances that are on for a long period (e.g. fridge, heater, and dryer/washer).
Revised: Thus, it often uses multiple components (clusters) to model appliances that are on for a long period (e.g. fridge, heater, and dryer/washer).

Original: These types of appliances generate many background events, making the algorithm focus less on activation events of other appliances.
Revised: These types of appliances generate many background events, making the algorithm focus less on activation events of other appliances.

Original: Comparing our method with baselines that also have location information (E-only-Kmeans and
Revised: Comparing our method with baselines that also have location information (E-only-Kmeans and

Original: EL-Kmeans), our approach still outperforms them signiﬁcantly.
Revised: EL-Kmeans), our approach still outperforms them signiﬁcantly.

Original: E-only-Kmeans performs better than
Revised: E-only-Kmeans performs better than

Original: AFAMAP and VarBOLT, showing that the presence of location data is highly related to activation events.
Revised: AFAMAP and VarBOLT, showing that the presence of location data is highly related to activation events.

Original: However, naively using the location data for clustering does not improve the results, as
Revised: However, naively using the location data for clustering does not improve the results, as

Original: EL-Kmeans performs similarly to E-only-Kmeans.
Revised: EL-Kmeans performs similarly to E-only-Kmeans.

Original: This is because not all location data is related to appliance events and vice versa.
Revised: This is because not all location data is related to appliance events and vice versa.

Original: Our approach “cleans up” the data by learning the relation between the two modalities and discovers clusters with strong cross-modal predictability.
Revised: Our approach “cleans up” the data by learning the relation between the two modalities and discovers clusters with strong cross-modal predictability.

Original: Table 2 shows a break down of our results for different appliances.
Revised: Table 2 shows a break down of our results for different appliances.

Original: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 6. A BLATION S TUDY
Revised: BLATION S TUDY

Original: We perform an ablation study to show our results are contributed by all components in our method.
Revised: We perform an ablation study to show our results are contributed by all components in our method.

Original: As shown in Table [Table] we compare our clustering algorithm (Method 1) with a different algorithm that concatenates the learned multi-modal embeddings ( z t,cat and p θ Le ( ·| z t,cat ) ) and directly clusters them with k-means (Method 2).
Revised: As shown in Table 3, we compare our clustering algorithm (Method 1) with a different algorithm that concatenates the learned multi-modal embeddings ( z t,cat and p θ Le ( ·| z t,cat ) ) and directly clusters them with k-means (Method 2).

Original: Our clustering algorithm is more effective than directly clustering the multi-modal embeddings, providing an improvement of 13.2% in the average F1 score.
Revised: Our clustering algorithm is more effective than directly clustering the multi-modal embeddings, providing an improvement of 13.2% in the average F1 score.

Original: This is because our clustering algorithm treats the two modalities differently.
Revised: This is because our clustering algorithm treats the two modalities differently.

Original: For location predictions, we can leverage our understanding of physical distance to set cluster boundaries.
Revised: For location predictions, we can leverage our understanding of physical distance to set cluster boundaries.

Original: For the energy embedding, since it is a non-linear mapping with no clear distance metric, we adopt a density propagation algorithm that only uses local neighborhood distance.
Revised: For the energy embedding, since it is a non-linear mapping with no clear distance metric, our algorithm iteratively groups together events in the embedding neighborhood that have approximately the same locations.

Original: Apart from our clustering algorithm, we evaluate the beneﬁts of our mixture component L g by experimenting with removing L g from the model, which reduces the F1 score by 4.1% (Method 1 vs 3).
Revised: Apart from our clustering algorithm, we evaluate the beneﬁts of our mixture component L g by experimenting with removing L g from the model, which reduces the F1 score by 4.1% (Method 1 vs 3).

Original: This shows the importance of having L g extract background motion to allow the location predictor, L e , to focus on modeling the person who interacts with the appliance. 
Revised: This shows the importance of having L g extract background motion to allow the location predictor, L e , to focus on modeling the person who interacts with the appliance. [Table]

Original:  We also consider removing both L g and L e , and clustering the input based only on the energy embedding z t,cat since there is no learned location predictions.
Revised: [Table] We also consider removing both L g and L e , and clustering the input based only on the energy embedding z t,cat since there is no learned location predictions.

Original: The results shown under Methoddemonstrate the importance of the location embedding generated by the combination of L g and L e .
Revised: The results shown under Methoddemonstrate the importance of the location embedding generated by the combination of L g and L e .

Original:  L EARNED A PPLIANCE L OCATIONS
Revised: 6. L EARNED A PPLIANCE L OCATIONS

Original: Our model also learns the locations where people interacted with appliances, which are typically close to the appliances’ physical locations (we discuss remotely activated appliances in Appendix 8.6).
Revised: Our model also learns the locations where people interacted with appliances, which are typically close to the appliances’ physical locations (we discuss remotely activated appliances in Appendix 8.6).

Original: For each appliance event, we take the location predicted by L e with the highest predictability score, and compare that with the ground truth appliance location measured by a laser meter.
Revised: For each appliance event, we take the location predicted by L e with the highest predictability score, and compare that with the ground truth appliance location measured by a laser meter.

Original: The average location prediction error is 0.77 meters with a standard deviation of 0.48 meters across homes.
Revised: The average location prediction error is 0.77 meters with a standard deviation of 0.48 meters across homes.

Original: Figure 4a shows the location predictions and their ground truth of several appliance events in Home 1.
Revised: Figure 4a shows the location predictions and their ground truth of several appliance events in Home 1.

Original: The corresponding energy signals are shown in Figure 4b - Figure 4e.
Revised: The corresponding energy signals are shown in Figure 4b - Figure 4e.

Original: The location information helps disambiguate appliances with similar energy signals.
Revised: The location information helps disambiguate appliances with similar energy signals.

Original: For example, although the hair dryer and kettle (Figure 4d and Figure 4e) have very similar energy signatures, their different locations (green and orange in Figure 4a) guide the model to encode their events differently.
Revised: For example, although the hair dryer and kettle (Figure 4d and Figure 4e) have very similar energy signatures, their different locations (green and orange in Figure 4a) guide the model to encode their events differently.

Original: [Figure]
Revised: [Figure]

Original:  L OCATION P REDICTIONS OF L e VS L g
Revised: 6. L OCATION P REDICTIONS OF L e VS L g

Original: We visualize the location predictions from the event-related predictor L e and the event-independent predictor L g to illustrate how they handle scenarios with multiple people.
Revised: We visualize the location predictions from the event-related predictor L e and the event-independent predictor L g to illustrate how they handle scenarios with multiple people.

Original: Figure 5 shows how the mixture design handles the two types of locations.
Revised: Figure 5 shows how the mixture design handles the two types of locations.

Original: Since L e is conditioned on energy events, it naturally learns to predict locations related to appliance events.
Revised: Since L e is conditioned on energy events, it naturally learns to predict locations related to appliance events.

Original: In this case, the location of the hair dryer is predicted by L e (Figure 5b).
Revised: In this case, the location of the hair dryer is predicted by L e (Figure 5b).

Original: On the other hand, L g predicts the typical locations people tend to stay (e [Figure] g., the couch in Figure 5c) based on the context.
Revised: On the other hand, L g predicts the typical locations people tend to stay (e.g., the couch in Figure 5c) based on the context.

Original: Having L g to explain the other locations helps L e focus on learning the event-related locations.
Revised: Having L g to explain the other locations helps L e focus on learning the event-related locations

Original: [Figure]

Original:  6. C ONTEXTUAL L OCATION I NFORMATION AND C LUSTER V ISUALIZATIONS
Revised: . 6. C ONTEXTUAL L OCATION I NFORMATION AND C LUSTER V ISUALIZATIONS

Original: In Appendix 8. [Figure] we discuss emerging contextual relations between indoor locations through learning cross-modal predictions.
Revised: In Appendix 8.2, we discuss emerging contextual relations between indoor locations through learning cross-modal predictions.

Original: We also visualize the learned event vectors to shed light on the design rationales behind our clustering algorithm in Appendix 8.3.
Revised: We also visualize the learned event vectors to shed light on the design rationales behind our clustering algorithm in Appendix 8.3.

Revised: 6 4 2 0 2x (meters)      y

Revised: ( m e t e r s )

Revised: (a) Observed locations (two people) 6 4 2 0 2x (meters)

Revised: y

Original:   C ONCLUSION
Revised: ( m e t e r s ) Hair dryer (b) L e ’s prediction (event-related) 6 4 2 0 2x (meters)      y ( m e t e r s ) Couch (c) L g ’s prediction (other locations)  C ONCLUSION

Original: We introduced a self-supervised solution for learning appliance usage patterns in homes.
Revised: We introduced a self-supervised solution for learning appliance usage patterns in homes.

Original: We infer appliance usage by learning from data streams of two modalities: the total energy consumed by the home and the residents’ location data.
Revised: We infer appliance usage by learning from data streams of two modalities: the total energy consumed by the home and the residents’ location data.

Original: Our approach improves on unsupervised appliance event detection signiﬁcantly, and learns appliance locations and usage patterns without any supervision.
Revised: Our approach improves on unsupervised appliance event detection signiﬁcantly and learns appliance locations and usage patterns without any supervision

Original:  A PPENDIX 8. S ENSORS D ETAILS
Revised:  A PPENDIX 8. S ENSORS D ETAILS

Original: In this section, we describe details of the sensors used in our dataset collection.
Revised: In this section, we describe details of the sensors used in our dataset collection.

Original: Aggregate energy signal For ﬂexible data collection, we install a sensor (emonPi) at the main circuit breaker in each house as a proxy for the utility meter.
Revised: Aggregate energy signal For ﬂexible data collection, we install a sensor (emonPi) at the main circuit breaker in each house as a proxy for the utility meter.

Original: We programmed the sensor to collect the raw aggregate energy signal at 1.2 kHz.
Revised: We programmed the sensor to collect the raw aggregate energy signal at 1.2 kHz.

Original: We down-sampled the data to 10 Hz for our problem to emulate the achievable data rate from a utility meter hardware (Armel et al., 2013).
Revised: We down-sampled the data to 10 Hz for our problem to emulate the achievable data rate from a utility meter hardware (Armel et al., 2013).

Original: Location data The wireless location sensor is built on a design similar to Hsu et al.
Revised: Location data The wireless location sensor is built on a design similar to Hsu et al.

Original: It is a single stand-alone sensor that hangs on the wall, and passively collects multiple people’s locations with decimeter-level accuracy.
Revised: It is a single stand-alone sensor that hangs on the wall, and passively collects multiple people’s locations with decimeter-level accuracy.

Original: We down-sampled the location streams to 1 Hz.
Revised: We down-sampled the location streams to 1 Hz.

Original: Appliance-level data (for ground truth labeling)
Revised: Appliance-level data (for ground truth labeling)

Original: We use TP-Link smart plugs (TP-link) with energy monitoring features for collecting appliance-level data.
Revised: We use TP-Link smart plugs (TP-link) with energy monitoring features for collecting appliance-level data.

Original: We wrote custom software using available APIs to collect appliance energy signals at 1 Hz.
Revised: We wrote custom software using available APIs to collect appliance energy signals at 1 Hz.

Original: For appliances that cannot be connected to a smart plug, we asked the residents to write down appliance usage times to help with manual labeling.
Revised: For appliances that cannot be connected to a smart plug, we asked the residents to write down appliance usage times to help with manual labeling.

Original: 8. C ONTEXTUAL L OCATION I NFORMATION VIA L EARNED A PPLIANCE E VENTS
Revised: 8. C ONTEXTUAL L OCATION I NFORMATION VIA L EARNED A PPLIANCE E VENTS

Original: By analyzing the location predictions of L e conditioned on different appliance events, we also discover interesting contextual relations between different indoor locations.
Revised: By analyzing the location predictions of L e conditioned on different appliance events, we also discover interesting contextual relations between different indoor locations.

Original: Figure 6 visualizes the location predictions at different frames around a kettle event.
Revised: Figure 6 visualizes the location predictions at different frames around a kettle event.

Original: We plot the per-frame location predictability score (or prediction conﬁdence) over time in Figure 6a.
Revised: We plot the per-frame location predictability score (or prediction conﬁdence) over time in Figure 6a.

Original: The score peaks around t = 0 s, the time of the event.
Revised: The score peaks around t = 0 s, the time of the event.

Original: This is because when people turn on a kettle, they may approach it from different locations, but the location when they push the button is consistent and can be predicted conﬁdently.
Revised: This is because when people turn on a kettle, they may approach it from different locations, but the location when they push the button is consistent and can be predicted conﬁdently.

Original: As a result, the prediction at t = 0 s correctly shows the kettle’s location (Figure 6d).
Revised: As a result, the prediction at t = 0 s correctly shows the kettle’s location (Figure 6d).

Original: Interestingly, a smaller peak of predictability score shows at t = − 10 s in Figure 6a.
Revised: Interestingly, a smaller peak of predictability score shows at t = − 10 s in Figure 6a.

Original: If we look at the location prediction from t = − 10 s to t = 0 s (Figure 6b - Figure 6d), we see how the prediction moves from the sink to the kettle 5 .
Revised: If we look at the location prediction from t = − 10 s to t = 0 s (Figure 6b - Figure 6d), we see how the prediction moves from the sink to the kettle 7 .

Original: This is because people often ﬁll water at the sink before starting the kettle.
Revised: This is because people often ﬁll water at the sink before starting the kettle.

Original: Through learning the cross-modal relation, contextual information among locations also emerges as different appliance events are discovered.
Revised: Through learning the cross-modal relation, contextual information among locations also emerges as different appliance events are discovered.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: p r e d i c t i o n c o n f i d e n c e
Revised: p r e d i c t i o n c o n f i d e n c e

Original: [Figure]
Revised: Fill water at sink

Original: Turn kettle on (a) Prediction conﬁdence over time
Revised: Turn kettle on (a) Prediction conﬁdence over time

Original: [Figure]
Revised: [Figure]

Original: 8. V ISUALIZATION OF THE L EARNED E VENT V ECTORS AND L OCATION P REDICTABILITY
Revised: 8. V ISUALIZATION OF THE L EARNED E VENT V ECTORS AND L OCATION P REDICTABILITY

Original: To illustrate what the model learns and the design rationales behind our clustering algorithm, we visualize the space of the learned event vectors z t,cat and their location predictability score
Revised: To illustrate what the model learns and the design rationales behind our clustering algorithm, we visualize the space of the learned event vectors z t,cat and their location predictability score

Original: s ( z ) .
Revised: s ( z ) .

Original: Figure 7 shows the t-SNE (Maaten & Hinton, 2008) visualization of the event vectors on a 2dimensional space.
Revised: Figure 7 shows the t-SNE (Maaten & Hinton, 2008) visualization of the event vectors on a 2dimensional space.

Original: We color coded the events with three metrics: location predictability scores (Figure 7a), cluster ID discovered by our algorithm (Figure 7b), and ground truth label (Figure 7c).
Revised: We color coded the events with three metrics: location predictability scores (Figure 7a), cluster ID discovered by our algorithm (Figure 7b), and ground truth label (Figure 7c).

Original: The predictability score depends on how strongly an appliance event co-occurred with a particular location.
Revised: The predictability score depends on how strongly an appliance event co-occurred with a particular location.

Original: As shown in Figure 7a, most appliances related to human activities have high predictability scores (e.g., kettle, hairdryer, microwave, coffee machine, etc).
Revised: As shown in Figure 7a, most appliances related to human activities have high predictability scores (e.g., kettle, hairdryer, microwave, coffee machine, etc).

Original: On the other hand, appliances that cycle in the background (e.g., heater) have very low predictability.
Revised: On the other hand, appliances that cycle in the background (e.g., heater) have very low predictability.

Original: The stove has many clusters of background events.
Revised: The stove has many clusters of background events.

Original: This is because when the stove is on, it cycles between a few power levels, and the cycle durations depend on the heating levels.
Revised: This is because when the stove is on, it cycles between a few power levels, and the cycle durations depend on the heating levels.

Original: Interestingly, we found that stove clusters with higher power levels (“stove-big-cycle”) also have high predictability scores, while others with cycling states (“stove-cycle”) show low scores.
Revised: Interestingly, we found that stove clusters with higher power levels (“stove-big-cycle”) also have high predictability scores, while others with cycling states (“stove-cycle”) show low scores.

Original: This is likely because people are next to the stove more often when the heating level is high.
Revised: This is likely because people are next to the stove more often when the heating level is high.

Original: We can also see that without clustering using both location predictions and event vectors, it is hard to separate some of the cluster boundaries.
Revised: We can also see that without clustering using both location predictions and event vectors, it is hard to separate some of the cluster boundaries.

Original: Besides, learning to relate energy events to location data enables us to measure the distances of events in a well-deﬁned physical space.
Revised: Besides, learning to relate energy events to location data enables us to measure the distances of events in a well-deﬁned physical space.

Original: [Figure]
Revised: [Figure]

Original: 8. N ETWORK I MPLEMENTATION AND T RAINING D ETAILS
Revised: 8. N ETWORK I MPLEMENTATION AND T RAINING D ETAILS

Original: In this section, we provide implementation and training details of our neural network model.
Revised: In this section, we provide implementation and training details of our neural network model.

Original: We use convolution and deconvolution layers for the energy encoder and decoder.
Revised: We use convolution and deconvolution layers for the energy encoder and decoder.

Original: Each module haslayers with a kernel size of 3 and a stride of 2.
Revised: Each module haslayers with a kernel size of 3 and a stride of 2.

Original: We choose the dimensions of z t,cat and z t,cont to be 128 and 3.
Revised: We choose the dimensions of z t,cat and z t,cont to be 128 and 3.

Original: The location predictors have 5 layers of 3D deconvolution with a kernel size ofand a stride of 2 in each dimension.
Revised: The location predictors have 5 layers of 3D deconvolution with a kernel size ofand a stride of 2 in each dimension.

Original: The frames of location images for each time window have× 32 × 32 pixels.
Revised: The frames of location images for each time window have× 32 × 32 pixels.

Original: We discretize the x, y, and time dimensions into 32 points, where the range of the x-y dimensions are 10 meters and the time dimension has 32 seconds.
Revised: We discretize the x, y, and time dimensions into 32 points, where the range of the x-y dimensions are 10 meters and the time dimension has 32 seconds.

Original: The neural networks are implemented in Tensorﬂow (Abadi et al., 2016).
Revised: The neural networks are implemented in Tensorﬂow (Abadi et al., 2016).

Original: For training, we use the Adam (Kingma & Ba, 2014) optimizer with a learning rate of 0.001 and a batch size of 64.
Revised: For training, we use the Adam (Kingma & Ba, 2014) optimizer with a learning rate of 0.001 and a batch size of 64.

Original: 8. C LUSTERING P ARAMETERS
Revised: 8. C LUSTERING P ARAMETERS

Original: In all experiments, we set η D loc = 0 .
Revised: In all experiments, we set η D loc = 0 .

Original: 4 meters, η z = 0 .
Revised: 4 meters, η z = 0 .

Original: 03 , η s = 0 .
Revised: 03 , η s = 0 .

Original: 2 , and N min = 10 .
Revised: 2 , and N min = 10 .

Original: These values are chosen based on physical and computational constraints.
Revised: These values are chosen based on physical and computational constraints.

Original: The value of η D loc is based on the minimum physical separation between two appliances.
Revised: The value of η D loc is based on the minimum physical separation between two appliances.

Original: The value of η z only affects the search space in each iteration, and is chosen to be small for computational efﬁciency.
Revised: The value of η z only affects the search space in each iteration, and is chosen to be small for computational efﬁciency.

Original: The minimum predictability score η s is chosen based on a validation set from one of the homes.
Revised: The minimum predictability score η s is chosen based on a validation set from one of the homes.

Original: The value of N min is set to 10 to say that we need the appliance to appear in the data at least 10 times before we trust that it is a real appliance.
Revised: The value of N min is set to 10 to say that we need the appliance to appear in the data at least 10 times before we trust that it is a real appliance.

Original: 8. R EMOTELY A CTIVATED A PPLIANCES
Revised: 8. L IMITATIONS

Revised: We discuss the limitations of our approach in this section.

Revised: One limitation is that some remotely activated appliances may not have predictable locations.

Original: From our experience collecting the dataset, the vast majority of the appliances used on a daily basis (Table 2) require human interaction.
Revised: However, from our experience collecting the dataset, the vast majority of the appliances used on a daily basis (Table 2) require human interaction.

Original: For example, a person has to put food into a microwave before turning it on, to hold a hair dryer while drying hair, and to push a button to get a coffee machine running.
Revised: For example, a person has to put food into a microwave before turning it on, to hold a hair dryer while drying hair, and to push a button to get a coffee machine running.

Original: Even for an appliance with a remote controller, as long as the person has a regular place to interact with the appliance from (e.g., always turning the TV on while sitting on the couch), our model can still learn to predict the location of interaction.
Revised: Even for an appliance with a remote controller, as long as the person has a regular place to interact with the appliance from (e.g., always turning the TV on while sitting on the couch), our model can still learn to predict the location of interaction.

Revised: Another limitation is that our location sensor has a limited coverage area (aroundfeet in radius).

Revised: This is enough to cover a typical one-bedroom apartment.

Revised: For a larger house, one could deploy a second sensor, similarly to how a WiFi repeater extends the coverage area.