Original: L EARNING P ROCEDURAL A BSTRACTIONS AND E VALUATING D ISCRETE L ATENT T EMPORAL S TRUCTURE
Revised: L EARNING P ROCEDURAL A BSTRACTIONS AND E VALUATING D ISCRETE L ATENT T EMPORAL S TRUCTURE

Original: A BSTRACT
Revised: A BSTRACT

Original: Clustering methods and latent variable models are often used as tools for pattern mining and discovery of latent structure in time-series data.
Revised: Clustering methods and latent variable models are often used as tools for pattern mining and discovery of latent structure in time-series data.

Original: In this work, we consider the problem of learning procedural abstractions from possibly highdimensional observational sequences, such as video demonstrations.
Revised: In this work, we consider the problem of learning procedural abstractions from possibly highdimensional observational sequences, such as video demonstrations.

Original: Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps.
Revised: Given a dataset of time-series, the goal is to identify the latent sequence of steps common to them and label each time-series with the temporal extent of these procedural steps.

Original: We introduce a hierarchical Bayesian model called P RISM that models the realization of a common procedure across multiple time-series and can recover procedural abstractions without supervision.
Revised: We introduce a hierarchical Bayesian model called P RISM that models the realization of a common procedure across multiple time-series and can recover procedural abstractions without supervision.

Original: We also bring to light two characteristics ignored by traditional evaluation criteria when evaluating latent temporal labelings ( temporal clusterings ) – segment structure , and repeated structure – and develop new metrics tailored to their evaluation.
Revised: We also bring to light two characteristics ignored by traditional evaluation criteria when evaluating latent temporal labelings ( temporal clusterings ) – segment structure , and repeated structure – and develop new metrics tailored to their evaluation.

Original: We demonstrate that our metrics improve interpretability and ease of analysis for evaluation on benchmark time-series datasets.
Revised: We demonstrate that our metrics improve interpretability and ease of analysis for evaluation on benchmark time-series datasets.

Original: Results on benchmark and video datasets indicate that P RISM outperforms standard sequence models as well as state-of-the-art techniques in identifying procedural abstractions
Revised: Results on benchmark and video datasets indicate that P RISM outperforms standard sequence models as well as state-of-the-art techniques in identifying procedural abstractions

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental problem in machine learning is the discovery of structure in unsupervised data.
Revised: A fundamental problem in machine learning is the discovery of structure in unsupervised data.

Original: In this work we are particularly interested in uncovering the latent structure in procedural data – potentially high-dimensional observational time-series resulting from some latent sequence of events.
Revised: In this work we are particularly interested in uncovering the latent structure in procedural data – potentially high-dimensional observational time-series resulting from some latent sequence of events.

Original: For example, a video showing how to change a tire might involve jacking up the car, removing one nut from the wheel, removing a second nut, etc.
Revised: For example, a video showing how to change a tire might involve jacking up the car, removing one nut from the wheel, removing a second nut, etc.

Original: There exists an enormous wealth of videos of procedures available and inferring the latent structure of such videos could be useful in a huge range of applications, from supporting search queries (”ﬁnd all segments where someone jacks up the car”) to adding to nascent work on learning from observation (Borsa et al., 2017; Stadie et al., 2017;
Revised: There exists an enormous wealth of videos of procedures available and inferring the latent structure of such videos could be useful in a huge range of applications, from supporting search queries (”ﬁnd all segments where someone jacks up the car”) to adding to nascent work on learning from observation (Borsa et al., 2017; Stadie et al., 2017;

Original: Torabi et al., 2018) in which robots may be taught merely by observing a procedure performed in a video.
Revised: Torabi et al., 2018) in which robots may be taught merely by observing a procedure performed in a video.

Original: Procedural learning is related to activity recognition and the broader ﬁeld of latent temporal dynamical learning, but focuses on the simpler but still ubiquitous setting where the latent activity sequence is a ﬁxed procedure.
Revised: Procedural learning is related to activity recognition and the broader ﬁeld of latent temporal dynamical learning, but focuses on the simpler but still ubiquitous setting where the latent activity sequence is a ﬁxed procedure.

Original: As we started to develop methods for performing latent procedure inference from temporal data, we considered how to evaluate our resulting methods.
Revised: As we started to develop methods for performing latent procedure inference from temporal data, we considered how to evaluate our resulting methods.

Original: One important evaluation protocol for unsupervised learning methods is external evaluation , where predicted labelings of the discrete variables are compared to ground-truth labels (Rosenberg & Hirschberg, 2007).
Revised: One important evaluation protocol for unsupervised learning methods is external evaluation , where predicted labelings of the discrete variables are compared to ground-truth labels (Rosenberg & Hirschberg, 2007).

Original: But the precise criteria for this evaluation is critical.
Revised: But the precise criteria for this evaluation is critical.

Original: Though there are an enormous number of methods for modeling and inferring latent structure in dynamical systems, including Hidden Markov models, dynamic Bayesian networks, and temporal clustering for time-series data e.g. Fox et al.
Revised: Though there are an enormous number of methods for modeling and inferring latent structure in dynamical systems, including Hidden Markov models, dynamic Bayesian networks, and temporal clustering for time-series data e.g. Fox et al.

Original: (2008b; 2014); Krishnan et al.
Revised: (2008b; 2014); Krishnan et al.

Original: Linderman et al.
Revised: Linderman et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2008; 2013); Vidal & Favaro (2014), the external evaluation for such approaches is typically done using the same clustering metrics that are used for non-temporal data: metrics like the normalized mutual information (NMI) (Strehl & Ghosh, 2002) or by computing cluster correspondences using the Munkres algorithm (Munkres, 1957).
Revised: (2008; 2013); Vidal & Favaro (2014), the external evaluation for such approaches is typically done using the same clustering metrics that are used for non-temporal data: metrics like the normalized mutual information (NMI) (Strehl & Ghosh, 2002) or by computing cluster correspondences using the Munkres algorithm (Munkres, 1957).

Original: Unfortunately such metrics disregard temporal information and can therefore make it hard to assess important considerations that arise in latent temporal structure extraction, including in procedural inference.
Revised: Unfortunately such metrics disregard temporal information and can therefore make it hard to assess important considerations that arise in latent temporal structure extraction, including in procedural inference.

Original: [Figure]
Revised: [Figure]

Original: To address the limitations of prior evaluation criteria for the external evaluation of discrete latent temporal structure, our ﬁrst contribution is the creation of new criteria.
Revised: To address the limitations of prior evaluation criteria for the external evaluation of discrete latent temporal structure, our ﬁrst contribution is the creation of new criteria.

Original: We identify at least two key aspects of temporal latent structure discovery that are not captured well with existing non-temporal metrics: segment structure , the temporal durations between transitions from one discrete latent state to another, and repeated structure , where a particular latent variable may appear in multiple segments.
Revised: We identify at least two key aspects of temporal latent structure discovery that are not captured well with existing non-temporal metrics: segment structure , the temporal durations between transitions from one discrete latent state to another, and repeated structure , where a particular latent variable may appear in multiple segments.

Original: Figure1 shows an example latent sequence associated with a person performing  cpr  – the colors denote the activity type (such as blowing air into the lungs) and the temporal duration of a ﬁxed activity type is shown as a continuous block of color.
Revised: Figure1 shows an example latent sequence associated with a person performing cardiopulmonary resuscitation ( cpr ) – the colors denote the activity type (such as blowing air into the lungs) and the temporal duration of a ﬁxed activity type is shown as a continuous block of color.

Original: Our new metrics can evaluate both segment structure and repeated structure, and we later demonstrate how these metrics may be of broader interest in latent temporal dynamical learning, as they can illustrate and illuminate different strengths and weaknesses of existing work on common benchmark datasets.
Revised: Our new metrics can evaluate both segment structure and repeated structure, and we later demonstrate how these metrics may be of broader interest in latent temporal dynamical learning, as they can illustrate and illuminate different strengths and weaknesses of existing work on common benchmark datasets.

Original: Equipped with new measures for evaluating success, we return to our key goal, inferring the latent procedure observed in a set of high dimensional demonstrations.
Revised: Equipped with new measures for evaluating success, we return to our key goal, inferring the latent procedure observed in a set of high dimensional demonstrations.

Original: Past expressive models like Hidden
Revised: Past expressive models like Hidden

Original: Markov Models can struggle in this setting, since the resulting procedure may not be Markov: while the setting can be forced to appear Markov by employing the history as part of the state, this can reduce sample efﬁciency (by artiﬁcially increasing the latent space size).
Revised: Markov models can struggle in this setting, since the resulting procedure may not be Markov: while the setting can be forced to appear Markov by employing the history as part of the state, this can reduce sample efﬁciency (by artiﬁcially increasing the latent space size).

Original: Additionally assuming the system is Markov when it is not can mean inferring stochastic latent dynamical structure when it is in fact deterministic.
Revised: Additionally assuming the system is Markov when it is not can mean inferring stochastic latent dynamical structure when it is in fact deterministic.

Original: Recent work (Sener & Yao, 2018) tackles a closely related setting, but does not allow repeated primitives, which is a key aspect of many procedures.
Revised: Recent work (Sener & Yao, 2018) tackles a closely related setting, but does not allow repeated primitives, which is a key aspect of many procedures.

Original: Instead we introduce a new generative Bayesian model (P RISM ) for identifying procedural abstractions without supervision.
Revised: Instead we introduce a new generative Bayesian model (P RISM ) for identifying procedural abstractions without supervision.

Original: Results on datasets for procedural tasks – including surgical demonstrations and how-to videos – show that our method can learn procedural abstractions effectively compared with state-of-the-art methods and popular baselines, and outperform prior work on both our new metrics and classic metrics when repeated structure is present.
Revised: Results on datasets for procedural tasks – including surgical demonstrations and how-to videos – show that our method can learn procedural abstractions effectively compared with state-of-the-art methods and popular baselines, and outperform prior work on both our new metrics and classic metrics when repeated structure is present.

Original:  D EFINITIONS AND S ETTING
Revised:  D EFINITIONS AND S ETTING

Original: Temporal Clustering.
Revised: Temporal Clustering.

Original: A time-series X i =
Revised: A time-series X i =

Original: ( x i 1 , x i 2 , . . .
Revised: ( x i 1 , x i 2 , . . .

Original: , x im i ) , i ∈ [ n ] is a sequence of feature vectors (or items) of length m i with x it ∈ R d .
Revised: , x im i ) , i ∈ [ n ] is a sequence of feature vectors (or items) of length m i with x it ∈ R d .

Original: A temporal clustering C =
Revised: A temporal clustering C =

Original: ( c 1 , . . .
Revised: ( c 1 , . . .

Original: , c m ) is a sequence of cluster labels with c t ∈ Γ C where Γ C is a set of cluster labels.
Revised: , c m ) is a sequence of cluster labels with c t ∈ Γ C where Γ C is a set of cluster labels.

Original: Temporal clusterings map each item in the feature trajectory X to a cluster label in Γ C .
Revised: Temporal clusterings map each item in the feature trajectory X to a cluster label in Γ C .

Original: We use C i as shorthand to denote the temporal clustering generated by some method for X i .
Revised: We use C i as shorthand to denote the temporal clustering generated by some method for X i .

Original: The ground truth temporal clustering for X i will be denoted by G i , with corresponding ground truth label set Γ G .
Revised: The ground truth temporal clustering for X i will be denoted by G i , with corresponding ground truth label set Γ G .

Original: For simplicity, we deﬁne C a : b
Revised: For simplicity, we deﬁne C a : b

Original: =
Revised: =

Original: ( c a , c a +1 , . . .
Revised: ( c a , c a +1 , . . .

Original: , c b ) where 1 ≤ a ≤ b ≤ m are time indices.
Revised: , c b ) where 1 ≤ a ≤ b ≤ m are time indices.

Original: For α ∈ Γ C , let
Revised: For α ∈ Γ C , let

Original: C [ α ] =
Revised: C [ α ] =

Original: { t | c t =
Revised: { t | c t =

Original: α, 1 ≤ t ≤ m } be the set of time indices in C whose cluster labels equal α .
Revised: α, 1 ≤ t ≤ m } be the set of time indices in C whose cluster labels equal α .

Original: C
Revised: C

Original: [ α ] is the cluster or partition (in the standard sense) corresponding to the label α .
Revised: [ α ] is the cluster or partition (in the standard sense) corresponding to the label α .

Original: Temporal clusterings are distinct from time-series segmentation (Chung et al., 2004) and changepoint detection (Killick et al., 2012), which consider only identifying boundaries of different dynamic regimes in the time-series.
Revised: Temporal clusterings are distinct from time-series segmentation (Chung et al., 2004) and changepoint detection (Killick et al., 2012), which consider only identifying boundaries of different dynamic regimes in the time-series.

Original: Segment.
Revised: Segment.

Original: A segment is a contiguous sequence of identical cluster labels in C .
Revised: A segment is a contiguous sequence of identical cluster labels in C .

Original: Each cluster in C may be split across multiple segments.
Revised: Each cluster in C may be split across multiple segments.

Original: We represent each segment as a pair of time indices ( a, b ) , where a represents the start time-index of the segment while b the end time-index (both included).
Revised: We represent each segment as a pair of time indices ( a, b ) , where a represents the start time-index of the segment while b the end time-index (both included).

Original: Formally, we let S ( C , α ) =
Revised: Formally, we let S ( C , α ) =

Original: { ( a, b ) |C a : b
Revised: { ( a, b ) |C a : b

Original: = ( c a = α, c a +1 = α, . . .
Revised: = ( c a = α, c a +1 = α, . . .

Original: , c b = α ) , 1 ≤ a ≤ b ≤ m } be a function
Revised: , c b = α ) , 1 ≤ a ≤ b ≤ m } be a function

Original: [Figure]
Revised: [Figure]

Original: that maps a temporal clustering C and cluster label α ∈ Γ to the set of segments associated with α in
Revised: that maps a temporal clustering C and cluster label α ∈ Γ to the set of segments associated with α in

Original: C .
Revised: C .

Original: We let S C
Revised: We let S C

Original: =
Revised: =

Original: (cid:83) α ∈ Γ C S ( C , α ) be the set of all segments in C .
Revised: (cid:83) α ∈ Γ C S ( C , α ) be the set of all segments in C .

Original: Procedure.
Revised: Procedure.

Original: Let P be a function that removes running duplicates from a sequence of labels, yielding a sequence of tokens .
Revised: Let P be a function that removes running duplicates from a sequence of labels, yielding a sequence of tokens .

Original: For instance, P (A , A , A , B , B , B , C , C , C , A , A , B) =
Revised: For instance, P (A , A , A , B , B , B , C , C , C , A , A , B) =

Original: (A , B , C , A , B) yielding 5 tokens.
Revised: (A , B , C , A , B) yielding 5 tokens.

Original: For a temporal clustering C , we deﬁne its corresponding procedure to be P ( C ) .
Revised: For a temporal clustering C , we deﬁne its corresponding procedure to be P ( C ) .

Original: The procedure captures the primitives (as tokens) present in the temporal clustering and the sequence in which they occur.
Revised: The procedure captures the primitives (as tokens) present in the temporal clustering and the sequence in which they occur.

Original: We say that
Revised: We say that

Original: C exhibits repeated structure if atleast one primitive is reused in C .
Revised: C exhibits repeated structure if atleast one primitive is reused in C .

Original: This corresponds to having multiple tokens of the same cluster label in P ( C ) .
Revised: This corresponds to having multiple tokens of the same cluster label in P ( C ) .

Original: The procedure (A , B , C , A , B) contains repeated structure since both A and B are reused.
Revised: The procedure (A , B , C , A , B) contains repeated structure since both A and B are reused.

Original: We also deﬁne a weight function
Revised: We also deﬁne a weight function

Original: W that counts the number of running duplicates in a sequence e.g.
Revised: W that counts the number of running duplicates in a sequence e.g.

Original: W (A , A , A , B , B , B , C , C , C , A , A , B) =
Revised: W (A , A , A , B , B , B , C , C , C , A , A , B) =

Original: Notice that
Revised: Notice that

Original: W computes the length of each segment in the sequence.
Revised: W computes the length of each segment in the sequence.

Original: Weights in
Revised: Weights in

Original: W ( C ) thus have a one-to-one correspondence with tokens in P ( C ) , equaling the length of each token in the procedure.
Revised: W ( C ) thus have a one-to-one correspondence with tokens in P ( C ) , equaling the length of each token in the procedure.

Original: Taken together W ( C ) , P ( C ) can be used to recover C .
Revised: Taken together W ( C ) , P ( C ) can be used to recover C .

Original: Lastly, we let H ( P 1 , P 2 , W 1 , W 2 ) be a similarity measure between 2 temporal clusterings represented by P 1 , W 1 and P 2 , W 2 respectively.
Revised: Lastly, we let H ( P 1 , P 2 , W 1 , W 2 ) be a similarity measure between 2 temporal clusterings represented by P 1 , W 1 and P 2 , W 2 respectively.

Original: Fig 2 illustrates these ideas for a temporal clustering deﬁned over the label set Γ =
Revised: Fig 2 illustrates these ideas for a temporal clustering deﬁned over the label set Γ =

Original: { A , B , C , D } .
Revised: { A , B , C , D } .

Original: The procedure exhibits repeated structure since B has (atleast) 2 tokens/segments corresponding to it.
Revised: The procedure exhibits repeated structure since B has (atleast) 2 tokens/segments corresponding to it.

Original:  E VALUATING L ATENT T EMPORAL S TRUCTURE
Revised:  E VALUATING L ATENT T EMPORAL S TRUCTURE

Original: External clustering evaluation is the process of comparing a clustering to a ground-truth labeling in order to evaluate it.
Revised: External clustering evaluation is the process of comparing a clustering to a ground-truth labeling in order to evaluate it.

Original: Unfortunately, as we discuss below, using traditional clustering evaluation criteria is problematic for evaluating temporal clusterings.
Revised: Unfortunately, as we discuss below, using traditional clustering evaluation criteria is problematic for evaluating temporal clusterings.

Original: Existing evaluation criteria.
Revised: Existing evaluation criteria.

Original: The use of standard clustering metrics for evaluation is prevalent in prior work that attempts to evaluate latent temporal structure (Zhou et al., 2008; 2013;
Revised: The use of standard clustering metrics for evaluation is prevalent in prior work that attempts to evaluate latent temporal structure (Zhou et al., 2008; 2013;

Original: Fox et al., 2008b; 2009; 2014; Krishnan et al., 2015; Sener & Yao, 2018; Hoai & De la Torre, 2012).
Revised: Fox et al., 2008b; 2009; 2014; Krishnan et al., 2015; Sener & Yao, 2018; Hoai & De la Torre, 2012).

Original: For a temporal clustering C the widely used purity (Rosenberg & Hirschberg, 2007) metric penalizes the presence of items from different ground truth labels in the same cluster.
Revised: For a temporal clustering C the widely used purity (Rosenberg & Hirschberg, 2007) metric penalizes the presence of items from different ground truth labels in the same cluster.

Original: This is a desirable property that a clustering evaluation criterion should have.
Revised: This is a desirable property that a clustering evaluation criterion should have.

Original: Purity is deﬁned as purity = 1 m (cid:80) α ∈ Γ C max β ∈ Γ G |G [ β ] ∩ C [ α ] | , A related metric is homogeneity (Rosenberg &
Revised: Purity is deﬁned as purity = 1 m (cid:80) α ∈ Γ C max β ∈ Γ G |G [ β ] ∩ C [ α ] | , A related metric is homogeneity (Rosenberg &

Original: Hirschberg, 2007), which captures a similar idea in a different way: homogeneity = 1 − H (Γ G | Γ C ) H (Γ G ) where H (Γ G ) is the entropy of the clustering G when partitioned by the labels Γ G ,
Revised: Hirschberg, 2007), which captures a similar idea in a different way: homogeneity = 1 − H (Γ G | Γ C ) H (Γ G ) where H (Γ G ) is the entropy of the clustering G when partitioned by the labels Γ G ,

Original: H (Γ G | Γ C ) is the conditional entropy of G given C .
Revised: H (Γ G | Γ C ) is the conditional entropy of G given C .

Original: Intuitively the conditional entropy term looks inside every cluster in C , and checks the entropy of the items inside in terms of the ground truth labels.
Revised: Intuitively the conditional entropy term looks inside every cluster in C , and checks the entropy of the items inside in terms of the ground truth labels.

Original: The conditional entropy will be low (and homogeneity high) if every cluster in C contains items of only a single ground truth label.
Revised: The conditional entropy will be low (and homogeneity high) if every cluster in C contains items of only a single ground truth label.

Original: Another important metric is completeness (Rosenberg & Hirschberg, 2007), which prefers clusterings where all items from a ground-truth label lie in the same cluster i.e. the ground truth label is not split across clusters.
Revised: Another important metric is completeness (Rosenberg & Hirschberg, 2007), which prefers clusterings where all items from a ground-truth label lie in the same cluster i.e. the ground truth label is not split across clusters.

Original: Both completeness and homogeneity are considered important criteria for clustering.
Revised: Both completeness and homogeneity are considered important criteria for clustering.

Original: Maximizing either at the expense of the other is bad – high homogeneity and low completeness imply a clustering that is too ﬁne-grained ( e.g. every item in its own cluster), while low homogeneity and high completeness imply a clustering that is too coarse ( e.g. a single cluster containing all items).
Revised: Maximizing either at the expense of the other is bad – high homogeneity and low completeness imply a clustering that is too ﬁne-grained ( e.g. every item in its own cluster), while low homogeneity and high completeness imply a clustering that is too coarse ( e.g. a single cluster containing all items).

Original: Therefore, metrics that balance these 2 criterion often perform best (Rosenberg & Hirschberg, 2007) normalized
Revised: Therefore, metrics that balance these 2 criterion often perform best (Rosenberg & Hirschberg, 2007) normalized

Original: mutual information (NMI) (
Revised: mutual information (NMI) (

Original: Strehl & Ghosh, 2002) and V-measure (Rosenberg &
Revised: Strehl & Ghosh, 2002) and V-measure (Rosenberg &

Original: Hirschberg, 2007) are examples of such metrics.
Revised: Hirschberg, 2007) are examples of such metrics.

Original: Another popular suite of metrics uses the Munkres algorithm
Revised: Another popular suite of metrics uses the Munkres algorithm

Original: (Zhou et al., 2008) – a one-to-one correspondence between clusters and ground-truth labels is computed so that any classiﬁcation criterion (commonly accuracy) under this correspondence is maximized.
Revised: (Zhou et al., 2008) – a one-to-one correspondence between clusters and ground-truth labels is computed so that any classiﬁcation criterion (commonly accuracy) under this correspondence is maximized.

Original: Characterizing latent temporal structure.
Revised: Characterizing latent temporal structure.

Original: Two major characteristics distinguish the evaluation of temporal clusterings from the standard clustering case.
Revised: Two major characteristics distinguish the evaluation of temporal clusterings from the standard clustering case.

Original: • Repeated structure.
Revised: • Repeated structure.

Original: The presence of repeated structure in ground-truth – ground-truth labels that are split across several segments in the trajectory.
Revised: The presence of repeated structure in ground-truth – ground-truth labels that are split across several segments in the trajectory.

Original: Ideally, we want that this repeated structure is identiﬁed correctly e
Revised: Ideally, we want that this repeated structure is identiﬁed correctly e

Original: g. in Fig 3 identifying that the segments A 0:2 , A 4:6 are repetitions of the same primitive.
Revised: g. in Fig 3 identifying that the segments A 0:2 , A 4:6 are repetitions of the same primitive.

Original: • Segment structure.
Revised: • Segment structure.

Original: The locations and sizes of the segments in ground-truth e.g. in Fig 3 we would like temporal clusterings that switch between primitives at exactly the time points t = 2 , 4 with the same 3 segment structure.
Revised: The locations and sizes of the segments in ground-truth e.g. in Fig 3 we would like temporal clusterings that switch between primitives at exactly the time points t = 2 , 4 with the same 3 segment structure.

Original: Neither repeated or segment structure are seen in standard clusterings, since they treat data as a bag of items.
Revised: Neither repeated or segment structure are seen in standard clusterings, since they treat data as a bag of items.

Original: Limitations of existing criteria.
Revised: Limitations of existing criteria.

Original: Existing metrics cannot score repeated or segment structure because they convert temporal clusterings to standard clusterings – a contingency matrix (Rosenberg & Hirschberg, 2007) – before evaluation.
Revised: Existing metrics cannot score repeated or segment structure because they convert temporal clusterings to standard clusterings – a contingency matrix (Rosenberg & Hirschberg, 2007) – before evaluation.

Original: This erases the temporal information important to score these ideas.
Revised: This erases the temporal information important to score these ideas.

Original: In fact, qualitatively distinct temporal clusterings can be mapped to the same contingency matrix, and receive the same score, which is clearly undesirable.
Revised: In fact, qualitatively distinct temporal clusterings can be mapped to the same contingency matrix, and receive the same score, which is clearly undesirable.

Original: We use Fig 3 as an illustrative example to highlight these shortcomings.
Revised: We use Fig 3 as an illustrative example to highlight these shortcomings.

Original: In Fig 3, C 1 perfectly matches
Revised: In Fig 3, C 1 perfectly matches

Original: G , while both C 2 and C 3 split the cluster G [A] – C 2 into C 2 [X] and C 2 [Z] and C 3 into C 3 [X] and C 3 [Z] .
Revised: G , while both C 2 and C 3 split the cluster G [A] – C 2 into C 2 [X] and C 2 [Z] and C 3 into C 3 [X] and C 3 [Z] .

Original: In addition, C 2 also misses the repeated structure in A while C 3 does not.
Revised: In addition, C 2 also misses the repeated structure in A while C 3 does not.

Original: C 3 captures this repeated structure as XZ , with X 0:1 Z 1:2 corresponding to A 0:2 and X 4:5 Z 5:6 corresponding to A 4:6 .
Revised: C 3 captures this repeated structure as XZ , with X 0:1 Z 1:2 corresponding to A 0:2 and X 4:5 Z 5:6 corresponding to A 4:6 .

Original: However, it does so at the cost of creating more segments than C 2 – ignoring labels, C 2 has a segmentation exactly like G .
Revised: However, it does so at the cost of creating more segments than C 2 – ignoring labels, C 2 has a segmentation exactly like G .

Original: Intuitively, we prefer C 1 > C 3 >
Revised: Intuitively, we prefer C 1 > C 3 >

Original: C
Revised: C

Original: 2 if we care more about repeated structure, and
Revised: 2 if we care more about repeated structure, and

Original: C 1 > C 2 > C 3 , if we care more about the segmentation quality.
Revised: C 1 > C 2 > C 3 , if we care more about the segmentation quality.

Original: However, all 3 temporal clusterings C 1 , C 2 , C 3 in Fig 3 are assigned a perfect purity or homogeneity, since as seen in Fig 3b, no cluster in any of the 3 contains items from more than one ground-truth label.
Revised: However, all 3 temporal clusterings C 1 , C 2 , C 3 in Fig 3 are assigned a perfect purity or homogeneity, since as seen in Fig 3b, no cluster in any of the 3 contains items from more than one ground-truth label.

Original: Moreover, no traditional clustering criteria can distinguish C 2 from C 3 since their cluster composition is identical, as seen in Fig 3b.
Revised: Moreover, no traditional clustering criteria can distinguish C 2 from C 3 since their cluster composition is identical, as seen in Fig 3b.

Original: We now discuss new evaluation criteria that can systematically evaluate and score both repeated and segment structure.
Revised: We now discuss new evaluation criteria that can systematically evaluate and score both repeated and segment structure.

Original: 3. E VALUATING R EPEATED S TRUCTURE
Revised: 3. E VALUATING R EPEATED S TRUCTURE

Original: We describe a new evaluation criteria, called the repeated structure score (RSS) below.
Revised: We describe a new evaluation criteria, called the repeated structure score (RSS) below.

Original: Algorithmically, the calculation of the RSS requires the following steps,
Revised: Algorithmically, the calculation of the RSS requires the following steps,

Original: Pick a ground-truth label β ∈ Γ G and ﬁnd the set of segments S ( G , β ) marked with β in G .
Revised: Pick a ground-truth label β ∈ Γ G and ﬁnd the set of segments S ( G , β ) marked with β in G .

Original: These segments all exhibit repeated structure in G . 2.
Revised: These segments all exhibit repeated structure in G . 2.

Original: Find C a : b for every ( a, b ) ∈ S ( G , β ) .
Revised: Find C a : b for every ( a, b ) ∈ S ( G , β ) .

Original: This is the predicted temporal clustering restricted to a particular segment ( a, b ) in G . 3.
Revised: This is the predicted temporal clustering restricted to a particular segment ( a, b ) in G . 3.

Original: For every such pair C a 1 : b 1 , C a 2 : b 2 , use a scoring function H to compute H ( C a 1 : b 1 , C a 2 : b 2 ) .
Revised: For every such pair C a 1 : b 1 , C a 2 : b 2 , use a scoring function H to compute H ( C a 1 : b 1 , C a 2 : b 2 ) .

Original: Aggregate these scores across all such pairs for every ground-truth label β ∈ Γ G , RSS =
Revised: Aggregate these scores across all such pairs for every ground-truth label β ∈ Γ G , RSS =

Original: [Equation]
Revised: [Equation]

Original: where the denominator normalizes RSS to lie in [0 , 1] .
Revised: where the denominator normalizes RSS to lie in [0 , 1] .

Original: (a) Example temporal clustering.
Revised: (a) Example temporal clustering.

Original: (b)
Revised: (b)

Original: Scoring
Revised: Scoring

Original: C 0:2 , C 4: with P ( C 0:2 ) =
Revised: C 0:2 , C 4: with P ( C 0:2 ) =

Original: (X) , W ( C 0:2 ) =
Revised: (X) , W ( C 0:2 ) =

Original: (2) , P ( C 4:7 ) =
Revised: (2) , P ( C 4:7 ) =

Original: (X , Z) , W ( C 4:7 ) =
Revised: (X , Z) , W ( C 4:7 ) =

Original: (1 , 2) and running H yields heaviest sub-sequence (X) with score 3.
Revised: (1 , 2) and running H yields heaviest sub-sequence (X) with score 3.

Original: C 5:7 (dotted) does not contribute to the score.
Revised: C 5:7 (dotted) does not contribute to the score.

Original: As a concrete example, in Fig 4a, we could pick A in G for step 1 with S ( G , β = A) = { (0 , 2) , (4 , 7) } .
Revised: As a concrete example, in Fig 4a, we could pick A in G for step 1 with S ( G , β = A) = { (0 , 2) , (4 , 7) } .

Original: For step 2, we ﬁnd C 0:

Revised: For step 2, we ﬁnd C 0:

Original: =
Revised: =

Original: (X , X) and C 4:7 =
Revised: (X , X) and C 4:7 =

Original: (X , Z , Z) .
Revised: (X , Z , Z) .

Original: For step 3, we must choose an appropriate scoring function H .
Revised: For step 3, we must choose an appropriate scoring function H .

Original: While many choices for H are possible, we let H ( C a 1 : b 1 , C a 2 : b 2 ) be the total weight of the heaviest common sub-sequence or substring (Jacobson & Vo, 1992) in C a 1 : b 1 and C
Revised: While many choices for H are possible, we let H ( C a 1 : b 1 , C a 2 : b 2 ) be the total weight of the heaviest common sub-sequence or substring (Jacobson & Vo, 1992) in C a 1 : b 1 and C

Original: a 2 : b 2 .
Revised: a 2 : b 2 .

Original: Fig 4b uses this choice of H to score the pair of segments we picked – we run
Revised: Fig 4b uses this choice of H to score the pair of segments we picked – we run

Original: H ((X , X) , (X , Z , Z)) and it returns a score of 3 .
Revised: H ((X , X) , (X , Z , Z)) and it returns a score of 3 .

Original: Intuitively, H tries to match procedures P ( C a 1 : b 1 ) to P ( C a 2 : b 2 ) – ﬁnding the (heaviest) sequence of tokens common to both
Revised: Intuitively, H tries to match procedures P ( C a 1 : b 1 ) to P ( C a 2 : b 2 ) – ﬁnding the (heaviest) sequence of tokens common to both

Original: P ( C a 1 : b 1 ) and P ( C a 2 : b 2 ) , while respecting the temporal order- ing of both procedures.
Revised: P ( C a 1 : b 1 ) and P ( C a 2 : b 2 ) , while respecting the temporal order- ing of both procedures.

Original: At one extreme, if H evaluates tothen no overlapping tokens exist in P ( C a 1 : b 1 ) and P ( C a 2 : b 2 ) .
Revised: At one extreme, if H evaluates tothen no overlapping tokens exist in P ( C a 1 : b 1 ) and P ( C a 2 : b 2 ) .

Original: This implies that C failed to identify repeated structure in the two segments and a score of 0 is appropriate.
Revised: This implies that C failed to identify repeated structure in the two segments and a score of 0 is appropriate.

Original: At the other extreme, if both segments follow identical procedures
Revised: At the other extreme, if both segments follow identical procedures

Original: P ( C a 1 : b 1 ) =
Revised: P ( C a 1 : b 1 ) =

Original: P ( C a 2 : b 2 ) , they receive the maximum possible score according to H ; reﬂecting that C identiﬁed repeated structure perfectly.
Revised: P ( C a 2 : b 2 ) , they receive the maximum possible score according to H ; reﬂecting that C identiﬁed repeated structure perfectly.

Original: H can be computed efﬁciently using a dynamic program in O ( |P 1 ||P 2 | ) .
Revised: H can be computed efﬁciently using a dynamic program in O ( |P 1 ||P 2 | ) .

Original: 3. E VALUATING S EGMENT S TRUCTURE
Revised: 3. E VALUATING S EGMENT S TRUCTURE

Original: We ﬁrst introduce a new information-theoretic criteria – the label-agnostic segmentation score (LASS) .
Revised: We ﬁrst introduce a new information-theoretic criteria – the label-agnostic segmentation score (LASS) .

Original: The main purpose of LASS is to score how well C predicts the location of transition points between segments in G .
Revised: The main purpose of LASS is to score how well C predicts the location of transition points between segments in G .

Original: We want to identify and score cases where C either oversegments (introduces extra segments) or undersegments (omits segments) the time-series compared to G .
Revised: We want to identify and score cases where C either oversegments (introduces extra segments) or undersegments (omits segments) the time-series compared to G .

Original: In any temporal clustering, the transition boundaries do not depend on the actual labeling, but only on where the labels switch.
Revised: In any temporal clustering, the transition boundaries do not depend on the actual labeling, but only on where the labels switch.

Original: Therefore, LASS is label-agnostic and only requires the transition points between segments.
Revised: Therefore, LASS is label-agnostic and only requires the transition points between segments.

Original: Prior work in changepoint detection and time-series segmentation (Killick et al., 2012) has attempted to quantify this using detection criteria – e.g. if a changepoint close to one in ground-truth is detected, a true positive is recorded.
Revised: Prior work in changepoint detection and time-series segmentation (Killick et al., 2012) has attempted to quantify this using detection criteria – e.g. if a changepoint close to one in ground-truth is detected, a true positive is recorded.

Original: However, these criteria (i) require specifying the tolerance interval for detection, and (ii) don’t degrade gracefully with the segmentation, which limits their applicability to temporal clusterings.
Revised: However, these criteria (i) require specifying the tolerance interval for detection, and (ii) don’t degrade gracefully with the segmentation, which limits their applicability to temporal clusterings.

Original: Recall that S G , S C are the set of all segments in G and C .
Revised: Recall that S G , S C are the set of all segments in G and C .

Original: To compute LASS, the main quantity we rely on is H ( S C | S G ) – a conditional entropy over segments in C .
Revised: To compute LASS, the main quantity we rely on is H ( S C | S G ) – a conditional entropy over segments in C .

Original: This can be written as,
Revised: This can be written as,

Original: H ( S C | S G ) = − (cid:88) ( a,b ) ∈ S G ( b − a + 1) n (cid:88) (˜ a, ˜ b ) ∈ S C a : b ( ˜ b − ˜ a + 1) ( b − a + 1) log ( ˜ b − ˜ a + 1) ( b − a + 1)
Revised: H ( S C | S G ) = − (cid:88) ( a,b ) ∈ S G ( b − a + 1) n (cid:88) (˜ a, ˜ b ) ∈ S C a : b ( ˜ b − ˜ a + 1) ( b − a + 1) log ( ˜ b − ˜ a + 1) ( b − a + 1)

Original: Intuitively, for each ground-truth segment ( a, b ) ∈ S G , we compute the (weighted) entropy of C restricted to ( a, b ) .
Revised: Intuitively, for each ground-truth segment ( a, b ) ∈ S G , we compute the (weighted) entropy of C restricted to ( a, b ) .

Original: This entropy will be > 0 only if C creates new segments in ( a, b ) .
Revised: This entropy will be > 0 only if C creates new segments in ( a, b ) .

Original: On the other hand, to detect under-segmentation, we can simply invert the roles of G and C by calculating H ( S G | S C ) .
Revised: On the other hand, to detect under-segmentation, we can simply invert the roles of G and C by calculating H ( S G | S C ) .

Original: Finally, noting that H ( S C | S G ) ≤ H ( S C ) , we deﬁne LASS := 1 − H ( S C | S G )
Revised: Finally, noting that H ( S C | S G ) ≤ H ( S C ) , we deﬁne LASS := 1 − H ( S C | S G )

Original: + H ( S G | S C )
Revised: + H ( S G | S C )

Original: H ( S C )
Revised: H ( S C )

Original: + H ( S G ) .
Revised: + H ( S G ) .

Original: Oversegmentation (and under-segmentation) can also be measured separately by deﬁning LASS − O := 1 − H ( S C | S G ) H ( S C ) and LASS − U :
Revised: Oversegmentation (and under-segmentation) can also be measured separately by deﬁning LASS − O := 1 − H ( S C | S G ) H ( S C ) and LASS − U :

Original: = 1 − H ( S G | S C ) H ( S G ) .
Revised: = 1 − H ( S G | S C ) H ( S G ) .

Original: The algebraic form chosen for the criteria relates it back to existing information-theoretic criteria such as the NMI (Vinh et al., 2010).
Revised: The algebraic form chosen for the criteria relates it back to existing information-theoretic criteria such as the NMI (Vinh et al., 2010).

Original: Next, we extend the completeness metric deﬁned earlier.
Revised: Next, we extend the completeness metric deﬁned earlier.

Original: While completeness is desirable for any temporal clustering, using completeness directly has an unintended drawback – it penalizes repeated structure.
Revised: While completeness is desirable for any temporal clustering, using completeness directly has an unintended drawback – it penalizes repeated structure.

Original: An example of this can be seen in Fig 3a, where C 3 is assigned the same completeness score as C 2 despite capturing repeated structure.
Revised: An example of this can be seen in Fig 3a, where C 3 is assigned the same completeness score as C 2 despite capturing repeated structure.

Original: Since the RSS already provides a systematic way of evaluating presence/absence of repeated structure, we modify the completeness score slightly.
Revised: Since the RSS already provides a systematic way of evaluating presence/absence of repeated structure, we modify the completeness score slightly.

Original: We deﬁne segmental completeness to be 1 − H (Γ C | S G ) H (Γ C ) , where the conditioning is now on ground-truth segments rather than on ground-truth labels.
Revised: We deﬁne segmental completeness to be 1 − H (Γ C | S G ) H (Γ C ) , where the conditioning is now on ground-truth segments rather than on ground-truth labels.

Original: Equivalently, we can view this as relabeling each segment in ground-truth to be distinct, and then computing completeness.
Revised: Equivalently, we can view this as relabeling each segment in ground-truth to be distinct, and then computing completeness.

Original: In this way, segmental completeness serves a similar function to completeness without interfering with repeated structure scoring.
Revised: In this way, segmental completeness serves a similar function to completeness without interfering with repeated structure scoring.

Original: Homogeneity can also be extended to deﬁne segmental homogeneity similarly.
Revised: Homogeneity can also be extended to deﬁne segmental homogeneity similarly.

Original: Lastly, we deﬁne an aggregate segment structure score (SSS) ,
Revised: Lastly, we deﬁne an aggregate segment structure score (SSS) ,

Original: [Equation]
Revised: [Equation]

Original: C OMBINED E VALUATION C RITERIA
Revised: C OMBINED E VALUATION C RITERIA

Original: We have introduced several new metrics: (i) the RSS systematically scores repeated structure; (ii) the
Revised: We have introduced several new metrics: (i) the RSS systematically scores repeated structure; (ii) the

Original: LASS scores how well the transition structure is captured; (iii) the SSS provides a uniﬁed metric for assessing segment structure.
Revised: LASS scores how well the transition structure is captured; (iii) the SSS provides a uniﬁed metric for assessing segment structure.

Original: Note that all of these metrics obey the useful property of n -invariance (Rosenberg & Hirschberg, 2007; Meil˘a, 2007) – scaling the number of items (stretching the temporal clustering) does not change the metric values.
Revised: Note that all of these metrics obey the useful property of n -invariance (Rosenberg & Hirschberg, 2007; Meil˘a, 2007) – scaling the number of items (stretching the temporal clustering) does not change the metric values.

Original: Finally, we provide a single evaluation measure for temporal clusterings, the temporal structure score (TSS) that balances the RSS with the SSS.
Revised: Finally, we provide a single evaluation measure for temporal clusterings, the temporal structure score (TSS) that balances the RSS with the SSS.

Original: [Equation]
Revised: [Equation]

Original: While it is preferable to examine the constituent scores individually to determine the strengths and weaknesses of methods, a uniﬁed metric allows us to choose methods that balance these criteria.
Revised: While it is preferable to examine the constituent scores individually to determine the strengths and weaknesses of methods, a uniﬁed metric allows us to choose methods that balance these criteria.

Original: β can be tuned to change the inﬂuence of the constituent metrics according to the desired application – we set β = 1 . 

Revised: β can be tuned to change the inﬂuence of the constituent metrics according to the desired application – we set β = 1 .

Original: in a problem-agnostic fashion in this paper.
Revised: in a problem-agnostic fashion in this paper.

Original: This choice is in line with previous work that deﬁnes compound criteria, such as Rosenberg & Hirschberg (2007).
Revised: This choice is in line with previous work that deﬁnes compound criteria, such as Rosenberg & Hirschberg (2007).

Original:  I DENTIFYING P ROCEDURAL A BSTRACTIONS
Revised:  I DENTIFYING P ROCEDURAL A BSTRACTIONS

Original: So far, we have argued the importance of evaluating repeated and segment structure correctly in temporal data.
Revised: So far, we have argued the importance of evaluating repeated and segment structure correctly in temporal data.

Original: Our new metrics enable a systematic evaluation of these ideas, whereas existing metrics did not speciﬁcally evaluate or emphasize them.
Revised: Our new metrics enable a systematic evaluation of these ideas, whereas existing metrics did not speciﬁcally evaluate or emphasize them.

Original: Given this, we now return to our original goal of identifying procedural abstractions in time-series data, which we term procedure identiﬁcation .
Revised: Given this, we now return to our original goal of identifying procedural abstractions in time-series data, which we term procedure identiﬁcation .

Original: x z p w λ u γ µ, Σ α β µ 0 , Σ 0 ,κ 0 ,ν
Revised: x z p w λ u γ µ, Σ α β µ 0 , Σ 0 ,κ 0 ,ν

Original: K s m m n
Revised: K s m m n

Original: Concretely, we assume a dataset of time-series such that each demonstrates the same task or phenomenon.
Revised: Concretely, we assume a dataset of time-series such that each demonstrates the same task or phenomenon.

Original: Crucially, we assume that all time-series share this single, latent procedure – different time-series are distinct realizations of this underlying procedure.
Revised: Crucially, we assume that all time-series share this single, latent procedure – different time-series are distinct realizations of this underlying procedure.

Original: The goal is to label the temporal structure present in each time-series while identifying the common, latent procedure.
Revised: The goal is to label the temporal structure present in each time-series while identifying the common, latent procedure.

Original: For example, this may involve identifying the common sequence of steps in a set of demonstrations that follow the same cooking recipe as well as their temporal extent in each demonstration.
Revised: For example, this may involve identifying the common sequence of steps in a set of demonstrations that follow the same cooking recipe as well as their temporal extent in each demonstration.

Original: Procedure identiﬁcation is related to recent efforts in computer vision that relate to unsupervised activity recognition in videos (Wu et al., 2015;
Revised: Procedure identiﬁcation is related to recent efforts in computer vision that relate to unsupervised activity recognition in videos (Wu et al., 2015;

Original: Yang et al., 2013; Krishnan et al., 2015; Sener & Yao, 2018) as well as prior work in latent variable modeling (Yang et al., 2013;
Revised: Yang et al., 2013; Krishnan et al., 2015; Sener & Yao, 2018) as well as prior work in latent variable modeling (Yang et al., 2013;

Original: Fox et al., 2008b; 2014; Johnson & Willsky, 2013; Linderman et al., 2017).
Revised: Fox et al., 2008b; 2014; Johnson & Willsky, 2013; Linderman et al., 2017).

Original: These papers typically assume a general setting where activities or latent tokens may be reused within or across multiple time-series but there is no onus on methods to discover this structure.
Revised: These papers typically assume a general setting where activities or latent tokens may be reused within or across multiple time-series but there is no onus on methods to discover this structure.

Original: Unlike prior work, procedure identiﬁcation requires that methods also return a common procedure in addition to the temporal clustering of each time-series.
Revised: Unlike prior work, procedure identiﬁcation requires that methods also return a common procedure in addition to the temporal clustering of each time-series.

Original: This additional requirement places a burden on the method to explicitly , and not implicitly reason about repeated structure.
Revised: This additional requirement places a burden on the method to explicitly , and not implicitly reason about repeated structure.

Original: Formally, ﬁnding a common procedure P corresponds to ﬁnding temporal clusterings
Revised: Formally, ﬁnding a common procedure P corresponds to ﬁnding temporal clusterings

Original: C i for each time-series, such that P = P ( C 1 )
Revised: C i for each time-series, such that P = P ( C 1 )

Original: =
Revised: =

Original: P ( C 2 ) = · · · = P ( C n ) .
Revised: P ( C 2 ) = · · · = P ( C n ) .

Original: Since a single procedure consistent with all temporal clusterings is desired, shared primitives must be identiﬁed across all videos.
Revised: Since a single procedure consistent with all temporal clusterings is desired, shared primitives must be identiﬁed across all videos.

Original: This requirement also makes evaluation in this setting particularly suited for our metrics.
Revised: This requirement also makes evaluation in this setting particularly suited for our metrics.

Original: Methods that cannot relate primitives across demonstrations (identify repeated structure) or reason about their temporal extents (identify segment structure) will be unable to recover the underlying procedure and should therefore be penalized.
Revised: Methods that cannot relate primitives across demonstrations (identify repeated structure) or reason about their temporal extents (identify segment structure) will be unable to recover the underlying procedure and should therefore be penalized.

Original: There may also be additional repeated structure within a time-series, for example if someone mixes cake batter, adds ingredients, and then mixes the cake batter again.
Revised: There may also be additional repeated structure within a time-series, for example if someone mixes cake batter, adds ingredients, and then mixes the cake batter again.

Original: A closely related setting was tackled by the method outlined in Sener & Yao (2018), who assume that each time-series is a permutation of a shared bag of activities.
Revised: A closely related setting was tackled by the method outlined in Sener & Yao (2018), who assume that each time-series is a permutation of a shared bag of activities.

Original: However, as they acknowledge, a major limitation of their method is that they allow each activity to occur atmost once within a time-series, which limits repeated structure.
Revised: However, as they acknowledge, a major limitation of their method is that they allow each activity to occur atmost once within a time-series, which limits repeated structure.

Original: Our problem setting allows for this possibility, which is often seen in real-world datasets e.g. the JIGSAWS surgical dataset (Gao et al., 2014) contains videos of surgeons doing multiple repetitions of primitives such as needle-passing or suturing.
Revised: Our problem setting allows for this possibility, which is often seen in real-world datasets e.g. the JIGSAWS surgical dataset (Gao et al., 2014) contains videos of surgeons doing multiple repetitions of primitives such as needle-passing or suturing.

Original: However, we do not allow the procedure to be permuted in our problem setting, which is a useful extension that we hope to explore in future work.
Revised: However, we do not allow the procedure to be permuted in our problem setting, which is a useful extension that we hope to explore in future work.

Original: 4. P RISM : PR OCEDURE I DENTIFICATION WITH A S EGMENTAL M IXTURE MODEL
Revised: 4. P RISM : PR OCEDURE I DENTIFICATION WITH A S EGMENTAL M IXTURE MODEL

Original: We now describe a statistical model called P RISM for the procedure identiﬁcation problem.
Revised: We now describe a statistical model called P RISM for the procedure identiﬁcation problem.

Original: P RISM is a generative hierarchical Bayesian model that explicitly separates reasoning about the procedure from its realization in each time-series in the dataset.
Revised: P RISM is a generative hierarchical Bayesian model that explicitly separates reasoning about the procedure from its realization in each time-series in the dataset.

Original: It utilizes this to create a mixture distribution over primitives for every observation in the dataset.
Revised: It utilizes this to create a mixture distribution over primitives for every observation in the dataset.

Original: Fig 5 illustrates the model.
Revised: Fig 5 illustrates the model.

Original: Unlike popular sequence models based on the Hidden Markov model, P RISM does not make a restrictive Markov assumption and can handle any procedure.
Revised: Unlike popular sequence models based on the Hidden Markov model, P RISM does not make a restrictive Markov assumption and can handle any procedure.

Original: And in contrast to a Gaussian mixture model, P RISM shares statistical strength across time-series by using a structured prior over the (discrete) local variables that generate each observation – amortizing the procedure across many timeseries.
Revised: And in contrast to a Gaussian mixture model, P RISM shares statistical strength across time-series by using a structured prior over the (discrete) local variables that generate each observation – amortizing the procedure across many timeseries.

Original: As we show later in experiments, this structure can help P RISM outperform these methods, especially when the observations are noisy.
Revised: As we show later in experiments, this structure can help P RISM outperform these methods, especially when the observations are noisy.

Original: Modeling the procedure.
Revised: Modeling the procedure.

Original: P RISM assumes that each time-series X i follows the same underlying procedure p
Revised: P RISM assumes that each time-series X i follows the same underlying procedure p

Original: =
Revised: =

Original: ( p 1 , . . .
Revised: ( p 1 , . . .

Original: , p s ) , where s is the number of steps in the procedure.
Revised: , p s ) , where s is the number of steps in the procedure.

Original: p is a vector of discrete, global latent variables in the model.
Revised: p is a vector of discrete, global latent variables in the model.

Original: This implies that each time-series will split into s segments, each segment corresponding to a single step in the procedure.
Revised: This implies that each time-series will split into s segments, each segment corresponding to a single step in the procedure.

Original: Next, let | Γ C |
Revised: Next, let | Γ C |

Original: = K be the number of cluster labels (primitives).
Revised: = K be the number of cluster labels (primitives).

Original: Each step in the procedure will correspond to one of these K labels
Revised: Each step in the procedure will correspond to one of these K labels

Original: i.e. p r ∈
Revised: i.e. p r ∈

Original: [ K ] , 1 ≤ r ≤ s .
Revised: [ K ] , 1 ≤ r ≤ s .

Original: Thus, p is an ordered sequence of high-level primitives e.g. to make tea, we would boil water → get cup → pour water and so on.
Revised: Thus, p is an ordered sequence of high-level primitives e.g. to make tea, we would boil water → get cup → pour water and so on.

Original: Formally, we assume a generation process,
Revised: Formally, we assume a generation process,

Original: [Equation]
Revised: [Equation]

Original: Each token in the procedure is independently generated from a separate Dirichlet prior λ r , which allows ﬁne-grained control over the distribution of tokens at each step.
Revised: Each token in the procedure is independently generated from a separate Dirichlet prior λ r , which allows ﬁne-grained control over the distribution of tokens at each step.

Original: Note that we do not disallow self-transitions, which can be utilized by the model to coalesce adjacent segments.
Revised: Note that we do not disallow self-transitions, which can be utilized by the model to coalesce adjacent segments.

Original: Modeling realizations in each time-series.
Revised: Modeling realizations in each time-series.

Original: Different time-series can be vastly different in how they implement the procedure e.g. they can be videos of different individuals making tea.
Revised: Different time-series can be vastly different in how they implement the procedure e.g. they can be videos of different individuals making tea.

Original: These individuals, while following the same procedure, may spend varying amounts of time on each step (or even skip steps).
Revised: These individuals, while following the same procedure, may spend varying amounts of time on each step (or even skip steps).

Original: In P RISM , the realization of the procedure in the i th time-series (having m i time-steps) is given by w i = ( w i, 1 , . . .
Revised: In P RISM , the realization of the procedure in the i th time-series (having m i time-steps) is given by w i = ( w i, 1 , . . .

Original: , w i,m i ) , a sorted sequence of discrete random variables.
Revised: , w i,m i ) , a sorted sequence of discrete random variables.

Original: Each w i,j ∈ [ s ] is an index into the procedure p .
Revised: Each w i,j ∈ [ s ] is an index into the procedure p .

Original: Its value dictates the step being followed in the procedure at the j th time-step in the i th time-series.
Revised: Its value dictates the step being followed in the procedure at the j th time-step in the i th time-series.

Original: We assume w i is generated as follows,
Revised: We assume w i is generated as follows,

Original: [Equation]
Revised: [Equation]

Original: Mathematically, to generate w i , we ﬁrst generate an unordered bag of m i independent categorical variables u i,j and then sort them (in ascending order).
Revised: Mathematically, to generate w i , we ﬁrst generate an unordered bag of m i independent categorical variables u i,j and then sort them (in ascending order).

Original: Each u i,j is generated from the same Dirichlet prior γ .
Revised: Each u i,j is generated from the same Dirichlet prior γ .

Original: Intuitively, if we use the stick-breaking metaphor for the Dirichlet distribution, γ is a prior over how long each token in the procedure is – γ r is the expected relative length of p r .
Revised: Intuitively, if we use the stick-breaking metaphor for the Dirichlet distribution, γ is a prior over how long each token in the procedure is – γ r is the expected relative length of p r .

Original: Thus, γ allows us to impose a prior on the relative length of each step of the procedure, in every time-series.
Revised: Thus, γ allows us to impose a prior on the relative length of each step of the procedure, in every time-series.

Original: An equivalent view into our model is that each w i is directly generated by a Multinomial distribution with m i draws on probabilities dictated by γ .
Revised: An equivalent view into our model is that each w i is directly generated by a Multinomial distribution with m i draws on probabilities dictated by γ .

Original: However, we ﬁnd that the view that we adopt yields a far more efﬁcient inference scheme (see Appendix).
Revised: However, we ﬁnd that the view that we adopt yields a far more efﬁcient inference scheme (see Appendix).

Original: Generating observations.
Revised: Generating observations.

Original: For every time-series, we can combine the common procedure p and the realization of the procedure w i to recover a temporal clustering z
Revised: For every time-series, we can combine the common procedure p and the realization of the procedure w i to recover a temporal clustering z

Original: i for that time-series.
Revised: i for that time-series.

Original: We write this temporal clustering as z i =
Revised: We write this temporal clustering as z i =

Original: ( z i, 1 , . . .
Revised: ( z i, 1 , . . .

Original: , z i,m i ) where z i,j ∈
Revised: , z i,m i ) where z i,j ∈

Original: [ K ] is a discrete primitive.
Revised: [ K ] is a discrete primitive.

Original: For each primitive k ∈ [ K ] , we assume a Gaussian observation model ( µ k , Σ k ) (with a
Revised: For each primitive k ∈ [ K ] , we assume a Gaussian observation model ( µ k , Σ k ) (with a

Original: Normal-Inverse Wishart prior) shared by all time-series.
Revised: Normal-Inverse Wishart prior) shared by all time-series.

Original: P RISM then generates each x ij independently from its local assignment
Revised: P RISM then generates each x ij independently from its local assignment

Original: z
Revised: z

Original: i,j and the appropriate observation model.
Revised: i,j and the appropriate observation model.

Original: We use Gibbs sampling (see Appendix) to perform posterior inference of the latent variables ( { p r } r ∈ [ s ] , { u i,j } i ∈ [ n ] ,j ∈ [ m i ] , { µ k } k ∈
Revised: We use Gibbs sampling (see Appendix) to perform posterior inference of the latent variables ( { p r } r ∈ [ s ] , { u i,j } i ∈ [ n ] ,j ∈ [ m i ] , { µ k } k ∈

Original: [ K ] , { Σ k } k ∈ [ K ] ) in the model.
Revised: [ K ] , { Σ k } k ∈ [ K ] ) in the model.

Original:  E XPERIMENTS WITH E VALUATION C RITERIA
Revised:  E XPERIMENTS WITH E VALUATION C RITERIA

Original: We ﬁrst compare our metrics to existing criteria by evaluating several competing methods on real datasets.
Revised: We ﬁrst compare our metrics to existing criteria by evaluating several competing methods on real datasets.

Original: We note that the methodology for validating our evaluation criteria is similar to that followed by Rosenberg & Hirschberg (2007), with the exception that we compare criteria on real-world data with a large number of methods.
Revised: We note that the methodology for validating our evaluation criteria is similar to that followed by Rosenberg & Hirschberg (2007), with the exception that we compare criteria on real-world data with a large number of methods.

Original: We compare to homogeneity (Hom), completeness (Com), NMI since they give the best combination of performance for clustering evaluation (Rosenberg & Hirschberg, 2007).
Revised: We compare to homogeneity (Hom), completeness (Com), NMI since they give the best combination of performance for clustering evaluation (Rosenberg & Hirschberg, 2007).

Original: We also compare to accuracy computed using the Munkres method since it is widely used e.g.
Revised: We also compare to accuracy computed using the Munkres method since it is widely used e.g.

Original: (Sener & Yao, 2018;
Revised: (Sener & Yao, 2018;

Original: Zhou et al., 2008) and the adjusted Rand index (ARI) (Vinh et al., 2010).
Revised: Zhou et al., 2008) and the adjusted Rand index (ARI) (Vinh et al., 2010).

Original: Methods.
Revised: Methods.

Original: We use several methods to generate temporal clusterings: Hierarchical Clustering (A GG );
Revised: We use several methods to generate temporal clusterings: Hierarchical Clustering (A GG );

Original: Hidden Markov Models (HMM); Hierarchical Dirichlet Process HMMs (HDP-HMM) (Teh et al., 2005); Sticky HDP-HMM (SHDP-HMM) (Fox et al., 2008a); HDP Hidden Semi-Markov Models (HDP-HSMM) (Johnson & Willsky, 2013); Switching Linear Dynamical Systems (SLDS) (Fox et al., 2008b); HDP-SLDS (Fox et al., 2008b);
Revised: Hidden Markov Models (HMM); Hierarchical Dirichlet Process HMMs (HDP-HMM) (Teh et al., 2005); Sticky HDP-HMM (SHDP-HMM) (Fox et al., 2008a); HDP Hidden Semi-Markov Models (HDP-HSMM) (Johnson & Willsky, 2013); Switching Linear Dynamical Systems (SLDS) (Fox et al., 2008b); HDP-SLDS (Fox et al., 2008b);

Original: Sticky HDP-SLDS (SHDP-SLDS) (Fox et al., 2008b); Ordered Subspace Clustering (OSC) (Tierney et al., 2014); Temporal SC (TSC) (Li et al., 2015); Low-Rank SC (LRSC) (Vidal & Favaro, 2014); Sparse SC (SSC) (Elhamifar & Vidal, 2013) and Spatial SC (SPATSC)
Revised: Sticky HDP-SLDS (SHDP-SLDS) (Fox et al., 2008b); Ordered Subspace Clustering (OSC) (Tierney et al., 2014); Temporal SC (TSC) (Li et al., 2015); Low-Rank SC (LRSC) (Vidal & Favaro, 2014); Sparse SC (SSC) (Elhamifar & Vidal, 2013) and Spatial SC (SPATSC)

Original: (Guo et al., 2013).
Revised: (Guo et al., 2013).

Original: Datasets.
Revised: Datasets.

Original: We use 2 common benchmark datasets (Fox et al., 2008b; 2009; 2014;
Revised: We use 2 common benchmark datasets (Fox et al., 2008b; 2009; 2014;

Original: Zhou et al., 2008; 2013).
Revised: Zhou et al., 2008; 2013).

Original: B EES consists of 6 time-series, each recording the dancing behavior of a bee.
Revised: B EES consists of 6 time-series, each recording the dancing behavior of a bee.

Original: Each time-series is built up of 3 primitive movements – waggle , left-turn and right-turn – repeated several times.
Revised: Each time-series is built up of 3 primitive movements – waggle , left-turn and right-turn – repeated several times.

Original: For
Revised: For

Original: B EES , we treat each sequence as a separate dataset (B EES 1 ,...,B EES 6 ) and run each method on them individually, as done in Fox et al.
Revised: B EES , we treat each sequence as a separate dataset (B EES 1 ,...,B EES 6 ) and run each method on them individually, as done in Fox et al.

Original: (2008b).
Revised: (2008b).

Original: Here, we focus on results on B EES 1 but results on
Revised: Here, we focus on results on B EES 1 but results on

Original: M OCAP 6, a motion-capture dataset and all B EES datasets can be found in the Appendix.
Revised: M OCAP 6, a motion-capture dataset and all B EES datasets can be found in the Appendix.

Original: Results.
Revised: Results.

Original: For each of our metrics, we ﬁrst illustrate that they qualitatively capture the characteristics that they were designed for.
Revised: For each of our metrics, we ﬁrst illustrate that they qualitatively capture the characteristics that they were designed for.

Original: Fig 6 visualizes the best and worst performing methods on B EESfor each of our metrics.
Revised: Fig 6 visualizes the best and worst performing methods on B EESfor each of our metrics.

Original: In Fig 6a, SSC is worst on LASS-O since it heavily over-segments the time-series by introducing many transitions, while TSC introduces far fewer transitions.
Revised: In Fig 6a, SSC is worst on LASS-O since it heavily over-segments the time-series by introducing many transitions, while TSC introduces far fewer transitions.

Original: In Fig 6b, under-segmentation is penalized and we can see that A GG should be worse since it contains longer segments and far fewer transitions than SSC.
Revised: In Fig 6b, under-segmentation is penalized and we can see that A GG should be worse since it contains longer segments and far fewer transitions than SSC.

Original: Combining these ideas in Fig 6c yields that HDPHSMM has the best LASS – observe that it contains fewer closely spaced transition points and creates a more even segmentation at the right level of granularity, explaining its score.
Revised: Combining these ideas in Fig 6c yields that HDPHSMM has the best LASS – observe that it contains fewer closely spaced transition points and creates a more even segmentation at the right level of granularity, explaining its score.

Original: Fig 6e shows the result of evaluating with just the RSS – we can clearly see that SpatSC is able to capture far more repeated structure than SHDP-HMM, which is unable to label segments from the same ground-truth label consistently.
Revised: Fig 6e shows the result of evaluating with just the RSS – we can clearly see that SpatSC is able to capture far more repeated structure than SHDP-HMM, which is unable to label segments from the same ground-truth label consistently.

Original: Lastly, we see that SpatSC is judged to be the best method on
Revised: Lastly, we see that SpatSC is judged to be the best method on

Original: B EES 1 by TSS in Fig 6f since it has the highest RSS and a SSS of 0 .
Revised: B EES 1 by TSS in Fig 6f since it has the highest RSS and a SSS of 0 .

Original: 81 which is close to the highest.
Revised: 81 which is close to the highest.

Original: Improving scores on evaluation metrics is common practice to demonstrate performance gains due to a method.
Revised: Improving scores on evaluation metrics is common practice to demonstrate performance gains due to a method.

Original: We now highlight cases where TSS disagrees with NMI, ARI and/or Munkres in the assessment of competing methods, since this discrepancy has signiﬁcant impact on which method is picked.
Revised: We now highlight cases where TSS disagrees with NMI, ARI and/or Munkres in the assessment of competing methods, since this discrepancy has signiﬁcant impact on which method is picked.

Original: We ﬁnd that such disagreements happen very often across all datasets.
Revised: We ﬁnd that such disagreements happen very often across all datasets.

Original: An illustrative example for B EES 1 is shown in Fig 6h: TSS and Munkres agree that SSC is better than HDP-HSMM, but NMI and ARI strongly disagree.
Revised: An illustrative example for B EES 1 is shown in Fig 6h: TSS and Munkres agree that SSC is better than HDP-HSMM, but NMI and ARI strongly disagree.

Original: Before diving into why this may be happening, we note that examining the RSS and SSS allows us to understand why TSS considers SSC to be better than
Revised: Before diving into why this may be happening, we note that examining the RSS and SSS allows us to understand why TSS considers SSC to be better than

Original: HDP-HSMM – it captures more repeated structure.
Revised: HDP-HSMM – it captures more repeated structure.

Original: On the other hand Munkres, while it agrees with
Revised: On the other hand Munkres, while it agrees with

Original: [Table]
Revised: [Table]

Original: TSS, is unable to give us this insight.
Revised: TSS, is unable to give us this insight.

Original: We ﬁnd that NMI is highly sensitive to the number of clusters –
Revised: We ﬁnd that NMI is highly sensitive to the number of clusters –

Original: SSC only uses 2 clusters, which means it is heavily penalized by NMI, in spite of capturing some repeated structure.
Revised: SSC only uses 2 clusters, which means it is heavily penalized by NMI, in spite of capturing some repeated structure.

Original: ARI is less sensitive than NMI, but is far less interpretable.
Revised: ARI is less sensitive than NMI, but is far less interpretable.

Original: Another example is shown in Fig 6g, where the TSS for both SLDS and HDP-HMM is comparable.
Revised: Another example is shown in Fig 6g, where the TSS for both SLDS and HDP-HMM is comparable.

Original: Interestingly, NMI/ARI and Munkres are all highly skewed and disagree strongly on the better method.
Revised: Interestingly, NMI/ARI and Munkres are all highly skewed and disagree strongly on the better method.

Original: Since HDP-HMM has more clusters, the clusters are more homogenous and Hom is much higher, pushing up NMI.
Revised: Since HDP-HMM has more clusters, the clusters are more homogenous and Hom is much higher, pushing up NMI.

Original: On the other hand, since Munkres ﬁnds a one-to-one correspondence between clusters and ground-truth labels, having more clusters means some cannot be matched.
Revised: On the other hand, since Munkres ﬁnds a one-to-one correspondence between clusters and ground-truth labels, having more clusters means some cannot be matched.

Original: This causes the score to strongly downweigh suchh clusterings.
Revised: This causes the score to strongly downweigh suchh clusterings.

Original: Overall, our metrics provide both interpretability and the ability to analyze where the method is falling short.
Revised: Overall, our metrics provide both interpretability and the ability to analyze where the method is falling short.

Original: Lastly, we conduct a sensitivity analysis for β as follows: assume that the most appropriate value for evaluation is β (cid:48) , but we instead use the default value of β = 1 . 0 .
Revised: Lastly, we conduct a sensitivity analysis for β as follows: assume that the most appropriate value for evaluation is β (cid:48) , but we instead use the default value of β = 1 . 0 .

Original: Our main idea is to compute how well the ranking of methods under β = 1 . 

Revised: Our main idea is to compute how well the ranking of methods under β = 1 .

Original: captures that under β (cid:48) .
Revised: captures that under β (cid:48) .

Original: To do this, we use the Normalized
Revised: To do this, we use the Normalized

Original: Discounted Cumulative Gain (NDCG), a common criteria from information retrieval.
Revised: Discounted Cumulative Gain (NDCG), a common criteria from information retrieval.

Original: Varying β (cid:48) , we see in Fig 6i that β
Revised: Varying β (cid:48) , we see in Fig 6i that β

Original: = 1 . 0 outputs similar rankings to a relatively wide range of other β values, other than at the extremes.
Revised: = 1 . 0 outputs similar rankings to a relatively wide range of other β values, other than at the extremes.

Original: This conﬁrms that most settings of β will be captured well by our choice of β , unless there is a strong reason to prefer either the RSS or the SSS only.
Revised: This conﬁrms that most settings of β will be captured well by our choice of β , unless there is a strong reason to prefer either the RSS or the SSS only.

Original:  E XPERIMENTS WITH P RISM
Revised:  E XPERIMENTS WITH P RISM

Original: We now evaluate P RISM on complex datasets with noisy features, including video data.
Revised: We now evaluate P RISM on complex datasets with noisy features, including video data.

Original: For P RISM , we set α = 1 . 0 , β = 0 .
Revised: For P RISM , we set α = 1 . 0 , β = 0 .

Original: 1 in all experiments.
Revised: 1 (except B EES )

Original: For all methods, we set K (number of mixture components) to equal the number of ground-truth labels.
Revised: and we set K (number of mixture components) to equal the number of ground-truth labels for all methods.

Original: Results are averaged over 10 runs.
Revised: Results are averaged over 10 random seeds.

Revised: Experimental details including hyperparameters can be found in the Appendix.

Original: Before our main results, we brieﬂy conduct a simple check on performance for the B EES dataset introduced earlier.
Revised: Before our main results, we brieﬂy conduct a simple check on performance for the B EES dataset introduced earlier.

Original: Though this only has a single example per procedure, we ﬁnd that our approach still does well (Table 1) compared to two simple baselines – all approaches use the same observation model with Gaussian emissions.
Revised: Though this only has a single example per procedure, we ﬁnd that our approach still does well (Table 1) compared to two simple baselines – all approaches use the same observation model with Gaussian emissions.

Original: Leveraging a ﬁxed procedure.
Revised: Leveraging a ﬁxed procedure.

Original: We now do a simple sanity check on the beneﬁt of our approach on the JIGSAWS dataset (Gao et al., 2014), which consists of surgeons with varying skill demonstrating common procedures, such as suturing, needle-passing and knot-tying.
Revised: We now evaluate the beneﬁt of our approach on the JIGSAWS dataset (Gao et al., 2014), which consists of surgeons with varying skill demonstrating common procedures, such as suturing, needle-passing and knot-tying.

Original: We use kinematic features that correspond to the grippers controlled by the surgeon(details are in the Appendix) for all methods.
Revised: We use kinematic features that correspond to the grippers controlled by the surgeon for all methods.

Original: For each surgical task, we select the 10 demonstrations that correspond to the 2 expert surgeons since they adhere most closely to the prescribed procedures.
Revised: For each surgical task, we select the 10 demonstrations that correspond to the 2 expert surgeons since they adhere most closely to the prescribed procedures.

Original: Even for a single expert, there is considerable variability across the 5 demonstrations they generate, so this is far from an idealized setting.
Revised: Even for a single expert, there is considerable variability across thedemonstrations they generate, so this is far from an idealized setting.

Original: Once again, all methods leverage the same observational model with the same features.
Revised: Once again, all methods leverage the same observational model with the same features.

Original: We set s = 25 for

Original: P RISM .

Original: Our baselines are two simple approaches – a Gaussian mixture model (GMM), and variants of a Bayesian Hidden Markov model.
Revised: Our baselines are two standard approaches – a Gaussian mixture model (GMM), and variants of a Bayesian Hidden Markov model.

Original: As expected, P RISM outperforms these baselines by a considerable margin.
Revised: As expected, P RISM outperforms these baselines by a considerable margin.

Original: On the suturing task, performance is improve by almost double-digits on both metrics, with performance gains also visible on the other tasks.
Revised: On the suturing task, performance is improve by almost double-digits on both metrics, with performance gains also visible on the other tasks.

Original: Leveraging the common procedure allows P RISM to share statistical strength
Revised: Leveraging the common procedure allows P RISM to share statistical strength

Original: [Table]
Revised: [Table]

Original: across demonstrations and make consistent predictions where possible, whereas the other methods must rely more strongly on the observation model to infer the latent sequence.
Revised: across demonstrations and make consistent predictions where possible, whereas the other methods must rely more strongly on the observation model to infer the latent sequence.

Original: Without very cleanly separated features for different primitives, they can often be led astray.
Revised: Without very cleanly separated features for different primitives, they can often be led astray.

Original: We also experiment with a smaller subset of 5 demonstrations provided by a single expert on the suturing task.
Revised: We also experiment with a smaller subset of 5 demonstrations provided by a single expert on the suturing task.

Original: This subset of data has the least amount of variability in terms of the expert’s procedure.
Revised: This subset of data has the least amount of variability in terms of the expert’s procedure.

Original: Because of this, we see that P RISM gives an improvement of ∼ 18 points on NMI and ∼ 10 points on the TSS.
Revised: Because of this, we see that P RISM gives an improvement of ∼ 18 points on NMI and ∼ 10 points on the TSS.

Original: To illustrate P RISM ’s sensitivity to hyperparameters, we vary s and K in Fig 7 on the single-expert suturing task.
Revised: To illustrate P RISM ’s sensitivity to hyperparameters, we vary s and K in Fig 7 on this single-expert suturing task.

Original: We ﬁnd that P RISM ’s performance is quite stable with respect to both – this holds as long as s is larger than the number of segments in ground-truth, and K is close to the true number of ground-truth clusters.
Revised: We ﬁnd that P RISM ’s performance is quite stable with respect to both – this holds as long as s is larger than the number of segments in ground-truth, and K is close to the true number of ground-truth clusters.

Original: Noisy procedures with video data.
Revised: Noisy procedures with video data.

Original: Next, we consider labeling demonstrations that are very noisy in their adherence to the procedure, with a great deal of variability in both the steps being followed, their order, as well as their relative lengths.
Revised: Next, we consider labeling demonstrations that are very noisy in their adherence to the procedure, with a great deal of variability in both the steps being followed, their order, as well as their relative lengths.

Original: Recent work by Sener & Yao (2018) achieved state-of-the-art performance without any form of supervision on two large video datasets – Breakfast actions (Kuehne et al., 2014) and INRIA instructional videos (Alayrac et al., 2016).
Revised: Recent work by Sener & Yao (2018) achieved state-of-the-art performance without any form of supervision on two large video datasets – Breakfast actions (Kuehne et al., 2014) and INRIA instructional videos (Alayrac et al., 2016).

Original: We expect our method to perform worse on the Breakfast actions dataset, as it has no repeated structure, in which case our model has no advantages over this prior model, and indeed uses a simpler observation model.
Revised: We expect our method to perform worse on the Breakfast actions dataset, as it has no repeated structure, in which case our model has no advantages over this prior model, and indeed uses a simpler observation model.

Original: Perhaps encouragingly, in the Breakfast actions dataset, we ﬁnd that our method only performs slightly worse than but close to Sener & Yao (2018), achieving a Munkres score of 33.5 compared to their score of 34.6.
Revised: Perhaps encouragingly, in the Breakfast actions dataset, we ﬁnd that our method only performs slightly worse than but close to Sener & Yao (2018), achieving a Munkres score of 33.5 compared to their score of 34.6.

Original: It is likely that this can be improved by tuning the number of segments for each procedure (we set s = 20 for all procedures).
Revised: It is likely that this can be improved by tuning the number of segments for each procedure (we set s = 20 for all procedures).

Original: However a key limitation of their work is that they do not model repeated structure.
Revised: However a key limitation of their work is that they do not model repeated structure.

Original: Therefore in domains with repeated structure we expect our approach to do well.
Revised: Therefore in domains with repeated structure we expect our approach to do well.

Original: Such structure is present in some of the activities in the INRIA dataset.
Revised: Such structure is present in some of the activities in the INRIA dataset.

Original: We compare to Sener & Yao (2018)’s results on the
Revised: We compare to Sener & Yao (2018)’s results on the

Original: INRIA dataset in Table 2.
Revised: INRIA dataset in Table 2.

Original: P RISM outperforms their best method on 3 out of the 5 activities despite not explicitly modeling background information, with a signiﬁcant difference on cpr and jump car .
Revised: P RISM outperforms their best method on 3 out of the 5 activities despite not explicitly modeling background information, with a signiﬁcant difference on cpr and jump car .

Original: On their comparable method which does not model background, P RISM is signiﬁcantly better across all procedures.
Revised: On their comparable method which does not model background, P RISM is signiﬁcantly better across all procedures.

Original: This is despite P RISM using a much weaker observation model (they simultaneously learn a feature representation for the videos) and no permutation model.
Revised: This is despite P RISM using a much weaker observation model (they simultaneously learn a feature representation for the videos) and no permutation model.

Original: An example for the cpr procedure is shown in Fig 1 and illustrates that unlike Sener & Yao (2018)’s method, P RISM can successfully recover repeated structure both within and across videos.
Revised: An example for the cpr procedure is shown in Fig 1 and illustrates that unlike Sener & Yao (2018)’s method, P RISM can successfully recover repeated structure both within and across videos.

Original: We expect that incorporating a background model in P RISM could enable our approach to yield even stronger performance.
Revised: We expect that incorporating a background model in P RISM could enable our approach to yield even stronger performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We developed P RISM , a hierarchical Bayesian model to identify latent procedures in time-series data.
Revised: We developed P RISM , a hierarchical Bayesian model to identify latent procedures in time-series data.

Original: Results on several datasets show that P RISM can be used to learn procedural abstractions more effectively than competing methods.
Revised: Results on several datasets show that P RISM can be used to learn procedural abstractions more effectively than competing methods.

Original: We also discussed new evaluation criteria for the problem of external evaluating temporal clusterings.
Revised: We also introduced new evaluation criteria for the problem of externally evaluating temporal clusterings.

Original: Our metrics addressexisting gaps in temporal clustering evaluation, and provide both interpretability and the potential for easier analysis.
Revised: Our metrics address gaps in temporal clustering evaluation, and provide both interpretability and the potential for easier analysis.

Original: In future work, we hope to extend P RISM by incorporating richer observation models, and relaxing the assumption that a single strictly-ordered procedure is followed by all time-series.
Revised: In future work, we hope to extend P RISM by incorporating richer observation models, and relaxing the assumption that a single strictly-ordered procedure is followed by all time-series.

Revised: Links to code can be found in the Appendix.

Original:  A CKNOWLEDGMENTS
Revised:  A CKNOWLEDGMENTS

Original: The authors would like to thank the Schmidt Foundation and an NSF Career Grant for their support.
Revised: The authors would like to thank the Schmidt Foundation and a NSF Career Grant for their support.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A A LGORITHM FOR R EPEATED S TRUCTURE S CORE
Revised: A A LGORITHM FOR R EPEATED S TRUCTURE S CORE

Original: Detailed pseudocode for reproducing the computation of the RSS is given in Algorithm 1.
Revised: Detailed pseudocode for reproducing the computation of the RSS is given in Algorithm 1.

Original: Require: Ground truth G , temporal clustering C , similarity function H 1:
Revised: Require: Ground truth G , temporal clustering C , similarity function H 1:

Original: \\ Optional Pruning Step 2: for all α ∈ Γ C do β ∗ α ← arg max β ∈ Γ G |G [ β ] ∩ C
Revised: \\ Optional Pruning Step 2: for all α ∈ Γ C do β ∗ α ← arg max β ∈ Γ G |G [ β ] ∩ C

Original: [ α ] | (cid:46)
Revised: [ α ] | (cid:46)

Original: ﬁnd β ∗ α which most overlaps α 3: for all ( a, b ) ∈ S ( G , β ) , β ∈ Γ G do (cid:46) for every segment of every β in G 4:
Revised: ﬁnd β ∗ α which most overlaps α 3: for all ( a, b ) ∈ S ( G , β ) , β ∈ Γ G do (cid:46) for every segment of every β in G 4:

Original: P ( C a : b ) , W ( C a : b ) :
Revised: P ( C a : b ) , W ( C a : b ) :

Original: = ( p 1 , p 2 , . .
Revised: = ( p 1 , p 2 , . .

Original: , p r ) , ( w 1 , w 2 , . .
Revised: , p r ) , ( w 1 , w 2 , . .

Original: . , w r ) (cid:46) locate segment in C 5: ˜
Revised: . , w r ) (cid:46) locate segment in C 5: ˜

Original: w i ← w i × I [ β = β ∗ p i ] (cid:46) trim weights based on purity 6: ˜ W ( C a : b ) ← ( ˜ w 1 , ˜ w 2 , . .
Revised: w i ← w i × I [ β = β ∗ p i ] (cid:46) trim weights based on purity 6: ˜ W ( C a : b ) ← ( ˜ w 1 , ˜ w 2 , . .

Original: ˜ w r ) (cid:46) construct new weights 7: \\ Repeated Structure Scoring 8: score ← (cid:80) β ∈ Γ G (cid:80) ( a 1 ,b 1 ) , ( a 2 ,b 2 ) ∈S ( G ,β ) H ( P ( C a 1 : b 1 ) , P ( C a 2 : b 2 ) , ˜ W ( C a 1 : b 1 ) , ˜ W ( C a 2 : b 2 )) 9: (cid:46) score every repeated structure pair for every ground-truth label β 10: max score ← 2 (cid:80) β ∈ Γ G |S ( G , β ) | (cid:80) ( a,b ) ∈S ( G ,β ) ( b − a + 1) 11: return RSS ← score max score
Revised: ˜ w r ) (cid:46) construct new weights 7: \\ Repeated Structure Scoring 8: score ← (cid:80) β ∈ Γ G (cid:80) ( a 1 ,b 1 ) , ( a 2 ,b 2 ) ∈S ( G ,β ) H ( P ( C a 1 : b 1 ) , P ( C a 2 : b 2 ) , ˜ W ( C a 1 : b 1 ) , ˜ W ( C a 2 : b 2 )) 9: (cid:46) score every repeated structure pair for every ground-truth label β 10: max score ← 2 (cid:80) β ∈ Γ G |S ( G , β ) | (cid:80) ( a,b ) ∈S ( G ,β ) ( b − a + 1) 11: return RSS ← score max score

Original: B E XPERIMENTS WITH E VALUATION C RITERIA
Revised: B E XPERIMENTS WITH E VALUATION C RITERIA

Original: The results presented for the evaluation criteria developed in this paper can be reproduced using code available at https://github.com/StanfordAI4HI/ICLR2019_evaluating_ discrete_temporal_structure .
Revised: The results presented for the evaluation criteria developed in this paper can be reproduced using code available at https://github.com/StanfordAI4HI/ICLR2019_evaluating_ discrete_temporal_structure .

Original: The temporal clusterings predicted by each method, as well as ground-truth data is available at the link.
Revised: The temporal clusterings predicted by each method, as well as ground-truth data is available at the link.

Original: For completeness [Table]
Revised: In addition, an implementation of only the newly proposed temporal clustering evaluation criteria can be found at https://github.com/StanfordAI4HI/tclust-eval .

Original:  we include the result of evaluation across all methods, datasets and criteria
Revised: For completeness, we include the result of evaluation across all methods, datasets and criteria.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C D ETAILS FOR P RISM
Revised: C D ETAILS FOR P RISM

Original: This section contains details related to both the modeling and inference performed by P RISM .
Revised: This section contains details related to both the modeling and inference performed by P RISM .

Original: [Table]
Revised: [Table]

Original: 0.79 0.79 TSS 0.
Revised: 0.79 0.79 TSS 0.

Original: C. G ENERATION IN P RISM
Revised: C. G ENERATION IN P RISM

Original: The generative process used in the hierarchical Bayesian model is given below,
Revised: The generative process used in the hierarchical Bayesian model is given below,

Original: [Equation]
Revised: [Equation]

Original: C.
Revised: C.

Original: [Equation]
Revised: [Equation]

Original: I NFERENCE IN P RISM
Revised: I NFERENCE IN P RISM

Original: We use
Revised: We use

Original: Gibbs sampling to perform posterior inference of the latent variables ( { p r } r ∈
Revised: Gibbs sampling to perform posterior inference of the latent variables ( { p r } r ∈

Original: [ s ] , { u i,j } i ∈ [ n ] ,j ∈ [ m i ] , { µ k } k ∈
Revised: [ s ] , { u i,j } i ∈ [ n ] ,j ∈ [ m i ] , { µ k } k ∈

Original: [ K ] , { Σ k } k ∈ [ K ] ) in the model.
Revised: [ K ] , { Σ k } k ∈ [ K ] ) in the model.

Original: Sampling u i,j .
Revised: Sampling u i,j .

Original: Let Φ = (cid:0) { µ k } k ∈ [ K ] , { Σ k } k ∈
Revised: Let Φ = (cid:0) { µ k } k ∈ [ K ] , { Σ k } k ∈

Original: [ K ] (cid:1) , and write the complete conditional for u i,j ,
Revised: [ K ] (cid:1) , and write the complete conditional for u i,j ,

Original: Pr( u i,j = r | u − i , u − j i , . . . )
Revised: Pr( u i,j = r | u − i , u − j i , . . . )

Original: ∝ Pr( u i,j
Revised: ∝ Pr( u i,j

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst term we have collapsed the Dirichlet prior γ .
Revised: where in the ﬁrst term we have collapsed the Dirichlet prior γ .

Original: The main difﬁculty is in the datalikelihood terms Pr( x i,j | u i,j = r, . . . )
Revised: The main difﬁculty is in the datalikelihood terms Pr( x i,j | u i,j = r, . . . )

Original: , where we must reason about the effect of changing u i,j .
Revised: , where we must reason about the effect of changing u i,j .

Original: To reason about this effect, we must look at w ( r ) i
Revised: To reason about this effect, we must look at w ( r ) i

Original: = Sort( u i,j = r, u − j i ) , combine it with p to yield z ( r ) i and then compute the observation-likelihood terms.
Revised: = Sort( u i,j = r, u − j i ) , combine it with p to yield z ( r ) i and then compute the observation-likelihood terms.

Original: Due to the sorting step that yields w ( r ) i , changing a single u i,j affects the entire sequence z ( r ) i .
Revised: Due to the sorting step that yields w ( r ) i , changing a single u i,j affects the entire sequence z ( r ) i .

Original: Naively, we could compute Pr( x i,j | u i,j = r, . . . )
Revised: Naively, we could compute Pr( x i,j | u i,j = r, . . . )

Original: for every j ∈ [ m i ] , r ∈ [ s ] but this would be extremely-inefﬁcient (compute s · m i terms).
Revised: for every j ∈ [ m i ] , r ∈ [ s ] but this would be extremely-inefﬁcient (compute s · m i terms).

Original: Instead, we note that the assignments z ( r ) i will be almost identical for every r ∈ [ s ] since u − j
Revised: Instead, we note that the assignments z ( r ) i will be almost identical for every r ∈ [ s ] since u − j

Original: i is held ﬁxed and only u i,j is varied.
Revised: i is held ﬁxed and only u i,j is varied.

Original: The only points at which the z ( r ) i are not identical are those right before and right after segment boundaries – O ( s ) such locations.
Revised: The only points at which the z ( r ) i are not identical are those right before and right after segment boundaries – O ( s ) such locations.

Original: We only need to compute the data-likelihood at these locations since no other locations are changed by sampling u i,j .
Revised: We only need to compute the data-likelihood at these locations since no other locations are changed by sampling u i,j .

Original: Using this insight, we only need to compute s 2 terms, a big reduction since we expect s << m i .
Revised: Using this insight, we only need to compute s 2 terms, a big reduction since we expect s << m i .

Original: We sample all the u i,j ’s sequentially using this update.
Revised: We sample all the u i,j ’s sequentially using this update.

Original: Sampling p r .
Revised: Sampling p r .

Original: To sample p r , we write the complete conditional,
Revised: To sample p r , we write the complete conditional,

Original: [Equation]
Revised: [Equation]

Original: Let z ( k ) i be the temporal clustering generated by combining ( p r = k, p − r ) with w i .
Revised: Let z ( k ) i be the temporal clustering generated by combining ( p r = k, p − r ) with w i .

Original: Note that in each time-series, setting p r =
Revised: Note that in each time-series, setting p r =

Original: k affects only those observations that are assigned to p r by w i .
Revised: k affects only those observations that are assigned to p r by w i .

Original: We 
Revised: We thus need to compute the observation-likelihood of only these terms,

Original: [Table]
Revised: [Equation]

Original: thus need to compute the observation-likelihood of only these terms,
Revised: n [Equation] i = m i (cid:89)

Original: [Equation]
Revised: [Equation]

Revised: Pr(

Revised: [Equation]

Original: We sample the p r ’s sequentially using this update.
Revised: We sample the p r ’s sequentially using this update.

Original: Sampling µ k , Σ k .
Revised: Sampling µ k , Σ k .

Original: Sampling for the observation model can be done directly from the posterior of the Normal-Wishart Inverse distribution.
Revised: Sampling for the observation model can be done directly from the posterior of the Normal-Wishart Inverse distribution.

Original: We set uninformed priors for the observation model, µ 0 = 

Revised: We set uninformed priors for the observation model, µ 0 =

Original: d , Σ 0 = I d × d , κ 

Revised: d , Σ 0 = I d × d , κ

Original: = 1 , ν 0 =
Revised: = 1 , ν 0 =

Original: d + 2 .
Revised: d + 2 .

Original: D N ON -M ARKOV S IMULATION S TUDY
Revised: D E XPERIMENTS WITH P RISM

Revised: The results presented for P RISM can be reproduced using code available at https://github. com/StanfordAI4HI/ICLR2019_prism , which contains a Python implementation of P RISM . D. D ATASETS

Revised: We now provide details for preprocessing each dataset, with data loaders available at the linked code.

Revised: JIGSAWS.

Revised: For the JIGSAWS dataset, we use the ﬁrst 38 kinematic features.

Revised: We subsample every 3rd timestep.

Revised: To preprocess the data, we perform Principal Components Analysis (PCA) for dimensionality reduction to 12 features.

Revised: We then stack features from 10 time-steps and perform PCA again to reduce dimensionality to 10 features.

Revised: Lastly, we standardize features to have zero mean and unit variance.

Revised: Breakfast actions.

Revised: We use the Fisher vector features taken from the ‘stereo01’ camera without any other preprocessing.

Revised: We use the coarse segmentation for evaluation.

Revised: INRIA instructional videos.

Revised: We use PCA to reduce the dimensionality of the features to 64 .

Revised: Bees.

Revised: We use 4 features – x, y, cos θ and sin θ .

Revised: D. E XPERIMENT DETAILS

Revised: We use the hyperparameter settings given in Table 10 for P RISM .

Revised: The Bayesian HMM has a single hyperparameter α which represents the hyperparameter for the Dirichlet prior over the transition matrix.

Revised: Hyperparameter

Revised: [Table]

Revised: D. R ESULTS ON B REAKFAST A CTIONS

Revised: Detailed results for the Breakfast actions dataset are summarized in Table 11.

Revised: [Table]

Revised: [Table]

Revised: D. N ON -M ARKOV S IMULATION S TUDY

Original: We generate 10 time-series using the following simple non-Markov sequence,
Revised: We generate 10 time-series using the following simple non-Markov sequence,

Original: AABBAACCAADDAAEEAAFFAAGGAAHHAAAAAAAA
Revised: AABBAACCAADDAAEEAAFFAAGGAAHHAAAAAAAA

Original: There are 8 distinct latent tokens in the sequence, and we generate noisy Gaussian 2-dimensional observations for each such token, using a mean of ( i, i ) for the i th token, and a standard-deviation of 0 .
Revised: There are 8 distinct latent tokens in the sequence, and we generate noisy Gaussian 2-dimensional observations for each such token, using a mean of ( i, i ) for the i th token, and a standard-deviation of 0 .

Original: 35 in either dimension.
Revised: 35 in either dimension.

Original: We compare the Hidden Markov model variants to P RISM and the Gaussian mixture model.
Revised: We compare the Hidden Markov model variants to P RISM and the Gaussian mixture model.

Original: All methods use the same observational model with Gaussian emissions.
Revised: All methods use the same observational model with Gaussian emissions.

Original: We expect that the main advantage of using an HMM vis-a-vis a GMM for recovering the latent temporal clustering, is in cases where the observations are noisy.
Revised: We expect that the main advantage of using an HMM vis-a-vis a GMM for recovering the latent temporal clustering, is in cases where the observations are noisy.

Original: In such cases, the learned transition dynamics can override the (noisy) observational likelihood to yield the correct sequence of tokens, as long as the procedure is Markov.
Revised: In such cases, the learned transition dynamics can override the (noisy) observational likelihood to yield the correct sequence of tokens, as long as the procedure is Markov.

Original: However, this beneﬁt should disappear in cases where the procedure is not Markov, like the sequence constructed by us above.
Revised: However, this beneﬁt should disappear in cases where the procedure is not Markov, like the sequence constructed by us above.

Original: In contrast to these methods, P RISM makes no Markov assumption about the underlying procedure.
Revised: In contrast to these methods, P RISM makes no Markov assumption about the underlying procedure.

Original: This allows it to learn any procedure, Markov or otherwise.
Revised: This allows it to learn any procedure, Markov or otherwise.

Original: In addition, by sharing this procedure across multiple time-series, P RISM can avoid being led astray by noisy observations (unlike the
Revised: In addition, by sharing this procedure across multiple time-series, P RISM can avoid being led astray by noisy observations (unlike the

Original: GMM, which relies purely on the observation to infer the local latent token).
Revised: GMM, which relies purely on the observation to infer the local latent token).

Original: This argument is conﬁrmed by the results in Table 10.
Revised: This argument is conﬁrmed by the results in Table 12.

Original: P RISM outperforms all methods on both evaluation metrics, while the HMM variants suffer due to their inability to abandon the Markov assumption that is baked into the model.
Revised: P RISM outperforms all methods on both evaluation metrics, while the HMM variants suffer due to their inability to abandon the Markov assumption that is baked into the model.