Original: Variational Beam Search for Novelty Detection
Revised: Variational Beam Search for Novelty Detection

Original: Anonymous Authors

Original: Anonymous Institution

Original: Abstract
Revised: Abstract

Original: We consider the problem of online learning in the presence of sudden distribution shifts, which may be hard to detect and can lead to a slow but steady degradation in model performance.
Revised: We consider the problem of online learning in the presence of sudden distribution shifts, which may be hard to detect and can lead to a slow but steady degradation in model performance.

Original: To address this problem we propose a new Bayesian meta-algorithm that can both (i) make inferences about subtle distribution shifts based on minimal sequential observations and (ii) accordingly adapt a model in an online fashion.
Revised: To address this problem we propose a new Bayesian meta-algorithm that can both (i) make inferences about subtle distribution shifts based on minimal sequential observations and (ii) accordingly adapt a model in an online fashion.

Original: The approach uses beam search over multiple change point hypotheses to perform inference on a hierarchical sequential latent variable modeling framework.
Revised: The approach uses beam search over multiple change point hypotheses to perform inference on a hierarchical sequential latent variable modeling framework.

Original: Our proposed approach is model-agnostic, applicable to both supervised and unsupervised learning, and yields signiﬁcant improvements over state-of-the-art Bayesian online learning approaches.
Revised: Our proposed approach is model-agnostic, applicable to both supervised and unsupervised learning, and yields signiﬁcant improvements over state-of-the-art Bayesian online learning approaches

Original: 1. Introduction
Revised: 1. Introduction

Original: Deployed machine learning systems are often faced with the problem of distribution shift, where the new data that the model processes is systematically different from the data the system was trained on.
Revised: Deployed machine learning systems are often faced with the problem of distribution shift, where the new data that the model processes is systematically different from the data the system was trained on.

Original: Furthermore, the shift can happen anytime after deployment, unbeknownst to the users, with dramatic consequences for systems such as self-driving cars, robots, trading algorithms, among many other examples.
Revised: Furthermore, the shift can happen anytime after deployment, unbeknownst to the users, with dramatic consequences for systems such as self-driving cars, robots, trading algorithms, among many other examples.

Original: Updating a deployed model on new, representative data can help mitigate these issues, as well as improve on general performance in most cases.
Revised: Updating a deployed model on new, representative data can help mitigate these issues, as well as improve on general performance in most cases.

Original: This task is commonly referred to as online learning .
Revised: This task is commonly referred to as online learning .

Original: A particular variant of online learning that focuses on adapting models to new or novel data (in either features and/or outputs) over time is known as continual learning .
Revised: A particular variant of online learning that focuses on adapting models to new or novel data (in either features and/or outputs) over time is known as continual learning .

Original: Approaches developed on this task typically focus on mitigating the degradation of performance over earlier data, often referred to as catastrophic forgetting .
Revised: Approaches developed on this task typically focus on mitigating the degradation of performance over earlier data, often referred to as catastrophic forgetting .

Original: For instance, variational continual learning (VCL) (Nguyen et al., 2017) employs a Bayesian online learning framework to prevent forgetting by modeling the model’s prior distribution for each new batch of data (referred to as a “task" in the continual learning literature) as the posterior from the previous one.
Revised: For instance, variational continual learning (VCL) (Nguyen et al., 2017) employs a Bayesian online learning framework to prevent forgetting by modeling the model’s prior distribution for each new batch of data (referred to as a “task" in the continual learning literature) as the posterior from the previous one.

Original: VCL and other Bayesian solutions are typically robust to catastrophic forgetting; however, they may suffer from an opposite problem that we are deﬁning as catastrophic remembering .
Revised: VCL and other Bayesian solutions are typically robust to catastrophic forgetting; however, they may suffer from an opposite problem that we are deﬁning as catastrophic remembering .

Original: In continually training a Bayesian model, the posterior distribution becomes progressively more and more conﬁdent as more data is used to inform it.
Revised: In continually training a Bayesian model, the posterior distribution becomes progressively more and more conﬁdent as more data is used to inform it.

Original: Given enough data, the model’s prior distribution (i.e. previous posterior distribution) will be too conﬁdent to adequately adapt to new data affected by a distributional shift (for brevity, we will refer to this shift as a novelty ).
Revised: Given enough data, the model’s prior distribution (i.e. previous posterior distribution) will be too conﬁdent to adequately adapt to new data affected by a distributional shift (for brevity, we will refer to this shift as a novelty ).

Original: In this paper, we propose a more robust approach that copes with the problems of both catastrophic forgetting and remembering.
Revised: In this paper, we propose a more robust approach that copes with the problems of both catastrophic forgetting and remembering.

Original: The idea is that the model not only requires a good mechanism to aggregate data, but also is able to partially forget information that has become obsolete.
Revised: The idea is that the model not only requires a good mechanism to aggregate data, but also is able to partially forget information that has become obsolete.

Original: To achieve this, we still use the Bayesian online learning framework; however, before combining the previously learned posterior with new data evidence, we introduce an intermediate step.
Revised: To achieve this, we still use the Bayesian online learning framework; however, before combining the previously learned posterior with new data evidence, we introduce an intermediate step.

Original: This step allows the model to decide between two actions to take: reduce the previous posterior’s conﬁdence to provide more “room” for novel information, or remain in the same state (i.e. retain the unchanged, previous posterior as the new prior).
Revised: This step allows the model to decide between two actions to take: reduce the previous posterior’s conﬁdence to provide more “room” for novel information, or remain in the same state (i.e. retain the unchanged, previous posterior as the new prior).

Original: We propose a mechanism for enabling this decision using a “spike and slab” novelty prior (described in Section 2.1).
Revised: We propose a mechanism for enabling this decision using a “spike and slab” novelty prior (described in Section 2.1).

Original: We further augment this decision process by introducing variational beam search , a new inference scheme that allows the model to consider multiple different hypothetical sequences of detected distributional shifts (or lack thereof).
Revised: We further augment this decision process by introducing variational beam search , a new inference scheme that allows the model to consider multiple different hypothetical sequences of detected distributional shifts (or lack thereof).

Original: We present experiments for Bayesian deep learning experiments using real datasets with artiﬁcially introduced shifts over time, as well as unsupervised experiments for analyzing semantic change over time in language (Section C.3 in supplement).
Revised: We present experiments for Bayesian deep learning experiments using real datasets with artiﬁcially introduced shifts over time, as well as unsupervised experiments for analyzing semantic change over time in language (Section C.3 in supplement).

Original: Our approach both leads to more semantic and compact word embeddings, as well as signiﬁcantly improves performance in the supervised tasks.
Revised: Our approach both leads to more semantic and compact word embeddings, as well as signiﬁcantly improves performance in the supervised tasks.

Original: 2. Methods
Revised: 2. Methods

Original: We consider a stream of data that arrives in batches ("tasks") x t at discrete times t .
Revised: We consider a stream of data that arrives in batches ("tasks") x t at discrete times t .

Original: For notational simplicity we focus on the unsupervised case, where the task is to model p ( x t ) using a model p ( x t | z t ) with parameters z t that we would like to optimally tune to each new batch 1 .
Revised: For notational simplicity we focus on the unsupervised case, where the task is to model p ( x t ) using a model p ( x t | z t ) with parameters z t that we would like to optimally tune to each new batch 1 .

Original: We furthermore assume that while the x t are i.i.d.
Revised: We furthermore assume that while the x t are i.i.d.

Original: within batches, they are not i.i.d.
Revised: within batches, they are not i.i.d.

Original: across batches, but rather come from a time-varying distribution p t ( x t ) (or p t ( x t , y t ) in the supervised cases) which is subject to distribution shifts.
Revised: across batches, but rather come from a time-varying distribution p t ( x t ) (or p t ( x t , y t ) in the supervised cases) which is subject to distribution shifts.

Original: We assume that these distribution shifts occur instantaneously (as opposed to gradually) and at unknown times, i.e., a change may (or may not) occur with each batch t and (if it occurs) will persist until the next change.
Revised: We assume that these distribution shifts occur instantaneously (as opposed to gradually) and at unknown times, i.e., a change may (or may not) occur with each batch t and (if it occurs) will persist until the next change.

Original: The challenge is to optimally adapt the parameters z t to each new task while borrowing statistical strength from previous tasks.
Revised: The challenge is to optimally adapt the parameters z t to each new task while borrowing statistical strength from previous tasks.

Original: Variational Continual Learning
Revised: Variational Continual Learning

Original: The basis of our approach is the insight that for sequential data, one can use a Bayesian model’s posterior at time t − 1 as a prior for the next task at time t .
Revised: The basis of our approach is the insight that for sequential data, one can use a Bayesian model’s posterior at time t − 1 as a prior for the next task at time t .

Original: Since typically the posterior is not available in closed-form, we must use approximate inference.
Revised: Since typically the posterior is not available in closed-form, we must use approximate inference.

Original: It is natural to use a variational posterior q t − 1 ( z t ) .
Revised: It is natural to use a variational posterior q t − 1 ( z t ) .

Original: This leads to a sequence of variational inference tasks (Zhang et al., 2018) known as variational continual learning (VCL)
Revised: This leads to a sequence of variational inference tasks (Zhang et al., 2018) known as variational continual learning (VCL)

Original: (Nguyen et al., 2017):
Revised: (Nguyen et al., 2017):

Original: [Equation]
Revised: [Equation]

Original:  where Q is the family of potential approximate posterior distributions (i.e. normal distributions).
Revised: ) where Q is the family of potential approximate posterior distributions (i.e. normal distributions).

Original: Eq. 1 is known as the evidence lower bound (ELBO) in variationa inference (Jordan et al., 1999;
Revised: Eq. 1 is known as the evidence lower bound (ELBO) in variationa inference (Jordan et al., 1999;

Original: Zhang et al., 2018); for every new task it is optimized until convergence.
Revised: Zhang et al., 2018); for every new task it is optimized until convergence.

Original: Note that in its original formulation, the goal of VCL is to train a model that performs well cumulatively on all the learning tasks previously encountered (Nguyen et al., 2017), as opposed to just the most recent task, as studied in this paper.
Revised: Note that in its original formulation, the goal of VCL is to train a model that performs well cumulatively on all the learning tasks previously encountered (Nguyen et al., 2017), as opposed to just the most recent task, as studied in this paper.

Original: Catastrophic Remembering
Revised: Catastrophic Remembering

Original: While continual learning mainly addresses catastrophic forgetting, where a model loses in performance on the tasks previously encountered, we address the opposite effect: catastrophic remembering (French, 1999).
Revised: While continual learning mainly addresses catastrophic forgetting, where a model loses in performance on the tasks previously encountered, we address the opposite effect: catastrophic remembering (French, 1999).

Original: Here, a model becomes overconﬁdent with time and loses its ability to adapt to distribution shifts.
Revised: Here, a model becomes overconﬁdent with time and loses its ability to adapt to distribution shifts.

Original: In Bayesian online learning, such catastrophic remembering is caused by an overconﬁdent posterior.
Revised: In Bayesian online learning, such catastrophic remembering is caused by an overconﬁdent posterior.

Original: To explain this effect, we note that a posterior’s variance shrinks as it encounters more data.
Revised: To explain this effect, we note that a posterior’s variance shrinks as it encounters more data.

Original: Assume that after a long sequence of updates, the posterior q t − 1 ( z t ) can be well-approximated by a point mass
Revised: Assume that after a long sequence of updates, the posterior q t − 1 ( z t ) can be well-approximated by a point mass

Original: δ ( z t − z 0 ) centered around some parameter z 0 .
Revised: δ ( z t − z 0 ) centered around some parameter z 0 .

Original: In this case, any new data evidence p ( x t | z t ) will have a diminishing effect on the next posterior q t ( z t ) as q t ( z t ) ∝ p ( x t | z t ) δ ( z t − z 0 )
Revised: In this case, any new data evidence p ( x t | z t ) will have a diminishing effect on the next posterior q t ( z t ) as q t ( z t ) ∝ p ( x t | z t ) δ ( z t − z 0 )

Original: ≈ q t − 1 ( z t ) , in other words, the strong prior over-rules the new data even though the training data that lead to it may have become obsolete due to a distribution shift.
Revised: ≈ q t − 1 ( z t ) , in other words, the strong prior over-rules the new data even though the training data that lead to it may have become obsolete due to a distribution shift.

Original: To prevent catastrophic remembering, we adapt Eq.
Revised: To prevent catastrophic remembering, we adapt Eq.

Original: 1 to the online learning scenario, speciﬁcally assuming irregular and instantaneous distribution shifts.
Revised: 1 to the online learning scenario, speciﬁcally assuming irregular and instantaneous distribution shifts.

Original: 2.1. Posterior Broadening Mechanisms
Revised: 2.1. Posterior Broadening Mechanisms

Original: In order to combat catastrophic remembering, the posterior in Eq.
Revised: In order to combat catastrophic remembering, the posterior in Eq.

Original: 1 needs to be broadened before it is combined with new data evidence.
Revised: 1 needs to be broadened before it is combined with new data evidence.

Original: This broadening mechanism erases learned information to free-up model capacity to adjust to the new data distribution.
Revised: This broadening mechanism erases learned information to free-up model capacity to adjust to the new data distribution.

Original: Among several possible options, we consider relative broadening , which amounts to tempering the prior by a ﬁxed amount, resulting in p β ( z t )
Revised: Among several possible options, we consider relative broadening , which amounts to tempering the prior by a ﬁxed amount, resulting in p β ( z t )

Original: ∝ q t − 1 ( z t ) β for 0 < β ≤ 1 .
Revised: ∝ q t − 1 ( z t ) β for 0 < β ≤ 1 .

Original: For a Gaussian q with diagonal variances σ 2 i in dimension z i , relative broadening removes an equal amount of information in each dimension, H i = 12 log(2 πeσ 2 i /β 2 )
Revised: For a Gaussian q with diagonal variances σ 2 i in dimension z i , relative broadening removes an equal amount of information in each dimension, H i = 12 log(2 πeσ 2 i /β 2 )

Original: = 12 log(2 πeσ 2 i ) − log β .
Revised: = 12 log(2 πeσ 2 i ) − log β .

Original: Since tempering broadens the posterior non-locally, this scheme does not possess a continuous latent time series interpretation 2 .
Revised: Since tempering broadens the posterior non-locally, this scheme does not possess a continuous latent time series interpretation 2 .

Original: Conditional Broadening
Revised: Conditional Broadening

Original: If novelties happened at a predictable and constant rate, we were done: tuning the parameter β to the expected rate of change (with a large β for a high change rate and β = 1 for no expected change).
Revised: If novelties happened at a predictable and constant rate, we were done: tuning the parameter β to the expected rate of change (with a large β for a high change rate and β = 1 for no expected change).

Original: However, in reality, novelties can be of varying strength, irregular, and unobserved .
Revised: However, in reality, novelties can be of varying strength, irregular, and unobserved .

Original: We therefore propose to model the novelty at time t with a binary latent variable s t , with s t = 0 for no change occurring, and s t
Revised: We therefore propose to model the novelty at time t with a binary latent variable s t , with s t = 0 for no change occurring, and s t

Original: = 1 indicating a shift.
Revised: = 1 indicating a shift.

Original: For s t
Revised: For s t

Original: = 1 , we broaden the posterior and use it as a prior.
Revised: = 1 , we broaden the posterior and use it as a prior.

Original: If no change occurs, we just use the previous posterior as the new prior and proceed.
Revised: If no change occurs, we just use the previous posterior as the new prior and proceed.

Original: This gives rise the the following conditional prior:
Revised: This gives rise the the following conditional prior:

Original: [Equation]
Revised: [Equation]

Original: We deﬁned the tempered approximate posterior at time t − 1 as q βt − 1 ( z t ) =
Revised: We deﬁned the tempered approximate posterior at time t − 1 as q βt − 1 ( z t ) =

Original: [Equation]
Revised: [Equation]

Revised: (cid:82) (

Revised: [Equation]

Original: Note that q β has a closed-form expression for a Gaussian q .)
Revised: 1 ( z [Equation] β d z t . (Note that q β has a closed-form expression for a Gaussian q .)

Original: The conditional broadening approach leads to the following online inference scheme:
Revised: The conditional broadening approach leads to the following online inference scheme:

Original: [Equation]
Revised: [Equation]

Original: This involves a joint variational distribution q ( z t | s t ) over the latents z t and change variable s t .
Revised: This involves a joint variational distribution q ( z t | s t ) over the latents z t and change variable s t .

Original: We place a Bernoulli prior p ( s t ) on the change variables.
Revised: We place a Bernoulli prior p ( s t ) on the change variables.

Original: As described below, we jointly infer z t and the change variables s t from data.
Revised: As described below, we jointly infer z t and the change variables s t from data.

Original: 2.2. Bayesian Reasoning over Distribution Shifts Structured Variational Inference
Revised: 2.2. Bayesian Reasoning over Distribution Shifts Structured Variational Inference

Original: According to our assumptions, the novelties occur at discrete times and are unobserved.
Revised: According to our assumptions, the novelties occur at discrete times and are unobserved.

Original: Therefore, both z t and s t have to be inferred from data.
Revised: Therefore, both z t and s t have to be inferred from data.

Original: We ﬁrst deﬁne the marginal likelihood over data given the change variable s t to be p β ( xt | s t ) :=
Revised: We ﬁrst deﬁne the marginal likelihood over data given the change variable s t to be p [Equation] x

Original: [Equation]
Revised: [Equation]

Original: which is often intractable.
Revised: which is often intractable.

Original: This intractability leaves the exact posterior over s t , p β ( s t | x t ) not available as well (by Bayes rule).
Revised: This intractability leaves the exact posterior over s t , p β ( s t | x t ) not available as well (by Bayes rule).

Original: However, we can follow a structured variational inference approach (Wainwright and Jordan, 2008; Hoffman and Blei, 2015; Zhang et al., 2018), deﬁning a joint variational distribution q ( z t , s t ) =
Revised: However, we can follow a structured variational inference approach (Wainwright and Jordan, 2008; Hoffman and Blei, 2015; Zhang et al., 2018), deﬁning a joint variational distribution q ( z t , s t ) =

Original: q ( s t ) q
Revised: q ( s t ) q

Original: ( z t | s t ) .
Revised: ( z t | s t ) .

Original: By deﬁnition, the conditional distributions q ( z t | s t ) are solutions to the optimization problem in
Revised: By deﬁnition, the conditional distributions q ( z t | s t ) are solutions to the optimization problem in

Original: Eq. 3.
Revised: Eq. 3.

Original: Absorbing the optimized conditional ELBO in Eq. 3, q ( s t ) has a closed-form solution:
Revised: Absorbing the optimized conditional ELBO in Eq. 3, q ( s t ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: where σ is the sigmoid function and ξ 

Revised: where σ is the sigmoid function and ξ

Original: = log p ( s t = 1) − log p ( s t = 0) are the log-odds of the prior p ( s t ) .
Revised: = log p ( s t = 1) − log p ( s t = 0) are the log-odds of the prior p ( s t ) .

Original: This speciﬁes the posterior over s t .
Revised: This speciﬁes the posterior over s t .

Original: We provide the derivation details, the resemblance to the exact inference, and the interpretation as a likelihood ratio test in the Supplement.
Revised: We provide the derivation details, the resemblance to the exact inference, and the interpretation as a likelihood ratio test in the Supplement.

Original: Finally, we obtain the marginal distribution over latent variables q ( z t ) at time t as a binary mixture with mixture weights q ( s t = 0)
Revised: Finally, we obtain the marginal distribution over latent variables q ( z t ) at time t as a binary mixture with mixture weights q ( s t = 0)

Original: = m
Revised: = m

Original: and q ( s t = 1) = 1 − m :
Revised: and q ( s t = 1) = 1 − m :

Original: [Equation]
Revised: [Equation]

Original: Exponential Branching
Revised: Exponential Branching

Original: We note that while we had originally started with a Gaussian variational posterior q t − 1 ( z t ) at the previous time, our inference scheme resulted in q t ( z t ) being a mixture of two Gaussians: 3 the inference scheme branches over two alternative hypotheses.
Revised: We note that while we had originally started with a Gaussian variational posterior q t − 1 ( z t ) at the previous time, our inference scheme resulted in q t ( z t ) being a mixture of two Gaussians: 3 the inference scheme branches over two alternative hypotheses.

Original: When we iterate, we encounter an exponential branching of possibilities, or hypotheses over possible sequences of regime shifts.
Revised: When we iterate, we encounter an exponential branching of possibilities, or hypotheses over possible sequences of regime shifts.

Original: To still be able to carry out the ﬁltering scheme efﬁciently, we need a truncation scheme, e.g., approximate the bimodal marginal distribution by a unimodal one.
Revised: To still be able to carry out the ﬁltering scheme efﬁciently, we need a truncation scheme, e.g., approximate the bimodal marginal distribution by a unimodal one.

Original: The next section will discuss several methods to achieve this goal.
Revised: The next section will discuss several methods to achieve this goal.

Original: 2.3. Online Inference and Variational Beam Search
Revised: 2.3. Online Inference and Variational Beam Search

Original: While the previous subsections have focused on a single update, we now investigate the possibility of doing multiple updates in a row.
Revised: While the previous subsections have focused on a single update, we now investigate the possibility of doing multiple updates in a row.

Original: This amounts to working out a truncation scheme to restrict the variational posterior to be a unimodal Gaussian ( Variational Greedy Search (VGS)) or, a mixture of ﬁxed size ( Variational Beam Search (VBS)) at every time step.
Revised: This amounts to working out a truncation scheme to restrict the variational posterior to be a unimodal Gaussian ( Variational Greedy Search (VGS)) or, a mixture of ﬁxed size ( Variational Beam Search (VBS)) at every time step.

Original: The simplest VGS trains the model in an online fashion by iterating over time steps t .
Revised: The simplest VGS trains the model in an online fashion by iterating over time steps t .

Original: For each t , it explores all possible branches and then truncates the branches.
Revised: For each t , it explores all possible branches and then truncates the branches.

Original: In general, VGS ﬁrst optimizes the conditional ELBO (Eq. 3) for both s t = 0 and s t = 1 (which corresponds to branches).
Revised: In general, VGS ﬁrst optimizes the conditional ELBO (Eq. 3) for both s t = 0 and s t = 1 (which corresponds to branches).

Original: With the optimized conditional ELBO, it computes the binary mixture weights using Eq. 4.
Revised: With the optimized conditional ELBO, it computes the binary mixture weights using Eq. 4.

Original: Finally VGS projects q t ( z t )
Revised: Finally VGS projects q t ( z t )

Original: (Eq. 5) on the more probable component.
Revised: (Eq. 5) on the more probable component.

Original: Detailed steps are in the Supplement.
Revised: Detailed steps are in the Supplement.

Original: Note that VGS updates z t at every time step t , resulting in continuously changing variational parameters µ t and σ t .
Revised: Note that VGS updates z t at every time step t , resulting in continuously changing variational parameters µ t and σ t .

Original: The fact that µ t and σ t change even between two detected change points makes the output of the algorithm poorly interpretable.
Revised: The fact that µ t and σ t change even between two detected change points makes the output of the algorithm poorly interpretable.

Original: One can get a better ﬁt if one outputs the variational parameters µ t and σ t at the end of a segment of constant z t , just before a detected change point s t = 1 .
Revised: One can get a better ﬁt if one outputs the variational parameters µ t and σ t at the end of a segment of constant z t , just before a detected change point s t = 1 .

Original: We call this a “shy” variant of VGS.
Revised: We call this a “shy” variant of VGS.

Original: More details are in supplement.
Revised: More details are in supplement.

Original: VGS and its shy variant make greedy decisions on s t , in the sense that they ignore the subsequent tasks.
Revised: VGS and its shy variant make greedy decisions on s t , in the sense that they ignore the subsequent tasks.

Original: A greedy search is prone to missing change points in data sets with a low signal/noise ratio per time step because it cannot accumulate evidence for a change point over a series of time steps.
Revised: A greedy search is prone to missing change points in data sets with a low signal/noise ratio per time step because it cannot accumulate evidence for a change point over a series of time steps.

Revised: Material.

Original: The most obvious improvement over greedy search that has the ability to accumulate evidence for a change point is beam search.
Revised: The most obvious improvement over greedy search that has the ability to accumulate evidence for a change point is beam search.

Original: Rather than deciding greedily whether a change occurred or not at each time step, beam search considers both cases in parallel, and it delays the decision as to which one is more likely.
Revised: Rather than deciding greedily whether a change occurred or not at each time step, beam search considers both cases in parallel [Table] and it delays the decision as to which one is more likely.

Original: Empirically, we ﬁnd that the naive beam search procedure does not reveal its full potential.
Revised: Empirically, we ﬁnd that the naive beam search procedure does not reveal its full potential.

Original: As commonly encountered in beam search, histories over change points are largely shared among all members of the beam.
Revised: As commonly encountered in beam search, histories over change points are largely shared among all members of the beam.

Original: We thus developed a simple beam diversiﬁcation scheme to encourage diverse beams.
Revised: We thus developed a simple beam diversiﬁcation scheme to encourage diverse beams.

Original: We found this beam diversiﬁcation scheme to work robustly across a variety of experiments.
Revised: We found this beam diversiﬁcation scheme to work robustly across a variety of experiments.

Original: More details about diversiﬁed VBS are described in the supplement
Revised: More details about diversiﬁed VBS are described in the supplement

Original: [Table]
Revised: [Table]

Original: 3. Experiments Overview
Revised: 3. Experiments Overview

Original: The objective of our experiments is to show that compared to other methods, variational beam search (VBS) (1) better reacts to novelties in supervised setups, while (2) revealing interpretable and temporally sparse latent structure in unsupervised setups.
Revised: The objective of our experiments is to show that compared to other methods, variational beam search (VBS) (1) better reacts to novelties in supervised setups, while (2) revealing interpretable and temporally sparse latent structure in unsupervised setups.

Original: To this end, we experiment on artiﬁcial data (Section 3.1), study Bayesian deep learning approaches on sequences of transformed CIFAR and SVHN images (3.2)
Revised: To this end, we experiment on artiﬁcial data (Section 3.1), study Bayesian deep learning approaches on sequences of transformed CIFAR and SVHN images (3.2)

Original: [Table]
Revised: [Table]

Original: and study the dynamics of word embeddings on historical text corpora (Section C.3 in supplement).
Revised: and study the dynamics of word embeddings on historical text corpora (Section C.3 in supplement).

Original: Additional details are in the Supplement.
Revised: Additional details are in the Supplement

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 3.1. Toy Experiments
Revised: 3.1. Toy Experiments

Original: To test VBS in a very simple setup [Table] we simulated noisy data points centered around a piecewiseconstant step function with two steps (Fig. 1).
Revised: To test VBS in a very simple setup, we simulated noisy data points centered around a piecewiseconstant step function with two steps (Fig. 1).

Original: The task is to infer the mean (black line) of a time-varying data distribution (black samples).
Revised: The task is to infer the mean (black line) of a time-varying data distribution (black samples).

Original: We evaluated VBS with beam sizes 1 and 2 in terms of their ability to correctly identify the latent jumps in hindsight.
Revised: We evaluated VBS with beam sizes 1 and 2 in terms of their ability to correctly identify the latent jumps in hindsight.

Original: Both algorithms start with similar performance initially.
Revised: Both algorithms start with similar performance initially.

Original: However, VBS with beam size 1 (“greedy”, orange) fails to recognize the two-fold jump correctly.
Revised: However, VBS with beam size 1 (“greedy”, orange) fails to recognize the two-fold jump correctly.

Original: In contrast, beam size 2 operates with two hypotheses over function levels.
Revised: In contrast, beam size 2 operates with two hypotheses over function levels.

Original: Around step 20, the initially unlikely hypothesis with two jumps becomes the dominant one.
Revised: Around step 20, the initially unlikely hypothesis with two jumps becomes the dominant one.

Original: 3.2. Supervised Experiments
Revised: 3.2. Supervised Experiments

Original: Next, we considered a supervised learning setup, in which an algorithm is exposed to a sequence classiﬁcation tasks, consisting of batches of CIFAR-10 and SVHN images.
Revised: Next, we considered a supervised learning setup, in which an algorithm is exposed to a sequence classiﬁcation tasks, consisting of batches of CIFAR-10 and SVHN images.

Original: To introduce novelties, every few tasks we transform all images globally by combining rotations, shifts, and scaling transformations.
Revised: To introduce novelties, every few tasks we transform all images globally by combining rotations, shifts, and scaling transformations.

Original: To make the approach compatible with our framework, we used a Bayesian convolutional neural network and applied variational beam search to the network’s weights.
Revised: To make the approach compatible with our framework, we used a Bayesian convolutional neural network and applied variational beam search to the network’s weights.

Original: We focused on the latest task performance and evaluated the classiﬁcation accuracy on a test set subject to the same transformations.
Revised: We focused on the latest task performance and evaluated the classiﬁcation accuracy on a test set subject to the same transformations.

Original: Datasets
Revised: [Figure]

Original: We used two standard datasets for image classiﬁcation: CIFAR-10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011), adopting the original training and test set splits. We further split the training set into batches for online learning, each batch consisting of a third of the full data.
Revised: l a t i v e E rr o r [Figure]

Original: [Figure] l a t i v e
Revised: c t i o n t o V C L

Original: [Figure]
Revised: [Figure]

Original: r f o r m a n ce t o V C
Revised: We used two standard datasets for image classiﬁcation: CIFAR-10 (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011), adopting the original training and test set splits.

Original: L
Revised: We further split the training set into batches for online learning, each batch consisting of a third of the full data.

Original: [Figure] Each transformation (either rotation, translation, or scaling) is generated from a ﬁxed, predeﬁned distribution (see Supplement C.2).
Revised: Each transformation (either rotation, translation, or scaling) is generated from a ﬁxed, predeﬁned distribution (see Supplement C.2).

Original: Changes are introduced every three tasks, where the total number of tasks was 100.
Revised: Changes are introduced every three tasks, where the total number of tasks was 100.

Original: Fig 2 (top panel) shows typical resulting transformations.
Revised: Fig 2 (top panel) shows typical resulting transformations.

Original: Baselines.
Revised: Baselines.

Original: In our supervised experiments, we compared VBS against established Bayesian online learning baselines and an independent batch learning baseline.
Revised: In our supervised experiments, we compared VBS against established Bayesian online learning baselines and an independent batch learning baseline.

Original: In addition to Variational Continual
Revised: In addition to Variational Continual

Original: Learning (VCL, see Section 2), we also compared against Laplace Propagation (LP) (Smola et al., 2003).
Revised: Learning (VCL, see Section 2), we also compared against Laplace Propagation (LP) (Smola et al., 2003).

Original: We refer to (Nguyen et al., 2017) for technical descriptions of LP.
Revised: We refer to (Nguyen et al., 2017) for technical descriptions of LP.

Original: Finally, we also adopt a trivial baseline: learning independent classiﬁers on each task.
Revised: Finally, we also adopt a trivial baseline: learning independent classiﬁers on each task.

Original: Here, we adopt a non-Bayesian neural network with the same architecture.
Revised: Here, we adopt a non-Bayesian neural network with the same architecture.

Original: See Supplement for more details.
Revised: See Supplement for more details.

Original: Architectures and Protocol.
Revised: Architectures and Protocol.

Original: All Bayesian methods use the same neural network architecture.
Revised: All Bayesian methods use the same neural network architecture.

Original: We used a truncated version of the VGG convolutional neural network (Supplementary Material C.2) on both datasets and conﬁrmed that our architecture achieved similar performance on CIFAR compared to the results reported by Zenke et al.
Revised: We used a truncated version of the VGG convolutional neural network (Supplementary Material C.2) on both datasets and conﬁrmed that our architecture achieved similar performance on CIFAR compared to the results reported by Zenke et al.

Original: (2017) and Lopez-Paz and Ranzato (2017) in a similar setting.
Revised: (2017) and Lopez-Paz and Ranzato (2017) in a similar setting.

Original: We stress that lower accuracies are obtained in our online learning setting due to the distribution shifts.
Revised: We stress that lower accuracies are obtained in our online learning setting due to the distribution shifts.

Original: We initialize each algorithm by training the model on the full, untransformed dataset.
Revised: We initialize each algorithm by training the model on the full, untransformed dataset.

Original: We set the relative broadening β = 2 / 3 for all supervised experiments.
Revised: We set the relative broadening β = 2 / 3 for all supervised experiments.

Original: During every new task, all algorithms are trained until convergence (see Supplement for details).
Revised: During every new task, all algorithms are trained until convergence (see Supplement for details).

Original: Results
Revised: Results

Original: The bottom panel of Fig.
Revised: The bottom panel of Fig.

Original: 2 shows our main results on CIFAR-10 (left) and SVHN (right).
Revised: 2 shows our main results on CIFAR-10 (left) and SVHN (right).

Original: To account for varying task difﬁculties, we show the  relative performance improvement relative to our main baseline, VCL.
Revised: To account for varying task difﬁculties, we show the percentage of the relative error reduction relative to our main baseline, VCL.

Original: VBS with a large beam size of K = 6 performs best, followed by VBS with K = 3 .
Revised: VBS with a large beam size of K = 6 performs best, followed by VBS with K = 3 .

Original: Variational greedy search (VGS), which corresponds to a beam size K = 1 , performed comparably with  VCL.
Revised: Variational greedy search (VGS), which corresponds to a beam size K = 1 , performed comparably with and slightly better than VCL.

Original: The reason is that, empirically, the greedy version of our algorithm only detected a small fraction of distribution shifts.
Revised: The reason is that, empirically, the greedy version of our algorithm only detected a small fraction of distribution shifts.

Original: This makes its performance  similar to VCL.
Revised: This makes its performance sometimes similar to VCL.

Original: This stresses the importance of beam search: for larger K , multiple changes were detected.
Revised: This stresses the importance of beam search: for larger K , multiple changes were detected.

Original: Table 1 shows the absolute performances of all considered methods, averaged across all of the 100 tasks for the two datasets.
Revised: Table 1 shows the absolute performances of all considered methods, averaged across all of the 100 tasks for the two datasets.

Original: Our proposed methods improved  over the best-performing baseline VCL by 1 . 1 percentage points on SVHN and by 3 percentage points on CIFAR-10.
Revised: Our proposed methods improved signiﬁcantly over the best-performing baseline VCL by 1 . 1 percentage points on SVHN and by 3 percentage points on CIFAR-10.

Original: The effect of beam search is also evident, with larger beam sizes consistently performing better.
Revised: The effect of beam search is also evident, with larger beam sizes consistently performing better.

Original: Appendix A. Structured Variational Inference
Revised: Appendix A. Structured Variational Inference

Original: According to the main paper we consider the genenrative model p β ( x t  z t , s t ) 
Revised: According to the main paper, we consider the generative model p β ( x t , z t , s t ) =

Original: = p ( s t ) p β ( z t | s t ) p ( x t | z t ) at time step t .
Revised: p ( s t ) p β ( z t | s t ) p ( x t | z t ) at time step t .

Original: Upon observing the data x t  both z t and s t are inferred.
Revised: Upon observing the data x t , both z t and s t are inferred.

Original: However, exact inference is not available due to the intractability of the marginal likelihood p β ( x t | s t ) .
Revised: However, exact inference is not available due to the intractability of the marginal likelihood p β ( x t | s t ) .

Original: To tackle this, we utilize structured variational inference for both the latent variables z t and the Bernoulli change variable s t .
Revised: To tackle this, we utilize structured variational inference for both the latent variables z t and the Bernoulli change variable s t .

Original: Towards this end, we deﬁne the joint variational distribution q ( z t , s t ) =
Revised: Towards this end, we deﬁne the joint variational distribution q ( z t , s t ) =

Original: q ( s t ) q
Revised: q ( s t ) q

Original: ( z t | s t ) .
Revised: ( z t | s t ) .

Original: Then the updating procedure for q ( s t ) and q ( z t | s t ) is obtained by maximizing the ELBO L ( q ) :
Revised: Then the updating procedure for q ( s t ) and q ( z t | s t ) is obtained by maximizing the ELBO L ( q ) :

Original: [Equation]
Revised: [Equation]

Original: Given the generative models, we can further expand L ( q ) to simplify the optimization:
Revised: Given the generative models, we can further expand L ( q ) to simplify the optimization:

Original: [Equation]
Revised: [Equation]

Original:  where the second step pushes inside the expectation with respect to q ( z t | s t ) , the third step re-orders the terms, and the ﬁnal step utilizes the deﬁnition of conditional ELBO (Eq. 3 in the main paper).
Revised: 6) where the second step pushes inside the expectation with respect to q ( z t | s t ) , the third step re-orders the terms, and the ﬁnal step utilizes the deﬁnition of conditional ELBO (Eq. 3 in the main paper).

Original: It therefore implies a two-step optimization to maximize L ( q ) :
Revised: It therefore implies a two-step optimization to maximize L ( q ) :

Original: ﬁrst maximize the conditional
Revised: ﬁrst maximize the conditional

Original: ELBO L ( q | s t ) to ﬁnd the optimal q t ( z t | s t = 1) and q t ( z t | s t = 0) , respectively, then compute the
Revised: ELBO L ( q | s t ) to ﬁnd the optimal q t ( z t | s t = 1) and q t ( z t | s t = 0) , respectively, then compute the

Original: Bernoulli distribution q ∗ ( s t ) by maximizing L ( q ) in which the conditional ELBOs L ( q t | s t ) are ﬁxed.
Revised: Bernoulli distribution q ∗ ( s t ) by maximizing L ( q ) in which the conditional ELBOs L ( q t | s t ) are ﬁxed.

Original: While q t ( z t | s t ) typically needs to be inferred by black box variational inference (Ranganath et al., 2014; Kingma and Welling, 2013; Zhang et al., 2018), the optimal q ∗ ( s t ) has a closed-form solution and bears resemblance to the exact inference counterpart (Eq. 7 in the main paper).
Revised: While q t ( z t | s t ) typically needs to be inferred by black box variational inference (Ranganath et al., 2014; Kingma and Welling, 2013; Zhang et al., 2018), the optimal q ∗ ( s t ) has a closed-form solution and bears resemblance to the exact inference counterpart (Eq. 7 in the main paper).

Original: To see this, we assume L ( q t | s t ) are present and q ( s t ) is parameterized by m ∈ R (for the Bernoulli distribution).
Revised: To see this, we assume L ( q t | s t ) are present and q ( s t ) is parameterized by m ∈ R (for the Bernoulli distribution).

Original: Rewriting Eq.
Revised: Rewriting Eq.

Original: 6 gives
Revised: 6 gives

Original: [Equation]
Revised: [Equation]

Original: which is concave due to the second derivative
Revised: which is concave due to the second derivative

Original: is negative.
Revised: is negative.

Original: Thus taking the derivative and setting it to zero leads to the optimal solution of
Revised: Thus taking the derivative and setting it to zero leads to the optimal solution of

Original: [Equation]
Revised: [Equation]

Original: which attains the closed-form solution as stated in Eq.
Revised: which attains the closed-form solution as stated in Eq.

Original: 4 in the main paper.
Revised: 4 in the main paper.

Original: Exact Inference
Revised: Exact Inference

Original: To see the similarities to the exact inference, recall p β ( x t | s t 
Revised: To see the similarities to the exact inference, recall p β ( x t | s t )

Original: [Equation]

Original: = (cid:82) p [Equation] x t | z [Equation]
Revised: = (cid:82) p ( x t | z t ) p β ( z t | s t ) d z t .

Original: p

Original: [Equation]

Original: ( z

Original: [Equation]

Original: | s t ) d z t .

Original: The exact posterior over s t is again a Bernoulli p β
Revised: The exact posterior over s t is again a Bernoulli p β

Original: ( s | x t ) =
Revised: ( s | x t ) =

Original: Bern( s t ; m ) with parameter
Revised: Bern( s t ; m ) with parameter

Original: [Equation]
Revised: [Equation]

Original: where σ is the sigmoid function and ξ 

Revised: where σ is the sigmoid function and ξ

Original: = log p ( s t = 1) − log p ( s t = 0) are the log-odds of the prior p ( s t ) .
Revised: = log p ( s t = 1) − log p ( s t = 0) are the log-odds of the prior p ( s t ) .

Original: Eq.
Revised: Eq.

Original: 7 has a simple interpretation as a likelihood ratio test: a change is more or less likely depending on whether or not the observations x t are better explained under the assumption of the broadened prior distribution q βt − 1 ( z t ) , in other words, a partial reset of previously learned information.
Revised: 7 has a simple interpretation as a likelihood ratio test: a change is more or less likely depending on whether or not the observations x t are better explained under the assumption of the broadened prior distribution q βt − 1 ( z t ) , in other words, a partial reset of previously learned information.

Original: Then we demonstrate the resemblance between the exact inference and the approximate inference for s t .
Revised: Then we demonstrate the resemblance between the exact inference and the approximate inference for s t .

Original: Notice the conditional ELBO in Eq.
Revised: Notice the conditional ELBO in Eq.

Original: 3 is a lower bound to the logarithm of the marginal likelihood log p β ( x t | s t ) , we can use the former as a proxy of the latter in Eq. 7, resulting in the update in Eq. 4.
Revised: 3 is a lower bound to the logarithm of the marginal likelihood log p β ( x t | s t ) , we can use the former as a proxy of the latter in Eq. 7, resulting in the update in Eq. 4.

Original: In other words, the tighter the conditional ELBO to log p β ( x t | s t ) , q ( s t ) is more precise.
Revised: In other words, the tighter the conditional ELBO to log p β ( x t | s t ) , q ( s t ) is more precise.

Original: Appendix B. Details on Online Learning and Variational Beam Search Variational Greedy Search
Revised: Appendix B. Details on Online Learning and Variational Beam Search Variational Greedy Search

Original: For each t , VGS udpates a truncated variational distribution via the following three steps:
Revised: For each t , VGS updates a truncated variational distribution via the following three steps:

Original: Compute the conditional prior p β ( z t | s t )
Revised: Compute the conditional prior p β ( z t | s t )

Original: (Eq. 2) based on q t − 1 and optimize the conditional ELBO (Eq. 3) for both s t = 0 and s t
Revised: (Eq. 2) based on q t − 1 and optimize the conditional ELBO (Eq. 3) for both s t = 0 and s t

Original: = 1 .
Revised: = 1 .

Original: This results in the optimized variational distributions q ( z t | s t = 0) and q ( z t | s t = 1) .
Revised: This results in the optimized variational distributions q ( z t | s t = 0) and q ( z t | s t = 1) .

Original: Compute the binary mixture weights m and (1 − m ) using Eq. 4, resulting in q t ( z t ) 
Revised: Compute the binary mixture weights m and (1 − m ) using Eq. 4, resulting in q t ( z t ) =

Original: = m q ( z t | s t = 0)
Revised: m q ( z t | s t = 0)

Original: + (1 − m ) q
Revised: + (1 − m ) q

Original: ( z t | s t = 1) (Eq. 5).
Revised: ( z t | s t = 1) (Eq. 5).

Original: Truncate q t ( z t ) to a uni-modal distribution to avoid branching.
Revised: Truncate q t ( z t ) to a uni-modal distribution to avoid branching.

Original: This can be achieved by projecting q t ( z t ) on the more probable component.
Revised: This can be achieved by projecting q t ( z t ) on the more probable component.

Original: The ﬁltering algorithm mentioned above iteratively updates the variational distribution over z t each time it observes new data x t .
Revised: The ﬁltering algorithm mentioned above iteratively updates the variational distribution over z t each time it observes new data x t .

Original: In the version of variational greedy search discussed above, the approach decides immediately, i.e., before observing subsequent data points, whether a change in z t has occurred (by projecting on one mixture component in the truncation step) or not.
Revised: In the version of variational greedy search discussed above, the approach decides immediately, i.e., before observing subsequent data points, whether a change in z t has occurred (by projecting on one mixture component in the truncation step) or not.

Original: This approach is illustrated in the blue part of Fig.
Revised: This approach is illustrated in the blue part of Fig.

Original: 3 (a).
Revised: 3 (a).

Original: Here, the dark blue line shows the ﬁtted mean µ t over time steps t , with ± 1 σ error bars in light blue.
Revised: Here, the dark blue line shows the ﬁtted mean µ t over time steps t , with ± 1 σ error bars in light blue.

Original: The fact that µ t and σ t change even between two detected change points makes the output of the algorithm poorly interpretable.
Revised: The fact that µ t and σ t change even between two detected change points makes the output of the algorithm poorly interpretable.

Original: “Shy” Variational Greedy Search
Revised: “Shy” Variational Greedy Search

Original: One obtains a better ﬁt if one outputs the variational parameters µ t and σ t at the end of a segment of constant z t .
Revised: One obtains a better ﬁt if one outputs the variational parameters µ t and σ t at the end of a segment of constant z t .

Original: More precisely, when the algorithm detects a change point s t
Revised: More precisely, when the algorithm detects a change point s t

Original: = 1 , it outputs the variational parameters µ t − 1 and σ t − 1 from just before the detected change point t .
Revised: = 1 , it outputs the variational parameters µ t − 1 and σ t − 1 from just before the detected change point t .

Original: These parameters deﬁne a variational distribution that has been ﬁtted, in an iterative way, to all data points since the preceding detected change point.
Revised: These parameters deﬁne a variational distribution that has been ﬁtted, in an iterative way, to all data points since the preceding detected change point.

Original: We call this the “shy” variant of the variational greedy search algorithm, because this variant quietly iterates over the data and only outputs a new ﬁt when it is as certain about it as it will ever be.
Revised: We call this the “shy” variant of the variational greedy search algorithm, because this variant quietly iterates over the data and only outputs a new ﬁt when it is as certain about it as it will ever be.

Original: The red lines and regions in Fig.
Revised: The red lines and regions in Fig.

Original: 3 (a) illustrate means and standard deviations outputted by the “shy” variant of variational greedy search.
Revised: 3 (a) illustrate means and standard deviations outputted by the “shy” variant of variational greedy search.

Original: Variational Beam Search
Revised: Variational Beam Search

Original: As we see in the main paper, the above two algorithm variants (“greedy” and “shy”) discussed so far are greedy in the detection of change points, prone to missing change points in noisy data sets.
Revised: As we see in the main paper, the above two algorithm variants (“greedy” and “shy”) discussed so far are greedy in the detection of change points, prone to missing change points in noisy data sets.

Original: An obvious improvement that can delay decisions to accumulate evidence is beam search.
Revised: An obvious improvement that can delay decisions to accumulate evidence is beam search.

Original: This algorithm is illustrated schematically in Fig.
Revised: This algorithm is illustrated schematically in Fig.

Original: 3 (b).
Revised: 3 (b).

Original: The algorithm keeps track of a ﬁxed number K > 1 of possible histories, i.e., sequences of change points.
Revised: The algorithm keeps track of a ﬁxed number K > 1 of possible histories, i.e., sequences of change points.

Original: For each history, it iteratively updates a Gaussian variational distribution as in the greedy variants.
Revised: For each history, it iteratively updates a Gaussian variational distribution as in the greedy variants.

Original: At each time step t , each history splits up into two for the two cases s t ∈ { 0 , 1 } , thus doubling the number of histories of which the algorithm has to keep track of.
Revised: At each time step t , each history splits up into two for the two cases s t ∈ { 0 , 1 } , thus doubling the number of histories of which the algorithm has to keep track of.

Original: To keep the computational requirements bounded, beam search thus discards half of the histories based on an exploration-exploitation trade-off.
Revised: To keep the computational requirements bounded, beam search thus discards half of the histories based on an exploration-exploitation trade-off.

Original: As follows, we present a more detailed explanation of the variational beam search procedure outlined in Section 2.2 of the main paper.
Revised: As follows, we present a more detailed explanation of the variational beam search procedure outlined in Section 2.2 of the main paper.

Original: Our beam search procedure deﬁnes an effective way to search for potential hypotheses with regards to sequences of inferred change points.
Revised: Our beam search procedure deﬁnes an effective way to search for potential hypotheses with regards to sequences of inferred change points.

Original: The procedure is completely deﬁned by detailing three sequential steps, that when executed, take a set of hypotheses found at time step t − 1 and transform them into the resulting set of likely hypotheses for time step t that have appropriately accounted for the new data seen at t .
Revised: The procedure is completely deﬁned by detailing three sequential steps, that when executed, take a set of hypotheses found at time step t − 1 and transform them into the resulting set of likely hypotheses for time step t that have appropriately accounted for the new data seen at t .

Original: The red arrows in Figure 4 illustrate these three steps for beam search with a beam size of K = 4 .
Revised: The red arrows in Figure 4 illustrate these three steps for beam search with a beam size of K = 4 .

Original: In Figure 4, each of the three steps maps a table of considered histories to a new table.
Revised: In Figure 4, each of the three steps maps a table of considered histories to a new table.

Original: Each table deﬁnes a mixture of Gaussian distributions where each mixture component corresponds to a different history and is represented by different a row in the table.
Revised: Each table deﬁnes a mixture of Gaussian distributions where each mixture component corresponds to a different history and is represented by different a row in the table.

Original: We start on the left with the (truncated) variational distribution q t − 1 ( z t − 1 ) from the previous time step, which is a mixture over
Revised: We start on the left with the (truncated) variational distribution q t − 1 ( z t − 1 ) from the previous time step, which is a mixture over

Original: K = 4 Gaussian distributions.
Revised: K = 4 Gaussian distributions.

Original: Each mixture component (row in the table) is labeled by a 0-1 vector s <t
Revised: Each mixture component (row in the table) is labeled by a 0-1 vector s <t

Original: = ( s 0 , s 1 , · · · , s t − 1 ) of the change variable values according to that history.
Revised: = ( s 0 , s 1 , · · · , s t − 1 ) of the change variable values according to that history.

Original: Each mixture component s <t further has a mixture weight q ( s <t ) ∈ [0 , 1] , a mean, and a standard deviation.
Revised: Each mixture component s <t further has a mixture weight q ( s <t ) ∈ [0 , 1] , a mean, and a standard deviation.

Original: We then obtain a prior for time step t by transporting each mixture component of q t − 1 ( z t − 1 ) forward in time via the broadening functional (“Step 1” in the above ﬁgure).
Revised: We then obtain a prior for time step t by transporting each mixture component of q t − 1 ( z t − 1 ) forward in time via the broadening functional (“Step 1” in the above ﬁgure).

Original: The prior p β ( z t ) (second table in the ﬁgure) is a mixture of 2 K Gaussian distributions because each previous history splits into two new ones for the two potential cases s t ∈ { 0 , 1 } .
Revised: The prior p β ( z t ) (second table in the ﬁgure) is a mixture of 2 K Gaussian distributions because each previous history splits into two new ones for the two potential cases s t ∈ { 0 , 1 } .

Original: The label for each mixture component (table row) is a new vector ( s <t , s t ) or s <t +1 , appending s t to the tail of s <t .
Revised: The label for each mixture component (table row) is a new vector ( s <t , s t ) or s <t +1 , appending s t to the tail of s <t .

Original: “Step 2” in the above ﬁgure takes the data x t and ﬁts a variational distribution q t ( z t ) that is also a mixture of 2 K Gaussian distributions.
Revised: “Step 2” in the above ﬁgure takes the data x t and ﬁts a variational distribution q t ( z t ) that is also a mixture of 2 K Gaussian distributions.

Original: To learn the variational distribution, we (i) numerically ﬁt each mixture component q ( z t | s <t , s t ) individually, using the corresponding mixture component of p β ( z t ) as the prior; (ii) evaluate (or estimate) the conditional ELBO of each ﬁtted mixture component, conditioned on ( s <t , s t ) ; (iii) compute the approximate posterior probability q ∗ ( s t ) of each mixture component, in the presence of the conditional ELBOs; and (iv) obtain the mixture weight equal to the posterior probability over ( s <t , s t ) , best approximated by q ( s <t ) q ∗ ( s t ) .
Revised: To learn the variational distribution, we (i) numerically ﬁt each mixture component q ( z t | s <t , s t ) individually, using the corresponding mixture component of p β ( z t ) as the prior; (ii) evaluate (or estimate) the conditional ELBO of each ﬁtted mixture component, conditioned on ( s <t , s t ) ; (iii) compute the approximate posterior probability q ∗ ( s t ) of each mixture component, in the presence of the conditional ELBOs; and (iv) obtain the mixture weight equal to the posterior probability over ( s <t , s t ) , best approximated by q ( s <t ) q ∗ ( s t ) .

Original: “Step 3” in the above ﬁgure truncates the variational distribution by discarding K of the 2 K mixture components.
Revised: “Step 3” in the above ﬁgure truncates the variational distribution by discarding K of the 2 K mixture components.

Original: The truncation scheme can be either the “vanilla” beam search or diversiﬁed beam search outlined in the main paper.
Revised: The truncation scheme can be either the “vanilla” beam search or diversiﬁed beam search outlined in the main paper.

Original: The truncated variational distribution q t ( z t ) is again a mixture of only K Gaussian distributions, and it can thus be used for subsequent update steps, i.e., from t to t + 1 .
Revised: The truncated variational distribution q t ( z t ) is again a mixture of only K Gaussian distributions, and it can thus be used for subsequent update steps, i.e., from t to t + 1 .

Original: Beam Search Diversiﬁcation
Revised: Beam Search Diversiﬁcation

Original: To encourage diverse beams, we constructed the following simple scheme: Let K be the number of hypotheses in a beam.
Revised: To encourage diverse beams, we constructed the following simple scheme: Let K be the number of hypotheses in a beam.

Original: While transitioning from time t − 1 to t , every hypothesis splits into two scenarios, one with s t = 0 and one with s t = 1 , resulting in 2 K hypotheses.
Revised: While transitioning from time t − 1 to t , every hypothesis splits into two scenarios, one with s t = 0 and one with s t = 1 , resulting in 2 K hypotheses.

Original: If two resulting hypotheses only differ in their most recent s t -value, we say that they come from the same "family."
Revised: If two resulting hypotheses only differ in their most recent s t -value, we say that they come from the same "family."

Original: Each member among the 2 K hypotheses is ranked according to its
Revised: Each member among the 2 K hypotheses is ranked according to its

Original: ELBO value (Eq. 3).
Revised: ELBO value (Eq. 3).

Original: In a ﬁrst step, we discard the bottom 1 / 3 of the 2 K hypotheses, leaving 4 / 3 K hypotheses.
Revised: In a ﬁrst step, we discard the bottom 1 / 3 of the 2 K hypotheses, leaving 4 / 3 K hypotheses.

Original: (We always take integer multiples of 3 for K ).
Revised: (We always take integer multiples of 3 for K ).

Original: To truncate the beam size from 4 / 3 K down to K , we rank the remaining hypotheses according to their ELBO and pick the top K ones
Revised: To truncate the beam size from 4 / 3 K [Table]

Original: [Table]
Revised: [Table]

Revised: down to K

Original: [Table]
Revised: [Table]

Original:  while also ensuring that we pick a member from every remaining family.
Revised: we rank the remaining hypotheses according to their ELBO and pick the top K ones while also ensuring that we pick a member from every remaining family.

Original: The diversiﬁcation scheme ensures that underperforming families can survive, leading to a more diverse set of hypotheses.
Revised: The diversiﬁcation scheme ensures that underperforming families can survive [Table] leading to a more diverse set of hypotheses [Table]

Original: Appendix C. Experiment Details and Results C.1. Toy Data Experiments Data Generating Process
Revised: Appendix C. Experiment Details and Results C.1. Toy Data Experiments Data Generating Process

Original: To generate Figure 1 in the main paper, we used a step-wise function as ground truth, where the step size was 1 and two step positions were chosen randomly.
Revised: To generate Figure 1 in the main paper [Table] we used a step-wise function as ground truth, where the step size was 1 and two step positions were chosen randomly.

Original: We sampledequally-spaced points with time spacing 1.
Revised: We sampledequally-spaced points with time spacing 1.

Original: To get noisy observations, Gaussian noise with standard deviation 0.5 was added to the points.
Revised: To get noisy observations, Gaussian noise with standard deviation 0.5 was added to the points.

Original: Model Parameters
Revised: Model Parameters

Original: In this simple one-dimensional model, we used absolute broadening with a
Revised: In this simple one-dimensional model, we used absolute broadening with a

Original: Gaussian transition kernel K ( z t , z (cid:48) t ) =
Revised: Gaussian transition kernel K ( z t , z (cid:48) t ) =

Original: N ( z t − z (cid:48) t , D ∆ t ) where D = 1 . 0 and ∆ t = 1 .
Revised: N ( z t − z (cid:48) t , D ∆ t ) where D = 1 . 0 and ∆ t = 1 .

Original: The inference is thus tractable because p ( z t | s t ) is conditional conjugate to p ( x t |
Revised: The inference is thus tractable because p ( z t | s t ) is conditional conjugate to p ( x t |

Original: z t , s t ) (and both are Gaussian distributed).
Revised: z t , s t ) (and both are Gaussian distributed).

Original: We set the prior log-odds (cid:15) 0 to log p ( s t =1) p ( s t =0) , where p ( s t = 1) = 0 .
Revised: We set the prior log-odds (cid:15) 0 to log p ( s t =1) p ( s t =0) , where p ( s t = 1) = 0 .

Original: We used beam size 1 and beam size 2 to do the inference, respectively.
Revised: We used beam size 1 and beam size 2 to do the inference, respectively.

Original: We reported the results with beam size 2, which also involved the resulting hypothesis for beam size 1 (greedy search) as the one with lower probability.
Revised: We reported the results with beam size 2, which also involved the resulting hypothesis for beam size 1 (greedy search) as the one with lower probability.

Original: C.2. Bayesian Deep Learning Experiments Transformations
Revised: C.2. Bayesian Deep Learning Experiments Transformations

Original: We used Albumentations (Buslaev et al., 2020) to implement the transformations as covariate shifts.
Revised: We used Albumentations (Buslaev et al., 2020) to implement the transformations as covariate shifts.

Original: As stated in the main paper, the transformation involved rotation, scaling, and translation.
Revised: As stated in the main paper, the transformation involved rotation, scaling, and translation.

Original: Each transformation factor followed a ﬁxed distribution: rotation degree conformed to N (0 , 10 2 ) ; scaling limit conformed to N (0 , 0 .
Revised: Each transformation factor followed a ﬁxed distribution: rotation degree conformed to N (0 , 10 2 ) ; scaling limit conformed to N (0 , 0 .

Original: ; and the magnitude of vertical and horizontaltranslation limit conformed to Beta (1 , 10) , and the sampled magnitude is then rendered positive or
Revised: ; and the magnitude of vertical and horizontal

Original: [Table]
Revised: [Table]

Original:  negative with equal probability.
Revised: translation limit conformed to Beta (1 , 10) , and the sampled magnitude is then rendered positive or negative with equal probability.

Original: The ﬁnal scaling and translation factor should be the corresponding sampled limit plus 1, respectively.
Revised: The ﬁnal scaling and translation factor should be the corresponding sampled limit plus 1, respectively.

Original: Neural Network Architecture
Revised: Neural Network Architecture

Original: We used the same convolutional neural network architecture for both datasets, which can be found in Table 2.
Revised: We used the same convolutional neural network architecture for both datasets, which can be found in Table 2.

Original: We implemented the Bayesian models using TensorFlow
Revised: We implemented the Bayesian models using TensorFlow

Original: Probability and the non-Bayesian counterpart (namely Laplace Propagation) using TensorFlow Keras.
Revised: Probability and the non-Bayesian counterpart (namely Laplace Propagation) using TensorFlow Keras.

Original: Every bias term in all the models were treated deterministically and were not affected by any regularization.
Revised: Every bias term in all the models were treated deterministically and were not affected by any regularization.

Original: Tempered Conditional ELBO
Revised: Tempered Conditional ELBO

Original: In the presence of massive observations and a large neural network, posterior distributions of change variables usually have very low entropy because of the very large magnitude of the difference between conditional ELBOs as in Eq.
Revised: In the presence of massive observations and a large neural network, posterior distributions of change variables usually have very low entropy because of the very large magnitude of the difference between conditional ELBOs as in Eq.

Original: Therefore change variables become over conﬁdent about the switch-state decisions.
Revised: Therefore change variables become over conﬁdent about the switch-state decisions.

Original: The situation gets even more severe in beam search settings where almost all probability mass is centered around the top hypothesis while the other hypotheses get little probability and thereby will not take effect in predictions.
Revised: The situation gets even more severe in beam search settings where almost all probability mass is centered around the top hypothesis while the other hypotheses get little probability and thereby will not take effect in predictions.

Original: A possible solution is to temper the conditional ELBO (or the marginal likelihood) and introduce more uncertainty into the change variables.
Revised: A possible solution is to temper the conditional ELBO (or the marginal likelihood) and introduce more uncertainty into the change variables.

Original: To this end, we divide the conditional ELBO by the number of observations.
Revised: To this end, we divide the conditional ELBO by the number of observations.

Original: While the tempering strategy has the model no longer work in strict Bayesian paradigm, it renders every hypothesis effective in beam search setting.
Revised: While the tempering strategy has the model no longer work in strict Bayesian paradigm, it renders every hypothesis effective in beam search setting.

Original: Hyperparameters, Initialization, and Model Training
Revised: Hyperparameters, Initialization, and Model Training

Original: The hyperparameters used across all of the models for the different datasets are listed in Tables 3 and 4.
Revised: The hyperparameters used across all of the models for the different datasets are listed in Tables 3 and 4.

Original: Regarding the model-speciﬁc parameters, we set ξ 0 to 0 for both datasets and searched β in the values { 5 / 6 , 2 / 3 , 1 / 2 , 1 / 4 } on a validation set.
Revised: Regarding the model-speciﬁc parameters, we set ξ 0 to 0 for both datasets and searched β in the values { 5 / 6 , 2 / 3 , 1 / 2 , 1 / 4 } on a validation set.

Original: We used the ﬁrst 5000 images in the original test set as the validation set, and the others as the test set.
Revised: We used the ﬁrst 5000 images in the original test set as the validation set, and the others as the test set.

Original: We found that β
Revised: We found that β

Original: = 2 / 3 performs relatively well for both data sets.
Revised: = 2 / 3 performs relatively well for both data sets.

Original: Optimization parameters, including learning rate, batch size, and number of epochs, were selected to have the best validation performance of the classiﬁer on one independent task.
Revised: Optimization parameters, including learning rate, batch size, and number of epochs, were selected to have the best validation performance of the classiﬁer on one independent task.

Original: To estimate the change variable s t ’s variational parameter, we approximated the conditional ELBOs 3 by averaging 10000 Monte Carlo samples.
Revised: To estimate the change variable s t ’s variational parameter, we approximated the conditional ELBOs 3 by averaging 10000 Monte Carlo samples.

Original: As outlined in the main paper, we initialized each algorithm by training the model on the full, untransformed dataset.
Revised: As outlined in the main paper, we initialized each algorithm by training the model on the full, untransformed dataset.

Original: The model weights used a standard Gaussian distribution as the prior for this meta-initialization step.
Revised: The model weights used a standard Gaussian distribution as the prior for this meta-initialization step.

Original: When optimizing with variational inference, we initialized q ( z t ) to be a point mass around zero for stability.
Revised: When optimizing with variational inference, we initialized q ( z t ) to be a point mass around zero for stability.

Original: When performing non-Bayesian optimization, we initialized the weights using Glorot
Revised: When performing non-Bayesian optimization, we initialized the weights using Glorot

Original: Uniform initializer (Glorot and Bengio, 2010).
Revised: Uniform initializer (Glorot and Bengio, 2010).

Original: All bias terms were initialized to be zero.
Revised: All bias terms were initialized to be zero.

Original: We performed both the Bayesian and non-Bayesian optimization using ADAM (Kingma and
Revised: We performed both the Bayesian and non-Bayesian optimization using ADAM (Kingma and

Original: Ba, 2014).
Revised: Ba, 2014).

Original: For additional parameters of the ADAM optimizer, we set β 1 = 0 .
Revised: For additional parameters of the ADAM optimizer, we set β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 forboth data sets.
Revised: 999 for

Original: For the deep Bayesian models speciﬁcally, which include VCL and VBS, we used

Original: [Figure]
Revised: [Figure]

Original: LnsLGLousDUtLILcemechDnLcDlIUDuGcoSyLngLmLtDtLveGeGuctLveSeUveUsLonGeceStLvecombLnDtLon
Revised: LnsLGLousDUtLILcemechDnLcDlIUDuGcoSyLngLmLtDtLveGeGuctLveSeUveUsLonGeceStLvecombLnDtLon

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  stochastic black box variational inference (Ranganath et al., 2014; Kingma and Welling, 2013; Zhang et al., 2018).
Revised: both data sets. For the deep Bayesian models speciﬁcally, which include VCL and VBS, we used stochastic black box variational inference (Ranganath et al., 2014; Kingma and Welling, 2013; Zhang et al., 2018).

Original: We also used the Flipout estimator (Wen et al., 2018) to reduce variance in the gradient estimator.
Revised: We also used the Flipout estimator (Wen et al., 2018) to reduce variance in the gradient estimator.

Original: Predictive Distributions
Revised: Predictive Distributions

Original: We evaluated the predictive posterior distribution of the test set by the following approximation:
Revised: We evaluated the predictive posterior distribution of the test set by the following approximation:

Original: [Equation]
Revised: [Equation]

Original: where K is the beam size, w k is the normalized weight of the k th hypothesis after truncation, and
Revised: where K is the beam size, w k is the normalized weight of the k th hypothesis after truncation, and

Original: S is the number of Monte Carlo samples from the variational posterior distribution q t ( z t ) .
Revised: S is the number of Monte Carlo samples from the variational posterior distribution q t ( z t ) .

Original: In our experiments we found S = 10 to be sufﬁcient.
Revised: In our experiments we found S = 10 to be sufﬁcient.

Original: We take arg max y t p ( y t | x t , D 1: t ) to be the predicted class.
Revised: We take arg max y t p ( y t | x t , D 1: t ) to be the predicted class.

Original: VGS and VCL set K = 1 for the single hypothesis in the above formula.
Revised: VGS and VCL set K = 1 for the single hypothesis in the above formula.

Original: LP further only used the MAP estimation z ∗ t to predict the test set: p ( y t | x t , D 1: t ) ≈ p ( y t | x t , z ∗ t ) .
Revised: LP further only used the MAP estimation z ∗ t to predict the test set: p ( y t | x t , D 1: t ) ≈ p ( y t | x t , z ∗ t ) .

Original: C.3. Unsupervised Experiments
Revised: C.3. Unsupervised Experiments

Original: Our third experiment focused on unsupervised learning, where our focus was different: instead of showing performance improvements in terms of higher accuracy, we demonstrated that VBS and
Revised: Our third experiment focused on unsupervised learning, where our focus was different: instead of showing performance improvements in terms of higher accuracy, we demonstrated that VBS and

Original: VGS help uncover interpretable latent structure in high-dimensional time series, such as localizing changes in the meanings of individual words over large time spans.
Revised: VGS help uncover interpretable latent structure in high-dimensional time series, such as localizing changes in the meanings of individual words over large time spans.

Original: We also show that these word embedding trajectories can be used for document dating.
Revised: We also show that these word embedding trajectories can be used for document dating.

Original: Datasets
Revised: Datasets

Original: We analyzed three large time-stamped text corpora, all of which are available online.
Revised: We analyzed three large time-stamped text corpora, all of which are available online.

Original: Our ﬁrst dataset is the Google Books corpus (Michel et al., 2011) consisting of n -grams, which is sufﬁcient for learning word embeddings.
Revised: Our ﬁrst dataset is the Google Books corpus (Michel et al., 2011) consisting of n -grams, which is sufﬁcient for learning word embeddings.

Original: We focused on the period from 1900 to 2000.
Revised: We focused on the period from 1900 to 2000.

Original: To have an approximately even amount of data per year, we sub-sampled 250M to 300M tokens per year.
Revised: To have an approximately even amount of data per year, we sub-sampled 250M to 300M tokens per year.

Original: Second, we used the Congressional Records data set (Gentzkow et al., 2018), which has 13M to 52M tokens per two-year period from 1875 to 2011.
Revised: Second, we used the Congressional Records data set (Gentzkow et al., 2018), which has 13M to 52M tokens per two-year period from 1875 to 2011.

Original: Third, we used the UN General Debates corpus (Jankin Mikhaylov et al., 2017), which has about 250k to 450k tokens per year from 1970 to 2018.
Revised: Third, we used the UN General Debates corpus (Jankin Mikhaylov et al., 2017), which has about 250k to 450k tokens per year from 1970 to 2018.

Original: The vocabulary size was 30000 for all three corpora.
Revised: The vocabulary size was 30000 for all three corpora.

Original: We further randomly split the corpus of every time step into training set (90%) and heldout test set (10%).
Revised: We further randomly split the corpus of every time step into training set (90%) and heldout test set (10%).

Original: Model and Baselines
Revised: Model and Baselines

Original: To analyze the semantic changes of individual words over time, we used
Revised: To analyze the semantic changes of individual words over time, we used

Original: Dynamic Word Embeddings (Bamler and Mandt, 2017) as our base model in all experiments.
Revised: Dynamic Word Embeddings (Bamler and Mandt, 2017) as our base model in all experiments.

Original: The model relies on a probabilistic interpretation of Word2Vec and learns smooth word embedding trajectories by imposing a time-series prior on the embeddings.
Revised: The model relies on a probabilistic interpretation of Word2Vec and learns smooth word embedding trajectories by imposing a time-series prior on the embeddings.

Original: We compared against the online version of dynamic word embeddings with a diffusion prior, which is equivalent to VCL.
Revised: We compared against the online version of dynamic word embeddings with a diffusion prior, which is equivalent to VCL.

Original: By construction, this prior does not encourage temporal sparsity, meaning the word embeddings change every year.
Revised: By construction, this prior does not encourage temporal sparsity, meaning the word embeddings change every year.

Original: For our proposed approach, we imposed our spike and slab novelty prior (Eq. 2) that induces temporal sparsity, allowing us to time-localize semantic changes of words.
Revised: For our proposed approach, we imposed our spike and slab novelty prior (Eq. 2) that induces temporal sparsity, allowing us to time-localize semantic changes of words.

Original: We set the relative broadening constant β = 0 .
Revised: We set the relative broadening constant β = 0 .

Original: 5 for Google books and Congressional records and let β = 0 .
Revised: 5 for Google books and Congressional records and let β = 0 .

Original: 25 for UN debates.
Revised: 25 for UN debates.

Original: We combined this prior with beam search ( K = 2 ), where we reported results on the most likely beam.
Revised: We combined this prior with beam search ( K = 2 ), where we reported results on the most likely beam.

Original: We also considered regular word embeddings trained on individual years (Mikolov et al., 2013).
Revised: We also considered regular word embeddings trained on individual years (Mikolov et al., 2013).

Original: While (Bamler and Mandt, 2017) already carried-out the comparison to dynamic word embeddings, we used them to measure the compressibility of word embedding trajectories for document dating tasks, as we describe below.
Revised: While (Bamler and Mandt, 2017) already carried-out the comparison to dynamic word embeddings, we used them to measure the compressibility of word embedding trajectories for document dating tasks, as we describe below.

Original: Qualitative Results
Revised: Qualitative Results

Original: Our ﬁrst experiments demonstrate that a spike & slab prior is more interpretable and results in more meaningful word semantics than the diffusion prior of (Bamler and
Revised: Our ﬁrst experiments demonstrate that a spike & slab prior is more interpretable and results in more meaningful word semantics than the diffusion prior of (Bamler and

Original: Mandt, 2017).
Revised: Mandt, 2017).

Original: Figure 5 shows three selected words ("simulation", "atom", and "race"–one taken from each corpus) and their nearest neighbors in latent space.
Revised: Figure 5 shows three selected words ("simulation", "atom", and "race"–one taken from each corpus) and their nearest neighbors in latent space.

Original: As time progresses the nearest neighboring words change, reﬂecting a semantic change of the words.
Revised: As time progresses the nearest neighboring words change, reﬂecting a semantic change of the words.

Original: While the horizontal axis shows the year, the vertical axis shows the cosine distance of the word’s embedding vector at the given year to its embedding vector in the last available year.
Revised: While the horizontal axis shows the year, the vertical axis shows the cosine distance of the word’s embedding vector at the given year to its embedding vector in the last available year.

Original: The plot reveals several interpretable semantic changes of each word.
Revised: The plot reveals several interpretable semantic changes of each word.

Original: For example, “atom” changes its meaning from “element” to “nuclear” in 1945–the year when two nuclear bombs were detonated.
Revised: For example, “atom” changes its meaning from “element” to “nuclear” in 1945–the year when two nuclear bombs were detonated.

Original: The word “race” changes from the cold-war era “arms”(-race) to its more prevalent meaning after 1991 when the cold war ended.
Revised: The word “race” changes from the cold-war era “arms”(-race) to its more prevalent meaning after 1991 when the cold war ended.

Original: The word "simulation" changes its dominant context from "deception" to "programming" with the advent of computers.
Revised: The word "simulation" changes its dominant context from "deception" to "programming" with the advent of computers.

Original: The plot also compares the nearest  neighbors obtained with the diffusion prior (Bamler and Mandt, 2017) in grey to our VBS-based approach in blue, with arguably more interpretable results.
Revised: The plot also compares the nearest [Figure] neighbors obtained with the diffusion prior (Bamler and Mandt, 2017) in grey to our VBS-based approach in blue, with arguably more interpretable results.

Original: Quantitative Results.
Revised: Quantitative Results.

Original: Another beneﬁt of dynamic sparsity, as induced by VBS, is a more compressible representation of the revealed latent states.
Revised: Another beneﬁt of dynamic sparsity, as induced by VBS, is a more compressible representation of the revealed latent states.

Original: For dynamic word embeddings, word embedding[Figure] [Table] trajectories can be stored as a list of embedding vectors with associated time stamps–one vector per update.
Revised: For dynamic word embeddings, word embedding trajectories can be stored as a list of embedding vectors with associated time stamps–one vector per update.

Original: In contrast, the work of (Bamler and Mandt, 2017) or (Mikolov et al., 2013) would require a separate embedding vector for every word for all years [Figure]
Revised: In contrast, the work of (Bamler and Mandt, 2017) or (Mikolov et al., 2013) would require a separate embedding vector for every word for all years.

Original: We analyzed a rate-distortion tradeoff in the classical compression sense, where we tuned the prior on the change variable s t (the prior log-odds ξ 0 in Eq. 4) to suppress semantic changes.
Revised: We analyzed a rate-distortion tradeoff in the classical compression sense, where we tuned the prior on the change variable s t (the prior log-odds ξ 0 in Eq. 4) to suppress semantic changes.

Original: With changes being suppressed, naturally the word embeddings get worse in quality, but lead to smaller ﬁle sizes as only updated word embeddings have to be stored.
Revised: With changes being suppressed, naturally the word embeddings get worse in quality, but lead to smaller ﬁle sizes as only updated word embeddings have to be stored.

Original: Figure 6 shows the results on
Revised: Figure 6 shows the results on

Original: Congressional Records data.
Revised: Congressional Records data.

Original: To measure model performance, we predicted the year in which each held-out document’s word-word co-occurrence statistics have the highest likelihood and measured
Revised: To measure model performance, we predicted the year in which each held-out document’s word-word co-occurrence statistics have the highest likelihood and measured

Original: L 1 error.
Revised: L 1 error.

Original: As a simple baseline, we assumed that we had separate word embeddings associated with episodes of L consecutive years.
Revised: As a simple baseline, we assumed that we had separate word embeddings associated with episodes of L consecutive years.

Original: For T years in total, the associated memory requirements would be proportional to V ∗ T/L , where V is the vocabulary size.
Revised: For T years in total, the associated memory requirements would be proportional to V ∗ T/L , where V is the vocabulary size.

Original: Assuming we could perfectly date the document up to L years results in the "binning" baseline in Figure 6.
Revised: Assuming we could perfectly date the document up to L years results in the "binning" baseline in Figure 6.

Original: We see that our approach results in a much better rate-distortion tradeoff since we are allowing each word to change at different times, as opposed to only collectively.
Revised: We see that our approach results in a much better rate-distortion tradeoff since we are allowing each word to change at different times, as opposed to only collectively.

Original: C.4. More on Dynamic Word Embeddings Experiments Model Assumptions
Revised: C.4. More on Dynamic Word Embeddings Experiments Model Assumptions

Original: As outlined in the main paper, we analyzed the semantic changes of individual words over time.
Revised: As outlined in the main paper, we analyzed the semantic changes of individual words over time.

Original: We augmented the probabilistic models proposed by Bamler and Mandt (2017) with the spike and slab novelty prior (Eq. 2 in the main paper) to encourage temporal sparsity.
Revised: We augmented the probabilistic models proposed by Bamler and Mandt (2017) with the spike and slab novelty prior (Eq. 2 in the main paper) to encourage temporal sparsity.

Original: We pre-trained the context word embeddings 4 using the whole corpus, and kept them constant when updating the target word embeddings.
Revised: We pre-trained the context word embeddings 4 using the whole corpus, and kept them constant when updating the target word embeddings.

Original: This practice denied possible interference on one target word embedding from the updates of the others.
Revised: This practice denied possible interference on one target word embedding from the updates of the others.

Original: If we did not employ this practice, the spike and  slab prior on word i would lead to two branches of the “remaining vocabulary” (embeddings of the remaining words in the vocabulary), conditioned either on the spike prior of word i or on the slab prior.
Revised: If we did not employ this practice, the spike and [Table] slab prior on word i would lead to two branches of the “remaining vocabulary” (embeddings of the remaining words in the vocabulary), conditioned either on the spike prior of word i or on the slab prior.

Original: This hypothetical situation gets severe when every word in the vocabulary can take two different priors, thus leading to exponential branching of the sequences of inferred change points.
Revised: This hypothetical situation gets severe when every word in the vocabulary can take two different priors, thus leading to exponential branching of the sequences of inferred change points.

Original: When this interference is allowed, the exponential scaling of hypotheses translates into exponential scaling of possible word embeddings for a single target word, which is not feasible to compute for any meaningful vocabulary sizes and number of time steps.
Revised: When this interference is allowed, the exponential scaling of hypotheses translates into exponential scaling of possible word embeddings for a single target word, which is not feasible to compute for any meaningful vocabulary sizes and number of time steps.

Original: To this end, while using a ﬁxed, pretrained context word embeddings induces a slight drop of predictive performance, the computational efﬁciency improves tremendously and the model can actually be learned.
Revised: To this end, while using a ﬁxed, pretrained context word embeddings induces a slight drop of predictive performance, the computational efﬁciency improves tremendously and the model can actually be learned.

Original: Hyperparameters and Optimization
Revised: Hyperparameters and Optimization

Original: Qualitative results in Figure 5 in the main paper were generated using the hyperparameters in Table 5.
Revised: Qualitative results in Figure 5 in the main paper were generated using the hyperparameters in Table 5.

Original: The initial prior distribution used for all latent embedding dimensions was a standard Gaussian distribution.
Revised: The initial prior distribution used for all latent embedding dimensions was a standard Gaussian distribution.

Original: We also initialized all variational distributions with standard Gaussian distributions.
Revised: We also initialized all variational distributions with standard Gaussian distributions.

Original: For model-speciﬁc hyperparameters β and ξ 0 , we ﬁrst searched the broadening constant β to have the desired jump magnitude observed from the semantic trajectories mainly for medium-frequency words.
Revised: For model-speciﬁc hyperparameters β and ξ 0 , we ﬁrst searched the broadening constant β to have the desired jump magnitude observed from the semantic trajectories mainly for medium-frequency words.

Original: We then tuned the bias term ξ 0 to have the desired change frequencies in general.
Revised: We then tuned the bias term ξ 0 to have the desired change frequencies in general.

Original: We did the searching for the ﬁrst several time steps.
Revised: We did the searching for the ﬁrst several time steps.

Original: We performed the optimization using black box variational inference and ADAM.
Revised: We performed the optimization using black box variational inference and ADAM.

Original: For additional parameters of ADAM optimizer, we set β 1 = 0 .
Revised: For additional parameters of ADAM optimizer, we set β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 for all three corpora.
Revised: 999 for all three corpora.

Original: In this case, we did not temper the conditional ELBO by the number of observations.
Revised: In this case, we did not temper the conditional ELBO by the number of observations.

Original: Quantitative results of VGS in Figure 6 in the main paper were generated by setting a smaller vocabulary size and embedding dimension, 10,000 and 20, respectively for all three corpora.
Revised: Quantitative results of VGS in Figure 6 in the main paper were generated by setting a smaller vocabulary size and embedding dimension, 10,000 and 20, respectively for all three corpora.

Original: Other hyperparameters were inherited from the qualitative experiments.
Revised: Other hyperparameters were inherited from the qualitative experiments.

Original: On the other hand, the baseline, “binning”, and had closed-form performance if we assume (i) a uniformly distributed year in which a document query is generated, (ii) “binning” perfectly locates the ground truth episode, and (iii) the dating result is uniformed distributed within the ground truth episode.
Revised: On the other hand, the baseline, “binning”, and had closed-form performance if we assume (i) a uniformly distributed year in which a document query is generated, (ii) “binning” perfectly locates the ground truth episode, and (iii) the dating result is uniformed distributed within the ground truth episode.

Original: The L 1 error associated with “binning” with episode length L is E t ∼U (1 ,L ) ,t (cid:48)
Revised: The L 1 error associated with “binning” with episode length L is E t ∼U (1 ,L ) ,t (cid:48)

Original: ∼U (1 ,L ) [ | t − t (cid:48)
Revised: ∼U (1 ,L ) [ | t − t (cid:48)

Original: | ]
Revised: | ]

Original: =
Revised: =

Original: L − 1 2 .
Revised: L − 1 2 .

Original: By varying L , we get binning’s rate-distortion curve in Figure 6 in the main paper.
Revised: By varying L , we get binning’s rate-distortion curve in Figure 6 in the main paper.

Original: Additional Results
Revised: Additional Results

Original: Additional qualitative results can be found in Figure 7.
Revised: Additional qualitative results can be found in Figure 7.

Original: it, again, reveals interpretable semantic changes of each word: the ﬁrst change of “computer” happens in 1940s–when modern computers appeared; “broadcast” adopts its major change shortly after the ﬁrst commercial radio stations were established; “climate” changes its meaning at the time when Intergovernmental
Revised: it, again, reveals interpretable semantic changes of each word: the ﬁrst change of “computer” happens in 1940s–when modern computers appeared; “broadcast” adopts its major change shortly after the ﬁrst commercial radio stations were established; “climate” changes its meaning at the time when Intergovernmental

Original: Panel on Climate Change (IPCC) was set up, and when it released the assessment reports to address the implications and potential risks of climate changes.
Revised: Panel on Climate Change (IPCC) was set up, and when it released the assessment reports to address the implications and potential risks of climate changes.

Original: Predictive Distributions
Revised: Predictive Distributions

Original: In the demonstration of the quantitative results, i.e., the document dating experiments, we predicted the year in which each held-out document’s word-word co-occurrence statistics x have the highest likelihood and measured L 1 error.
Revised: In the demonstration of the quantitative results, i.e., the document dating experiments, we predicted the year in which each held-out document’s word-word co-occurrence statistics x have the highest likelihood and measured L 1 error.

Original: To be speciﬁc, for a given document in year t , we approximated its likelihood under year t (cid:48) by evaluating 1 | V | log p ( x t | z ∗ t (cid:48) ) , where z ∗ t (cid:48) is the mode embedding in year t (cid:48) and | V | is the vocabulary size.
Revised: To be speciﬁc, for a given document in year t , we approximated its likelihood under year t (cid:48) by evaluating 1 | V | log p ( x t | z ∗ t (cid:48) ) , where z ∗ t (cid:48) is the mode embedding in year t (cid:48) and | V | is the vocabulary size.

Original: We predicted the year t ∗ = arg max t (cid:48) 1 | V | log p ( x t | z ∗ t (cid:48) ) .
Revised: We predicted the year t ∗ = arg max t (cid:48) 1 | V | log p ( x t | z ∗ t (cid:48) ) .

Original: We then measured the L 1 error by 1 T (cid:80)
Revised: We then measured the L 1 error by 1 T (cid:80)

Original: Ti | t i − t ∗ i | given T truth-prediction pairs.
Revised: Ti | t i − t ∗ i | given T truth-prediction pairs.