Original: C ONSTRAINING L INEAR - CHAIN CRF S TO R EGULAR L ANGUAGES
Revised: C ONSTRAINING L INEAR - CHAIN CRF S TO R EGULAR L ANGUAGES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A major challenge in structured prediction is to represent the interdependencies within output structures.
Revised: A major challenge in structured prediction is to represent the interdependencies within output structures.

Original: When outputs are structured as sequences, linear-chain conditional random ﬁelds (CRFs) are a widely used model class which can learn local dependencies in the output.
Revised: When outputs are structured as sequences, linear-chain conditional random ﬁelds (CRFs) are a widely used model class which can learn local dependencies in the output.

Original: However, the CRF’s Markov assumption makes it impossible for CRFs to represent distributions with nonlocal dependencies, and standard CRFs are unable to respect nonlocal constraints of the data (such as global arity constraints on output labels).
Revised: However, the CRF’s Markov assumption makes it impossible for CRFs to represent distributions with nonlocal dependencies, and standard CRFs are unable to respect nonlocal constraints of the data (such as global arity constraints on output labels).

Original: We present a generalization of CRFs that can enforce a broad class of constraints, including nonlocal ones, by specifying the space of possible output structures as a regular language L .
Revised: We present a generalization of CRFs that can enforce a broad class of constraints, including nonlocal ones, by specifying the space of possible output structures as a regular language L .

Original: The resulting regular-constrained CRF (RegCCRF) has the same formal properties as a standard
Revised: The resulting regular-constrained CRF (RegCCRF) has the same formal properties as a standard

Original: CRF, but assigns zero probability to all label sequences not in L .
Revised: CRF, but assigns zero probability to all label sequences not in L .

Original: Notably, RegCCRFs can incorporate their constraints during training, while related models only enforce constraints during decoding.
Revised: Notably, RegCCRFs can incorporate their constraints during training, while related models only enforce constraints during decoding.

Original: We prove that constrained training is never worse than constrained decoding, and show empirically that it can be substantially better in practice.
Revised: We prove that constrained training is never worse than constrained decoding, and show empirically that it can be substantially better in practice.

Original: Additionally, we demonstrate a practical beneﬁt on downstream tasks by incorporating a RegCCRF into a deep neural model for semantic role labeling, exceeding state-of-the-art results on a standard dataset.
Revised: Finally, we demonstrate a practical beneﬁt on downstream tasks by incorporating a RegCCRF into a deep neural model for semantic role labeling, exceeding state-of-the-art results on a standard dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Structured prediction is a ﬁeld of machine learning where outputs are expected to obey some predeﬁned discrete structure.
Revised: Structured prediction is a ﬁeld of machine learning where outputs are expected to obey some predeﬁned discrete structure.

Original: Instances of structured prediction with various output structures occur in many applications, including computer vision (e.g., scene graph generation (Johnson et al., 2015) with graph-structured output), natural language processing (e.g., linguistic parsing (Niculae et al., 2018) with tree-structured output, relation extraction (Roth & Yih, 2004) with tuple-structured output) or modeling the spatial structure of physical entities and processes (Jiang, 2020).
Revised: Instances of structured prediction with various output structures occur in many applications, including computer vision (e.g., scene graph generation (Johnson et al., 2015) with graph-structured output), natural language processing (e.g., linguistic parsing (Niculae et al., 2018) with tree-structured output, relation extraction (Roth & Yih, 2004) with tuple-structured output) or modeling the spatial structure of physical entities and processes (Jiang, 2020).

Original: A key difﬁculty faced by all models is to tractably model interdependencies between different parts of the output.
Revised: A key difﬁculty faced by all models is to tractably model interdependencies between different parts of the output.

Original: As output spaces tend to be combinatorially large, special techniques, approximations, and independence assumptions must be used to work with the associated probability distributions.
Revised: As output spaces tend to be combinatorially large, special techniques, approximations, and independence assumptions must be used to work with the associated probability distributions.

Original: Considerable research has investigated speciﬁc structures for which such approaches make machine learning tractable.
Revised: Considerable research has investigated speciﬁc structures for which such approaches make machine learning tractable.

Original: For instance, when outputs are trees over a ﬁxed set of nodes, maximal arborescence algorithms allow for exact inference (Chu, 1965; Edmonds, 1967); when outputs are graphstructured, loopy belief propagation can provide approximate inference (Murphy et al., 1999).
Revised: For instance, when outputs are trees over a ﬁxed set of nodes, maximal arborescence algorithms allow for exact inference (Chu, 1965; Edmonds, 1967); when outputs are graphstructured, loopy belief propagation can provide approximate inference (Murphy et al., 1999).

Original: If the output forms a linear sequence, a Markov assumption is often made: model outputs depend only on their immediate neighbors, but not (directly) on more distant ones.
Revised: If the output forms a linear sequence, a Markov assumption is often made: model outputs depend only on their immediate neighbors, but not (directly) on more distant ones.

Original: A common model that uses this assumption is the linear-chain conditional random ﬁeld (CRF) (Lafferty et al., 2001), which has found ubiquitous use for sequence labeling tasks, including part-of-speech tagging (Gimpel et al., 2011) and named entity recognition (Lample et al., 2016).
Revised: A common model that uses this assumption is the linear-chain conditional random ﬁeld (CRF) (Lafferty et al., 2001), which has found ubiquitous use for sequence labeling tasks, including part-of-speech tagging (Gimpel et al., 2011) and named entity recognition (Lample et al., 2016).

Original: This model became popular with the use of hand-crafted feature vectors in the 2000s, and is nowadays commonly used as an output layer in neural networks to encourage the learning of structural properties of the output sequence.
Revised: This model became popular with the use of hand-crafted feature vectors in the 2000s, and is nowadays commonly used as an output layer in neural networks to encourage the learning of structural properties of the output sequence.

Original: The Markov assumption makes training tractable, but also limits the CRF’s expressive power, which can hamper performance, especially for long sequences (Scheible et al., 2016).
Revised: The Markov assumption makes training tractable, but also limits the CRF’s expressive power, which can hamper performance, especially for long sequences (Scheible et al., 2016).

Original: Semi-Markov CRFs (Sarawagi & Cohen, 2004) and skip-chain CRFs (Sutton & McCallum, 2004) are techniques for relaxing the Markov assumption, but both come with drawbacks in performance and expressiveness.
Revised: Semi-Markov CRFs (Sarawagi & Cohen, 2004) and skip-chain CRFs (Sutton & McCallum, 2004) are techniques for relaxing the Markov assumption, but both come with drawbacks in performance and expressiveness.

Original: In this work, we propose a new method to tractably relax the Markov assumption in CRFs.
Revised: In this work, we propose a new method to tractably relax the Markov assumption in CRFs.

Original: Specifically, we show how to constrain the output of a CRF to a regular language , such that the resulting regular-constrained CRF (RegCCRF) is guaranteed to output label sequences from that language.
Revised: Specifically, we show how to constrain the output of a CRF to a regular language , such that the resulting regular-constrained CRF (RegCCRF) is guaranteed to output label sequences from that language.

Original: Since regular languages can encode long-distance dependencies between the symbols in their strings,
Revised: Since regular languages can encode long-distance dependencies between the symbols in their strings,

Original: RegCCRFs provide a simple model for structured prediction guaranteed to respect these constraints.
Revised: RegCCRFs provide a simple model for structured prediction guaranteed to respect these constraints.

Original: The domain knowledge specifying these constraints is deﬁned via regular expressions, a straightforward, well understood formalism.
Revised: The domain knowledge specifying these constraints is deﬁned via regular expressions, a straightforward, well understood formalism.

Original: We show that our method is distinct from approaches that enforce constraints at decoding time, and that it better approximates the true data distribution.
Revised: We show that our method is distinct from approaches that enforce constraints at decoding time, and that it better approximates the true data distribution.

Original: We evaluate our model empirically as the output layer of a neural network and attain state-of-the-art performance for semantic role labeling (Weischedel et al., 2011; Pradhan et al., 2012).
Revised: We evaluate our model empirically as the output layer of a neural network and attain state-of-the-art performance for semantic role labeling (Weischedel et al., 2011; Pradhan et al., 2012).

Original: Our PyTorch RegCCRF implementation can be used as a drop-in replacement for standard CRFs.
Revised: Our PyTorch RegCCRF implementation can be used as a drop-in replacement for standard CRFs.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: We identify three areas of structured prediction that are relevant to our work: constrained decoding, which can enforce output constraints at decoding time, techniques for weakening the Markov assumption of CRFs to learn long-distance dependencies, and weight-learning in ﬁnite-state transducers.
Revised: We identify three areas of structured prediction that are relevant to our work: constrained decoding, which can enforce output constraints at decoding time, techniques for weakening the Markov assumption of CRFs to learn long-distance dependencies, and weight-learning in ﬁnite-state transducers.

Original: Constrained decoding.
Revised: Constrained decoding.

Original: A common approach to enforcing constraints in model output is constrained decoding : Models are trained in a standard fashion, and decoding ensures that the model output satisﬁes the constraints.
Revised: A common approach to enforcing constraints in model output is constrained decoding : Models are trained in a standard fashion, and decoding ensures that the model output satisﬁes the constraints.

Original: This almost always corresponds to ﬁnding or approximating a version of the model’s distribution conditionalized on the output obeying the speciﬁed constraints.
Revised: This almost always corresponds to ﬁnding or approximating a version of the model’s distribution conditionalized on the output obeying the speciﬁed constraints.

Original: This approach is useful if constraints are not available at training time, such as in the interactive information extraction task of Kristjansson et al.
Revised: This approach is useful if constraints are not available at training time, such as in the interactive information extraction task of Kristjansson et al.

Original: They present constrained conditional random ﬁelds , which can enforce that particular tokens are or are not assigned particular labels (positive and negative constraints, respectively).
Revised: They present constrained conditional random ﬁelds , which can enforce that particular tokens are or are not assigned particular labels (positive and negative constraints, respectively).

Original: Formally, our work is a strict generalization of this approach, as position-wise constraints can be formulated as a regular language, but regular languages go beyond position-wise constraints.
Revised: Formally, our work is a strict generalization of this approach, as position-wise constraints can be formulated as a regular language, but regular languages go beyond position-wise constraints.

Original: Other studies treat decoding with constraints as a search problem, searching for the most probable decoding path which satisﬁes all constraints.
Revised: Other studies treat decoding with constraints as a search problem, searching for the most probable decoding path which satisﬁes all constraints.

Original: For example, He et al.
Revised: For example, He et al.

Original: (2017) train a neural network to predict token-wise output probabilities for semantic role labeling following the BIO label-alphabet (Ramshaw & Marcus, 1999), and then use exact A* search to ensure that the output forms a valid BIO sequence and that particular task-speciﬁc constraints are satisﬁed.
Revised: (2017) train a neural network to predict token-wise output probabilities for semantic role labeling following the BIO label-alphabet (Ramshaw & Marcus, 1999), and then use exact A* search to ensure that the output forms a valid BIO sequence and that particular task-speciﬁc constraints are satisﬁed.

Original: For autoregressive models, constrained beam search (Hokamp & Liu, 2017; Anderson et al., 2017; Hasler et al., 2018) can enforce regular-language constraints during search.
Revised: For autoregressive models, constrained beam search (Hokamp & Liu, 2017; Anderson et al., 2017; Hasler et al., 2018) can enforce regular-language constraints during search.

Original: We further discuss constrained decoding as it relates to RegCCRFs in Section 5.
Revised: We further discuss constrained decoding as it relates to RegCCRFs in Section 5.

Original: Markov relaxations.
Revised: Markov relaxations.

Original: While our approach can relax the Markov assumption of CRFs through nonlocal hard constraints, another strand of work has developed models which can directly learn nonlocal dependencies in CRFs: Semi-Markov CRFs (Sarawagi & Cohen, 2004) relax the Markov property to the semi-Markov property.
Revised: While our approach can relax the Markov assumption of CRFs through nonlocal hard constraints, another strand of work has developed models which can directly learn nonlocal dependencies in CRFs: Semi-Markov CRFs (Sarawagi & Cohen, 2004) relax the Markov property to the semi-Markov property.

Original: In this setting, CRFs are tasked with segmentation, where individual segments may depend only on their immediate neighbors, but model behavior within a particular segment need not be Markovian.
Revised: In this setting, CRFs are tasked with segmentation, where individual segments may depend only on their immediate neighbors, but model behavior within a particular segment need not be Markovian.

Original: As such, semi-Markov CRFs are capable of capturing nonlocal dependencies between output variables, but only to a range of one segment and inside of a segment.
Revised: As such, semi-Markov CRFs are capable of capturing nonlocal dependencies between output variables, but only to a range of one segment and inside of a segment.

Original: Skip-chain CRFs (Sutton & McCallum, 2004) change the expressiveness of CRFs by relaxing the assumption that only the linear structure of the input matters: they add explicit dependencies between distant nodes in an otherwise linear-chain CRF.
Revised: Skip-chain CRFs (Sutton & McCallum, 2004) change the expressiveness of CRFs by relaxing the assumption that only the linear structure of the input matters: they add explicit dependencies between distant nodes in an otherwise linear-chain CRF.

Original: These dependencies are picked based on particular properties, e.g., input variables of the same value or which share other properties.
Revised: These dependencies are picked based on particular properties, e.g., input variables of the same value or which share other properties.

Original: In doing so, they add loops to the model’s factor graph, which makes exact training and inference intractable, and leads to the use of approximation techniques such as loopy belief propagation and Gibbs sampling.
Revised: In doing so, they add loops to the model’s factor graph, which makes exact training and inference intractable, and leads to the use of approximation techniques such as loopy belief propagation and Gibbs sampling.

Original: Weight learning for ﬁnite-state transducers.
Revised: Weight learning for ﬁnite-state transducers.

Original: While our approach focuses on the task of constraining the CRF distribution to a known regular language, a related task is that of learning a weighted regular language from data.
Revised: While our approach focuses on the task of constraining the CRF distribution to a known regular language, a related task is that of learning a weighted regular language from data.

Original: This task is usually formalized as learning the weights of a weighted ﬁnite-state transducer (FST), as in e.g. Eisner (2002) with directly parameterized weights and Rastogi et al.
Revised: This task is usually formalized as learning the weights of a weighted ﬁnite-state transducer (FST), as in e.g. Eisner (2002) with directly parameterized weights and Rastogi et al.

Original: (2016) with weights parameterized by a neural network.
Revised: (2016) with weights parameterized by a neural network.

Original: Despite the difference in task-setting, this task is quite similar to ours in the formal sense, and in fact our proposal can be viewed as a particularly well-behaved special case of FST weight learning for an appropriately chosen transducer architecture and parameterization.
Revised: Despite the difference in task-setting, this task is quite similar to ours in the formal sense, and in fact our proposal can be viewed as a particularly well-behaved special case of FST weight learning for an appropriately chosen transducer architecture and parameterization.

Original: We discuss this connection further in Section 4.3.
Revised: We discuss this connection further in Section 4.3.

Original:  P RELIMINARIES AND NOTATION
Revised:  P RELIMINARIES AND NOTATION

Original: As our construction involves ﬁnite-state automata and conditional random ﬁelds, we deﬁne these here and specify the notation we will use in the remainder of this work.
Revised: As our construction involves ﬁnite-state automata and conditional random ﬁelds, we deﬁne these here and specify the notation we will use in the remainder of this work.

Original: Finite-state automata.
Revised: Finite-state automata.

Original: All automata are taken to be nondeterministic ﬁnite-state automata (NFAs) without epsilon transitions.
Revised: All automata are taken to be nondeterministic ﬁnite-state automata (NFAs) without epsilon transitions.

Original: Let such an NFA be deﬁned as a 4-tuple (Σ , Q,  F, E ) , where Σ = { a 1 , a 2 , ..., a | Σ | } is a ﬁnite alphabet of symbols, Q = { q 1 , q 2 , ..., q | Q | } is a ﬁnite set of states. By convention, q 1  is the sole starting state. F ⊆ Q is a set of accepting states. E ⊆ Q × Σ × Q is a set of directed, symbol-labeled edges between states.
Revised: Let such an NFA be deﬁned as a 5-tuple (Σ , Q, q 1 , F, E ) , where Σ = { a 1 , a 2 , ..., a | Σ | } is a ﬁnite alphabet of symbols, Q = { q 1 , q 2 , ..., q | Q | } is a ﬁnite set of states, q 1 ∈ Q is the sole starting state, F ⊆ Q is a set of accepting states, and E ⊆ Q × Σ × Q is a set of directed, symbol-labeled edges between states.

Original: These edges deﬁne the NFA’s transition function ∆ : Q × Σ → 2 Q , with r ∈ ∆( q, a ) ↔ ( q, a, r ) ∈ E .
Revised: The edges deﬁne the NFA’s transition function ∆ : Q × Σ → 2 Q , with r ∈ ∆( q, a ) ↔ ( q, a, r ) ∈ E .

Original: An automaton is said to accept a string s ∈ Σ ∗ if there exists a contiguous path of edges from q 1 to some accepting state whose edge labels are exactly the symbols of s .
Revised: An automaton is said to accept a string s ∈ Σ ∗ iff there exists a contiguous path of edges from q 1 to some accepting state whose edge labels are exactly the symbols of s .

Original: The language deﬁned by an automaton is the set of all such accepted strings.
Revised: The language deﬁned by an automaton is the set of all such accepted strings.

Original: A language is regular if and only if it is the language of some NFA.
Revised: A language is regular if and only if it is the language of some NFA.

Original: Linear-chain conditional random ﬁelds.
Revised: Linear-chain conditional random ﬁelds.

Original: Linear-chain conditional random ﬁelds (CRFs) Lafferty et al.
Revised: Linear-chain conditional random ﬁelds (CRFs) Lafferty et al.

Original: (2001) are a sequence labeling model.
Revised: (2001) are a sequence labeling model.

Original: Parameterized by θ , they use a global exponential model to represent the conditional distribution over label sequences y =
Revised: Parameterized by θ , they use a global exponential model to represent the conditional distribution over label sequences y =

Original: (cid:104)
Revised: (cid:104)

Original: y 1 , y 2 , ..., y t (cid:105) conditioned on input sequences x
Revised: y 1 , y 2 , ..., y t (cid:105) conditioned on input sequences x

Original: = (cid:104) x 1 , x 2 , ..., x t (cid:105) :
Revised: = (cid:104) x 1 , x 2 , ..., x t (cid:105) :

Original: P θ ( y | x ) ∝ exp (cid:80)
Revised: P θ ( y | x ) ∝ exp (cid:80)

Original: j f jθ ( x , y ) , with individual observations x i coming from some observation space X , and outputs y
Revised: j f jθ ( x , y ) , with individual observations x i coming from some observation space X , and outputs y

Original: i coming from some ﬁnite alphabet Y .
Revised: i coming from some ﬁnite alphabet Y .

Original: In this work, we use CRFs for sequence labeling problems, but the dataset labels do not correspond directly to the CRF’s outputs y i .
Revised: In this work, we use CRFs for sequence labeling problems, but the dataset labels do not correspond directly to the CRF’s outputs y i .

Original: In order to avoid ambiguity, and since the term “state” already has a meaning for NFAs, we call y the CRF’s tag sequence , and each y i a tag .
Revised: In order to avoid ambiguity, and since the term “state” already has a meaning for NFAs, we call y the CRF’s tag sequence , and each y i a tag .

Original: The terms label sequence and label will thus unambiguously refer to the original dataset labels.
Revised: The terms label sequence and label will thus unambiguously refer to the original dataset labels.

Original: Each f jθ is a potential function of x and y , parameterized by θ .
Revised: Each f jθ is a potential function of x and y , parameterized by θ .

Original: Importantly, in a linear-chain CRF, these potential functions are limited to two kinds: The transition function g θ ( y i , y i +1 ) assigns a potential to each pair ( y i , y i +1 ) of adjacent tags in y , and the emission function h
Revised: Importantly, in a linear-chain CRF, these potential functions are limited to two kinds: The transition function g θ ( y i , y i +1 ) assigns a potential to each pair ( y i , y i +1 ) of adjacent tags in y , and the emission function h

Original: θ ( y i | x , i ) assigns a potential to each possible output tag y i given the observation sequence x and its position i .
Revised: θ ( y i | x , i ) assigns a potential to each possible output tag y i given the observation sequence x and its position i .

Original: With these, the distribution deﬁned by a CRF is
Revised: With these, the distribution deﬁned by a CRF is

Original: [Equation]
Revised: [Equation]

Original: Limiting our potential functions in this way imposes a Markov assumption on our model, as potential functions can only depend on a single tag or a single pair of adjacent tags.
Revised: Limiting our potential functions in this way imposes a Markov assumption on our model, as potential functions can only depend on a single tag or a single pair of adjacent tags.

Original: This makes learning and inference tractable: the forward algorithm (Jurafsky & Martin, 2009) can calculate negative loglikelihood (NLL) loss during training, and the Viterbi algorithm (Viterbi, 1967; Jurafsky & Martin, 2009) can be used for inference.
Revised: This makes learning and inference tractable: the forward algorithm (Jurafsky & Martin, 2009) can calculate negative loglikelihood (NLL) loss during training, and the Viterbi algorithm (Viterbi, 1967; Jurafsky & Martin, 2009) can be used for inference.

Original: Both are linear in t , and quadratic in | Y | in both time and space.
Revised: Both are linear in t , and quadratic in | Y | in both time and space.

Original:  R EGULAR - CONSTRAINED CRF S
Revised:  R EGULAR - CONSTRAINED CRF S

Original: Given a regular language L , we would like to constrain a CRF to L .
Revised: Given a regular language L , we would like to constrain a CRF to L .

Original: We formalize this notion of constraint with conditional probabilities – a CRF constrained to L is described by a (further) conditionalized version of that CRF’s distribution P θ ( y | x ) , conditioned on the event that the tag sequence y is in L .
Revised: We formalize this notion of constraint with conditional probabilities – a CRF constrained to L is described by a (further) conditionalized version of that CRF’s distribution P θ ( y | x ) , conditioned on the event that the tag sequence y is in L .

Original: We write this distribution as
Revised: We write this distribution as

Original: [Equation]
Revised: [Equation]

Original: with α ≥ 1 deﬁned as α − 1 = (cid:80) y ∈L P θ ( y | x ) .
Revised: with α ≥ 1 deﬁned as α − 1 = (cid:80) y ∈L P θ ( y | x ) .

Original: In order to utilize this distribution for machine learning, we need to be able to compute NLL losses and perform MAP inference.
Revised: In order to utilize this distribution for machine learning, we need to be able to compute NLL losses and perform MAP inference.

Original: As discussed in Section 3, both of these are efﬁciently computable for CRFs.
Revised: As discussed in Section 3, both of these are efﬁciently computable for CRFs.

Original: Thus, if we can construct a separate CRF whose output distribution can be interpreted as
Revised: Thus, if we can construct a separate CRF whose output distribution can be interpreted as

Original: P ( y | x , L ) , both of these operations will be available.
Revised: P ( y | x , L ) , both of these operations will be available.

Original: We do this in the next section.
Revised: We do this in the next section.

Original: [Figure]
Revised: [Figure]

Original: where y (cid:48) ∈ Y (cid:48)∗ corresponds to the sole accepting path of y through M (marked).
Revised: where y (cid:48) ∈ Y (cid:48)∗ corresponds to the sole accepting path of y through M (marked).

Original: 4. C ONSTRUCTION
Revised: 4. C ONSTRUCTION

Original: Let M :=
Revised: Let M :=

Original: (Σ , Q, F, E ) be an NFA that describes L .
Revised: (Σ , Q, F, E ) be an NFA that describes L .

Original: We assume that M is unambiguous – i.e., every string in L is accepted by exactly one path through M .
Revised: We assume that M is unambiguous – i.e., every string in L is accepted by exactly one path through M .

Original: As every NFA can be transformed into an equivalent unambiguous NFA (Mohri, 2012), this assumption involves no loss of generality.
Revised: As every NFA can be transformed into an equivalent unambiguous NFA (Mohri, 2012), this assumption involves no loss of generality.

Original: Our plan is to represent P θ ( y | x , L ) by constructing a separate CRF with a distinct tag set, whose output sequences can be interpreted directly as paths through M .
Revised: Our plan is to represent P θ ( y | x , L ) by constructing a separate CRF with a distinct tag set, whose output sequences can be interpreted directly as paths through M .

Original: As M is unambiguous, each label sequence in L corresponds to exactly one such path.
Revised: As M is unambiguous, each label sequence in L corresponds to exactly one such path.

Original: We parameterize this auxiliary CRF identically to our original CRF – that is, with label-wise (not tag-wise) transition and emission functions.
Revised: We parameterize this auxiliary CRF identically to our original CRF – that is, with label-wise (not tag-wise) transition and emission functions.

Original: Thus, for all parameterizations θ , both distributions P θ ( y | x ) and P θ ( y | x , L ) are well deﬁned.
Revised: Thus, for all parameterizations θ , both distributions P θ ( y | x ) and P θ ( y | x , L ) are well deﬁned.

Original: There are many ways to construct such a CRF.
Revised: There are many ways to construct such a CRF.

Original: As CRF training and inference are quadratic in the size of the tag set Y , we would prefer a construction which minimizes | Y | .
Revised: As CRF training and inference are quadratic in the size of the tag set Y , we would prefer a construction which minimizes | Y | .

Original: However, for clarity, we will ﬁrst present a conceptually simple construction, andthen discuss approaches to reduce | Y | .
Revised: However, for clarity, we will ﬁrst present a conceptually simple construction, and discuss approaches to reduce | Y | in

Original:  We start with our original CRF, parameterized by θ , with tag set Y = Σ , transition function g θ , and emission function h θ , describing the probability distribution P θ ( y | x ) , y ∈ Σ ∗ .
Revised: Section 4.2. We start with our original CRF, parameterized by θ , with tag set Y = Σ , transition function g θ , and emission function h θ , describing the probability distribution P θ ( y | x ) , y ∈ Σ ∗ .

Original: From this, we construct a new CRF, also parameterized by the same θ , but with tag set Y (cid:48) , transition function g (cid:48) θ , and emission function h (cid:48) θ .
Revised: From this, we construct a new CRF, also parameterized by the same θ , but with tag set Y (cid:48) , transition function g (cid:48) θ , and emission function h (cid:48) θ .

Original: This auxiliary CRF describes the distribution P (cid:48) θ ( y (cid:48)
Revised: This auxiliary CRF describes the distribution P (cid:48) θ ( y (cid:48)

Original: | x ) (with y (cid:48) ∈ Y (cid:48)∗ ), which we will be able to interpret as P θ ( y | x , L ) .
Revised: | x ) (with y (cid:48) ∈ Y (cid:48)∗ ), which we will be able to interpret as P θ ( y | x , L ) .

Original: The construction is as follows:
Revised: The construction is as follows:

Original: [Equation]
Revised: [Equation]

Original: This means that the tags of our new CRF are the edges of M , the transition function assigns zero probability to transitions between edges which do not pass through a shared NFA state, and the emission function assigns zero probability to tag sequences which do not begin at the starting state or end at an accepting state.
Revised: This means that the tags of our new CRF are the edges of M , the transition function assigns zero probability to transitions between edges which do not pass through a shared NFA state, and the emission function assigns zero probability to tag sequences which do not begin at the starting state or end at an accepting state.

Original: Apart from these constraints, the transition and emission functions depend only on edge labels, and not on the edges themselves, and agree with the standard CRF’s g θ
Revised: Apart from these constraints, the transition and emission functions depend only on edge labels, and not on the edges themselves, and agree with the standard CRF’s g θ

Original: and h θ when no constraints are violated.
Revised: and h θ when no constraints are violated.

Original: As M is unambiguous, every tag sequence y corresponds to a single path through M , representable as an edge sequence π = (cid:104) π 1 , π 2 , ..., π t (cid:105) , π i ∈ E .
Revised: As M is unambiguous, every tag sequence y corresponds to a single path through M , representable as an edge sequence π = (cid:104) π 1 , π 2 , ..., π t (cid:105) , π i ∈ E .

Original: Since this path is a tag sequence for our auxiliary
Revised: Since this path is a tag sequence for our auxiliary

Original: CRF, we can directly calculate the auxiliary CRF’s P (cid:48)
Revised: CRF, we can directly calculate the auxiliary CRF’s P (cid:48)

Original: θ ( π | x ) .
Revised: θ ( π | x ) .

Original: From the construction of g (cid:48)
Revised: From the construction of g (cid:48)

Original: θ and h (cid:48)
Revised: θ and h (cid:48)

Original: θ , this must be equal to P θ ( y | x , L ) , as it is proportional to P θ ( y | x ) for y ∈ L , and zero (or, more correctly, undeﬁned) otherwise.
Revised: θ , this must be equal to P θ ( y | x , L ) , as it is proportional to P θ ( y | x ) for y ∈ L , and zero (or, more correctly, undeﬁned) otherwise.

Original: Figure 1 illustrates this construction with a concrete example.
Revised: Figure 1 illustrates this construction with a concrete example.

Original: 4. T AG - SET MINIMIZATION
Revised: 4. T IME AND SPACE EFFICIENCY

Original: As the Viterbi and forward algorithms are quadratic in | Y | , very large tag sets can lead to performance issues.
Revised: As the Viterbi and forward algorithms are quadratic in | Y | , very large tag sets can lead to performance issues, possibly making training or inference intractible in extreme cases.

Original: To alleviate this, we would like| Y | to  be as small as possible. Without changing L ,there are two approaches we can take here: First, we can select  M to have as few edges as possible.
Revised: Thus, we would like to characterize under which conditions a RegCCRF can be used tractibly, and identify techniques for improving performance. As Y corresponds to the edges of M , we would like to select our unambiguous automaton M to have as few edges as possible.

Original: This problemfor unambiguous NFAs is NP-completein the general case (Jiang & Ravikumar, 1991), and, assuming P (cid:54) =
Revised: For arbitrary languages, this problem is NP-complete (Jiang & Ravikumar, 1991), and, assuming P (cid:54) =

Original: NP, is not even efﬁciently approximable (Gruber & Holzer, 2007).
Revised: NP, is not even efﬁciently approximable (Gruber & Holzer, 2007).

Original: Nonetheless, in practice many languages can be minimized manually, and heuristic approaches can reduce, if not minimize, the size of NFAs.
Revised: Nonetheless, for many common classes of languages, there exist approaches to obtain a tractably small automaton. One straightforward method is to construct M directly from a short unambiguous regular expression.

Revised: Br¨uggemann-Klein & Wood (1992) present a simple algorithm for constructing an unambiguous automaton from an unambiguous regular expression, with | Q | linear in the length of the expression.

Revised: Using this method to construct M , the time- and space-complexity of Viterbi are polynomial in the length of our regular expression, with a worst-case of quartic complexity when the connectivity graph of M is dense.

Original: Second, we can  reduce the size of | Y | by adjusting our construction.
Revised: For many other tasks, a reasonable approach is to leverage domain knowledge about the constraints to manually construct a small unambiguous automaton. For example, if the constraints require that a particular label occurs exactly n times in the output sequence, an automaton could be constructed manually to count ocurrences of that label. Multiple constraints of this type can then be composed via automaton union and intersection. Without making changes to M , we can also reduce the size of | Y | by adjusting our construction.

Original: Instead of making each edge of M a tag, we can adopt equivalence classes of edges.
Revised: Instead of making each edge of M a tag, we can adopt equivalence classes of edges.

Original: Reminiscent of standard NFA minimization, we deﬁne ( q, a, r ) ∼ ( q (cid:48) , a (cid:48) , r (cid:48) )
Revised: Reminiscent of standard NFA minimization, we deﬁne ( q, a, r ) ∼ ( q (cid:48) , a (cid:48) , r (cid:48) )

Original: ↔ ( q, a ) =
Revised: ↔ ( q, a ) =

Original: ( q (cid:48) , a (cid:48) ) .
Revised: ( q (cid:48) , a (cid:48) ) .

Original: When constructing our CRF, whenever a transition would have been allowed between two edges, we allow a transition between their corresponding equivalence classes.
Revised: When constructing our CRF, whenever a transition would have been allowed between two edges, we allow a transition between their corresponding equivalence classes.

Original: We do the same to determine which classes are allowed as initial or ﬁnal tags.
Revised: We do the same to determine which classes are allowed as initial or ﬁnal tags.

Original: As each equivalence class corresponds (non-uniquely) to a single symbol a , we can translate between tag sequences and strings of L just as before.
Revised: As each equivalence class corresponds (non-uniquely) to a single symbol a , we can translate between tag sequences and strings of L just as before.

Original: 4. I NTERPRETATION AS A WEIGHTED FINITE - STATE TRANSDUCER
Revised: 4. I NTERPRETATION AS A WEIGHTED FINITE - STATE TRANSDUCER

Original: While we present our model as a variation of a standard CRF which enforces regular-language constraints, an alternate characterization is as a weighted ﬁnite-state transducer with the transducer topology and weight parameterization chosen so as to yield the distribution P θ ( y | x , L ) .
Revised: While we present our model as a variation of a standard CRF which enforces regular-language constraints, an alternate characterization is as a weighted ﬁnite-state transducer with the transducer topology and weight parameterization chosen so as to yield the distribution P θ ( y | x , L ) .

Original: In order to accommodate CRF transition weights, such an approach involves weight-learning in an auxiliary automaton whose edges correspond to edge-pairs in M – we give a full construction in Appendix B.
Revised: In order to accommodate CRF transition weights, such an approach involves weight-learning in an auxiliary automaton whose edges correspond to edge-pairs in M – we give a full construction in Appendix C.

Original: This interpretation enables direct comparison to studies on weight learning in ﬁnite-state transducers, such as Rastogi et al.
Revised: This interpretation enables direct comparison to studies on weight learning in ﬁnite-state transducers, such as Rastogi et al.

Original: While RegCCRFs can be viewed as special cases of neural-weighted
Revised: While RegCCRFs can be viewed as special cases of neural-weighted

Original: FSTs, they inherit a number of useful properties from CRFs not possessed by neural-weighted automata in general.
Revised: FSTs, they inherit a number of useful properties from CRFs not possessed by neural-weighted automata in general.

Original: Firstly, as | y | is necessarily equal to | x | , the partition function (cid:80) y ∈L P θ ( y | x , L ) is guaranteed to be ﬁnite, and P θ ( y | x , L ) is a well-deﬁned probability distribution for all θ , which is not true for weighted transducers in general, which may admit paths with unbounded lengths and weights.
Revised: Firstly, as | y | is necessarily equal to | x | , the partition function (cid:80) y ∈L P θ ( y | x , L ) is guaranteed to be ﬁnite, and P θ ( y | x , L ) is a well-deﬁned probability distribution for all θ , which is not true for weighted transducers in general, which may admit paths with unbounded lengths and weights.

Original: Secondly, as M is assumed to be unambiguous, string probabilities correspond exactly to path probabilities, allowing for exact MAP inference with the Viterbi algorithm.
Revised: Secondly, as M is assumed to be unambiguous, string probabilities correspond exactly to path probabilities, allowing for exact MAP inference with the Viterbi algorithm.

Original: In contrast, ﬁnding the most probable string in the highly ambiguous automata used when learning edge weights for an unknown language is NP-Hard
Revised: In contrast, ﬁnding the most probable string in the highly ambiguous automata used when learning edge weights for an unknown language is NP-Hard

Original: (Casacuberta & de la Higuera, 1999), necessitating approximation methods such as crunching (May & Knight, 2006).
Revised: (Casacuberta & de la Higuera, 1999), necessitating approximation methods such as crunching (May & Knight, 2006).

Original: Finally, as each RegCCRF can be expressed as a CRF with a particular parameterization, the convexity guarantees of standard CRFs carry over, in that the loss is convex with respect to emission and transition scores.
Revised: Finally, as each RegCCRF can be expressed as a CRF with a particular parameterization, the convexity guarantees of standard CRFs carry over, in that the loss is convex with respect to emission and transition scores.

Original: In contrast, training losses in general weighted ﬁnite-state transducers are usually nonconvex (Rastogi et al., 2016).
Revised: In contrast, training losses in general weighted ﬁnite-state transducers are usually nonconvex (Rastogi et al., 2016).

Original:  C ONSTRAINED TRAINING VS . CONSTRAINED DECODING
Revised:  C ONSTRAINED TRAINING VS . CONSTRAINED DECODING

Original: Our construction suggests two possible use cases for a RegCCRF: constrained decoding , where a
Revised: Our construction suggests two possible use cases for a RegCCRF: constrained decoding , where a

Original: CRF is trained unconstrained, and the learned weights are then used in a RegCCRF at decoding time, and constrained training , where a RegCCRF is both trained and decoded with constraints.
Revised: CRF is trained unconstrained, and the learned weights are then used in a RegCCRF at decoding time, and constrained training , where a RegCCRF is both trained and decoded with constraints.

Original: In this section, we compare between these two approaches and a standard, unconstrained CRF .
Revised: In this section, we compare between these two approaches and a standard, unconstrained CRF .

Original: We assume a machine learning setting where we have access to samples from some data distribution (cid:101) P ( x , y ) , with each x ∈ X ∗ , and each y of matching length in some regular language L ⊆ Σ ∗ .
Revised: We assume a machine learning setting where we have access to samples from some data distribution (cid:101) P ( x , y ) , with each x ∈ X ∗ , and each y of matching length in some regular language L ⊆ Σ ∗ .

Original: We wish to model the conditional distribution (cid:101) P ( y | x ) with either a CRF or a RegCCRF, by way of maximizing the model’s (log) likelihood given the data distribution.
Revised: We wish to model the conditional distribution (cid:101) P ( y | x ) with either a CRF or a RegCCRF, by way of maximizing the model’s (log) likelihood given the data distribution.

Original: The unconstrained CRF corresponds to a CRF that has been trained, without constraints, on data from (cid:101) P ( x , y ) , and is used directly for inference: It makes no use of the language L .
Revised: The unconstrained CRF corresponds to a CRF that has been trained, without constraints, on data from (cid:101) P ( x , y ) , and is used directly for inference: It makes no use of the language L .

Original: The model’s output distribution is P θ u ( y | x ) , with parameter vector θ u minimizing the NLL objective:
Revised: The model’s output distribution is P θ u ( y | x ) , with parameter vector θ u minimizing the NLL objective:

Original: [Equation]
Revised: [Equation]

Original: In constrained decoding, a CRF is trained unconstrained, but its weights are used in a RegCCRF at decoding time.
Revised: In constrained decoding, a CRF is trained unconstrained, but its weights are used in a RegCCRF at decoding time.

Original: The output distribution of such a model is P θ u ( y | x , L ) .
Revised: The output distribution of such a model is P θ u ( y | x , L ) .

Original: It is parameterized by the same parameter vector θ u as the unconstrained CRF, as the training procedure is identical, but the output distribution is conditioned on y ∈ L .
Revised: It is parameterized by the same parameter vector θ u as the unconstrained CRF, as the training procedure is identical, but the output distribution is conditioned on y ∈ L .

Original: Constrained training involves directly optimizing a RegCCRF’s output distribution, avoiding any asymmetry between training and decoding time.
Revised: Constrained training involves directly optimizing a RegCCRF’s output distribution, avoiding any asymmetry between training and decoding time.

Original: In this case, the output distribution of the model is
Revised: In this case, the output distribution of the model is

Original: P θ c ( y | x , L ) , where
Revised: P θ c ( y | x , L ) , where

Original: [Equation]
Revised: [Equation]

Original: is the parameter vector which minimizes the NLL of the RegCCRF’s constrained distribution.
Revised: is the parameter vector which minimizes the NLL of the RegCCRF’s constrained distribution.

Original: In the remainder of this section, we will demonstrate that these three approaches form a hierarchy in terms of their ability to match the data distribution.
Revised: These three approaches form a hierarchy in terms of their ability to match the data distribution:

Original: When compared by NLL, we show that L unconstrained ≥ L constrained decoding ≥ L constrained training , with each L corresponding to that model’s negative log-likelihood.
Revised: L unconstrained ≥ L constrained decoding ≥ L constrained training , with each L corresponding to the negative loglikelihood assigned by each model to the data; see Appendix B for a proof.

Original: This suggests we should prefer the constrained training regimen.
Revised: This suggests we should prefer the constrained training regimen.

Original: Theorem 1.

Original: For arbitrary θ : E x , y ∼ (cid:101) P [ −

Original: [Equation]

Original: P

Original: [Equation]

Original: ( y |

Original: [Equation]

Original: ] ≥ E x , y

Original: [Equation]

Original: P [ − ln P θ ( y | x , L )]

Original: Here we compare the distributions P θ ( y | x ) and P θ ( y | x , L ) .

Original: We wish to demonstrate that

Original: P θ ( y | x ) can never achieve lower NLL than P θ ( y | x , L ) , and that the two distributions achieve identical NLL only when P θ ( y | x ) =

Original: P θ ( y | x , L ) i.e. when constraints have no effect.

Original: Proof.

Original: Since every y in (cid:101) P is in L ,

Original: [Equation]

Original: with α ≥ 1 .

Original: Thus, the NLL of the regular-constrained CRF is

Original: [Equation]

Original: This differs from the NLL of the unconstrained CRF only by the term − ln α .

Original: As α ≥ 1 , the regularconstrained CRF’s NLL is less than or equal to that of the unconstrained CRF, with equality only when α = 1 and therefore P θ ( y | x ) =

Original: P θ ( y | x , L ) .

Original: (cid:4)

Original: Theorem 2.

Original: For arbitrary θ : E x , y ∼ (cid:101) P [ − ln P θ c ( y | x , L )] ≥ E x , y ∼ (cid:101) P [ − ln P θ c ( y | x , L )]

Original: In this case, we compare the distributions P θ u ( y | x , L ) and P θ c ( y | x , L ) .

Original: We will demonstrate that the former cannot achieve a lower NLL against the data distribution than the latter.

Original: Proof.

Original: This follows directly from our deﬁnitions, as we deﬁne θ c to minimize the NLL of

Original: P θ ( y | x , L ) against the data distribution.

Original: Thus, P θ u ( y | x , L ) could never yield a lower NLL than

Original: P θ c ( y | x , L ) , as that would contradict our deﬁnition of θ c .

Original: (cid:4)

Original:  S YNTHETIC D ATA E XPERIMENTS
Revised:  S YNTHETIC D ATA E XPERIMENTS

Original: Whilewe have shown that constrained training cannot underperform constrained decoding, the conditions where it is strictly better depend on exactly how the transition and emission functions are parameterized, and are not easily stated in general terms.
Revised: While constrained training cannot underperform constrained decoding, the conditions where it is strictly better depend on exactly how the transition and emission functions are parameterized, and are not easily stated in general terms.

Original: In this section, we compare the two regimens empirically to demonstrate simple cases where the two are not equivalent.
Revised: We now empirically show two simple experiments on synthetic data where the two are not equivalent.

Original: The procedure is similar for bothsynthetic data experiments.
Revised: The procedure is similar for both experiments.

Original: For each experiment, we specify a regular language L , an observation alphabet X , and a joint data distribution (cid:101) P ( x , y ) over observation sequences in X ∗ and label sequences in L .
Revised: We specify a regular language L , an observation alphabet X , and a joint data distribution (cid:101) P ( x , y ) over observation sequences in X ∗ and label sequences in L .

Original: We then train two models, one with a RegCCRF, parameterized by θ c , and one with an unconstrained CRF, parameterized by θ u .
Revised: We then train two models, one with a RegCCRF, parameterized by θ c , and one with an unconstrained CRF, parameterized by θ u .

Original: For each model, we initialize parameters randomly, then use stochastic gradient descent to minimize NLL with (cid:101) P .
Revised: For each model, we initialize parameters randomly, then use stochastic gradient descent to minimize NLL with (cid:101) P .

Original: We directly generate samples from (cid:101) P to use as training data.
Revised: We directly generate samples from (cid:101) P to use as training data.

Original: After optimizing θ c and θ u , construct a RegCCRF with θ u for use as a constrained-decoding model, and we compare the constrained-training distribution P θ c ( y | x , L ) with the constrained-decoding distribution P θ u ( y | x , L ) .
Revised: After optimizing θ c and θ u , construct a RegCCRF with θ u for use as a constrained-decoding model, and we compare the constrained-training distribution P θ c ( y | x , L ) with the constrained-decoding distribution P θ u ( y | x , L ) .

Original: We use a simple architecture for our models, with both the transition functions g θ and emission functions h θ represented as parameter matrices.
Revised: We use a simple architecture for our models, with both the transition functions g θ and emission functions h θ represented as parameter matrices.

Original: We list training hyperparameters in Appendix A.
Revised: We list training hyperparameters in Appendix A.

Original: [Figure]

Original: hood

Original: Constrained decoding Constrained training

Original: [Figure]

Original: 6. A RBITRARY DIFFERENCES IN LIKELIHOOD
Revised: 6. A RBITRARILY LARGE DIFFERENCES IN LIKELIHOOD

Original: We would like to demonstrate that, when comparing constrained training to constrained decoding in terms of likelihood, constrained training can outperform constrained decoding by an arbitrary margin.
Revised: We would like to demonstrate that, when comparing constrained training to constrained decoding in terms of likelihood, constrained training can outperform constrained decoding by an arbitrary margin.

Original: We choose L = ( ac ) ∗ | ( bc ) ∗ to make conditional independence particularly relevant – as every even-indexed label is c , an unconstrained CRF must model odd-indexed labels independently, while a constrained CRF can use its constraints to account for nonlocal dependencies.
Revised: We choose L = ( ac ) ∗ | ( bc ) ∗ to make conditional independence particularly relevant – as every even-indexed label is c , an unconstrained CRF must model odd-indexed labels independently, while a constrained CRF can use its constraints to account for nonlocal dependencies.

Original: For simplicity, we hold the input sequence constant, with X = { o } .
Revised: For simplicity, we hold the input sequence constant, with X = { o } .

Original: Our approach is to construct sequences of various lengths.
Revised: Our approach is to construct sequences of various lengths.

Original: For k ∈ N , we let our data distribution be
Revised: For k ∈ N , we let our data distribution be

Original: [Equation]
Revised: [Equation]

Original:  P [Equation]
Revised: [Figure] k P (( ac ) k )

Original: and
Revised: Constrained decoding Constrained training Data distribution         k N e g a ti v e l og - li k e li hood Constrained decoding Constrained training

Original: [Equation]
Revised: Data distribution entropy [Table] As the marginal distributions for odd-indexed characters are not independent, an unconstrained CRF cannot exactly represent the distribution (cid:101) P .

Original: We train and evaluate individual models for each sequence length k .
Revised: We train and evaluate individual models for each sequence length k .

Original: Figure 2 plots model probabilities and NLL losses for various k .
Revised: Figure 2 plots model probabilities and NLL losses for various k .

Original: We see that, regardless of k , P θ c ( y | x , L ) is able to match (cid:101) P ( y | x ) almost exactly, with only small deviations due to sampling noise in SGD.
Revised: We see that, regardless of k , P θ c ( y | x , L ) is able to match (cid:101) P ( y | x ) almost exactly, with only small deviations due to sampling noise in SGD.

Original: On the other hand, as sequence length increases, P θ u ( y | x , L ) becomes progressively “lop-sided”, assigning almost all of its probability mass to the string ( ac ) k .
Revised: On the other hand, as sequence length increases, P θ u ( y | x , L ) becomes progressively “lop-sided”, assigning almost all of its probability mass to the string ( ac ) k .

Original: This behavior is reﬂected  the models’ likelihoods – constrained training stays at near-constant likelihood for all k , while the negative log-likelihood of constrained decoding grows linearly with k .
Revised: This behavior is reﬂected in the models’ likelihoods – constrained training stays at near-constant likelihood for all k , while the negative log-likelihood of constrained decoding grows linearly with k .

Original: 6. D IFFERENCES IN MAP INFERENCE
Revised: 6. D IFFERENCES IN MAP INFERENCE

Original: We show here that constrained training and constrained decoding can disagree about which label sequence they deem most likely.
Revised: We show here that constrained training and constrained decoding can disagree about which label sequence they deem most likely.

Original: Furthermore, in this case, MAP inference agrees with the data distribution’s mode for constrained training, but not for constrained decoding.
Revised: Furthermore, in this case, MAP inference agrees with the data distribution’s mode for constrained training, but not for constrained decoding.

Original: To do this, we construct a ﬁxed-length output language L = acd | bcd | bce , where an unconstrained CRF isbe limited by the Markov property to predict y ’s preﬁx and sufﬁx independently.
Revised: To do this, we construct a ﬁxed-length output language L = acd | bcd | bce , where an unconstrained CRF is limited by the Markov property to predict y ’s preﬁx and sufﬁx independently, and choose a data distribution which violates this independence assumption.

Original: We select our data distribution,(cid:101) P ( ooo , acd )
Revised: We select our data distribution,

Original: = 0 .
Revised: [Equation]

Original: 4 and (cid:101) P ( ooo , bcd ) = 0 . 3 and (cid:101) P ( ooo , bce ) = 0 . 3 , (11) to be close to uniform, but with one label sequence holding the slight majority, and we ensure that the majority label sequence is not the label sequence with both the majority preﬁx and the majority sufﬁx (i.e. bcd ).
Revised: to be close to uniform, but with one label sequence holding the slight majority, and we ensure that the majority label sequence is not the label sequence with both the majority preﬁx and the majority sufﬁx (i.e. bcd ).

Original: As before, we hold the observation sequence as a constant ( ooo ).
Revised: As before, we hold the observation sequence as a constant ( ooo ).

Original: We train a constrained and an unconstrained CRF to convergence, and compare P θ u ( y | x , L ) to P θ c ( y | x , L ) .
Revised: We train a constrained and an unconstrained CRF to convergence, and compare P θ u ( y | x , L ) to P θ c ( y | x , L ) .

Original: Table 1 shows P θ u
Revised: Table 1 shows P θ u

Original: ( y | x , L ) and P θ c ( y | x , L ) as they compare to (cid:101) P ( y | x ) .
Revised: ( y | x , L ) and P θ c ( y | x , L ) as they compare to (cid:101) P ( y | x ) .

Original: We ﬁnd that, while the mode of (cid:101) P ( y | x ) is acd , with probability of 0.4, the mode of constrained decoding distribution
Revised: We ﬁnd that, while the mode of (cid:101) P ( y | x ) is acd , with probability of 0.4, the mode of constrained decoding distribution

Original: P θ u ( y | x , L ) is bcd , the string with the majority preﬁx and the majority sufﬁx, to which the model assigns a probability of 0.48.
Revised: P θ u ( y | x , L ) is bcd , the string with the majority preﬁx and the majority sufﬁx, to which the model assigns a probability of 0.48.

Original: Conversely, the constrained training distribution P θ c ( y | x , L ) matches the data distribution almost exactly, and predicts the correct mode.[Table]
Revised: Conversely, the constrained training distribution P θ c ( y | x , L ) matches the data distribution almost exactly, and predicts the correct mode.

Original:  R EAL - WORLD DATA EXPERIMENT : SEMANTIC ROLE LABELING
Revised:  R EAL - WORLD DATA EXPERIMENT : SEMANTIC ROLE LABELING

Original: Task.
Revised: Task.

Original: As a ﬁnal experiment, we apply our RegCCRF to the NLP task of semantic role labeling (SRL) in the popular PropBank framework (Palmer et al., 2005).
Revised: As a ﬁnal experiment, we apply our RegCCRF to the NLP task of semantic role labeling (SRL) in the popular PropBank framework (Palmer et al., 2005).

Original: In line with previous work, we adopt the known-predicate setting , where events are given and the task is to mark token spans as (types of) event participants.
Revised: In line with previous work, we adopt the known-predicate setting , where events are given and the task is to mark token spans as (types of) event participants.

Original: PropBank assumes 7 semantic core roles (ARG0 through ARG5 plus
Revised: PropBank assumes 7 semantic core roles (ARG0 through ARG5 plus

Original: ARGA)
Revised: ARGA)

Original: plus 21 non-core roles for modiﬁers such as times or locations.
Revised: plus 21 non-core roles for modiﬁers such as times or locations.

Original: For example, in [ ARG0 Peter ] saw [ ARG1 Paul ] [ ARGM-TMP yesterday ] , the argument labels inform us who does the seeing (ARG0), who is seen (ARG1), and when the event took place (ARGM-TMP).
Revised: For example, in [ ARG0 Peter ] saw [ ARG1 Paul ] [ ARGM-TMP yesterday ] , the argument labels inform us who does the seeing (ARG0), who is seen (ARG1), and when the event took place (ARGM-TMP).

Original: In addition, role spans may be labeled as continuations of previous role spans, or as references to another role span in the sentence.
Revised: In addition, role spans may be labeled as continuations of previous role spans, or as references to another role span in the sentence.

Original: SRL can be framed naturally as a sequence labeling task (He et al., 2017).
Revised: SRL can be framed naturally as a sequence labeling task (He et al., 2017).

Original: However, the task comes with a number of hard constraints that are not automatically satisﬁed by standard CRFs, namely: (1)
Revised: However, the task comes with a number of hard constraints that are not automatically satisﬁed by standard CRFs, namely: (1)

Original: Each core role may occur at most once per event; (2) for continuations, the span type must occur previously in the sentence; (3) for references, the span type must occur elsewhere in the sentence.
Revised: Each core role may occur at most once per event; (2) for continuations, the span type must occur previously in the sentence; (3) for references, the span type must occur elsewhere in the sentence.

Original: Data.
Revised: Data.

Original: In line with previous work (Ouchi et al., 2018), we work with the OntoNotes corpus as used in the CoNLL 2012 shared task 1 (Weischedel et al., 2011; Pradhan et al., 2012), whose training set comprises 66 roles (7 core roles, 21 non-core roles, 19 continuation types, and 19 reference types).
Revised: In line with previous work (Ouchi et al., 2018), we work with the OntoNotes corpus as used in the CoNLL 2012 shared task 1 (Weischedel et al., 2011; Pradhan et al., 2012), whose training set comprises 66 roles (7 core roles, 21 non-core roles, 19 continuation types, and 19 reference types).

Original: RegCCRF Models.
Revised: RegCCRF Models.

Original: To encode the three constraints listed above in a RegCCRF, we deﬁne a regular language describing valid BIO sequences (Ramshaw & Marcus, 1999) over the 66 roles.
Revised: To encode the three constraints listed above in a RegCCRF, we deﬁne a regular language describing valid BIO sequences (Ramshaw & Marcus, 1999) over the 66 roles.

Original: A minimal unambiguous NFA for this language has more than 2 2 · 3 19 states, which is too large to run the
Revised: A minimal unambiguous NFA for this language has more than 2 2 · 3 19 states, which is too large to run the

Original: Viterbi algorithm on our hardware.
Revised: Viterbi algorithm on our hardware.

Original: However, as many labels are very rare, we can shrink our automaton by discarding them at the cost of imperfect recall.
Revised: However, as many labels are very rare, we can shrink our automaton by discarding them at the cost of imperfect recall.

Original: We achieve further reductions in size by ignoring constraints on reference roles, treating them identically to non-core roles.
Revised: We achieve further reductions in size by ignoring constraints on reference roles, treating them identically to non-core roles.

Original: Our ﬁnal automaton recognizes 5 core role types (ARG0 through ARG4), 17 non-core / reference roles, and one continuation role type (for ARG1).
Revised: Our ﬁnal automaton recognizes 5 core role types (ARG0 through ARG4), 17 non-core / reference roles, and one continuation role type (for ARG1).

Original: This automaton has 672 states, yielding a RegCCRF with 2592 tags.
Revised: This automaton has 672 states, yielding a RegCCRF withtags.

Revised: A description of our procedure for constructing this automaton can be found in Appendix D.

Original: Our model architecture is given by this RegCCRF, with emission scores provided by a linear projection of the output of a pretrained RoBERTa network Liu et al.
Revised: Our model architecture is given by this RegCCRF, with emission scores provided by a linear projection of the output of a pretrained RoBERTa network Liu et al.

Original: In order to provide the model with event information, the given predicates are preﬁxed by a special reserved token in the input sequence.
Revised: In order to provide the model with event information, the given predicates are preﬁxed by a special reserved token in the input sequence.

Original: RoBERTa parameters are ﬁne-tuned during the learning of transition scores and projection weights.
Revised: RoBERTa parameters are ﬁne-tuned during the learning of transition scores and projection weights.

Original: We perform experiments with both constrained training and constrained decoding settings – we will refer to these as CT and CD respectively.
Revised: We perform experiments with both constrained training and constrained decoding settings – we will refer to these as ConstrTr and ConstrDec respectively.

Original: A full description of the training procedure is provided in Appendix A. As RegCCRF loss is only ﬁnite for label sequences in L , we must ensure that our training data do not violate our constraints.
Revised: A full description of the training procedure, including training times, is provided in Appendix A. As RegCCRF loss is only ﬁnite for label sequences in L , we must ensure that our training data do not violate our constraints.

Original: We discard some roles, as described above, by simply removing the offending labels from the training data.
Revised: We discard some roles, as described above, by simply removing the offending labels from the training data.

Original: In six cases, training instances directly conﬂict with the constraints speciﬁed — all cases involve continuation roles missing a valid preceding role.
Revised: In six cases, training instances directly conﬂict with the constraints speciﬁed — all cases involve continuation roles missing a valid preceding role.

Original: We discard these instances for CT.
Revised: We discard these instances for ConstrTr .

Original: CRF Baseline.
Revised: CRF Baselines.

Original: Asa baseline model, we use the same architecture, but with a standard CRF instead of the RegCCRF.
Revised: As baseline models, we use the same architecture, but with a standard CRF replacing the RegCCRF.

Original: Sincefor this baseline, we are not limited by GPU memory, we  include all role types present in the training set, and use the complete training set.
Revised: Since we are not limited by GPU memory for CRFs, we are optionally able to include all role types present in the training set, using the complete training set.

Original: [Table]
Revised: We present two CRF baseline models: CRF-full , which is trained on all role-types from the training set, and CRFreduced , which includes the same subset of roles as the RegCCRF models. For CRF-reduced , we use the same learned weights as for CD, but we decode without constraints.

Original: Results and analysis.
Revised: Results and analysis.

Original: We evaluate ourthree models— CRF, CT, and CD — on the evaluation partition, and measure performance using F 1 score for exact span matches.
Revised: We evaluate our models on the evaluation partition, and measure performance using F 1 score for exact span matches.

Original: For comparability with prior work, we use the evaluation script 2 for the CoNLL-2005 shared task (Carreras & M`arquez, 2005).
Revised: For comparability with prior work, we use the evaluation script 2 for the CoNLL-2005 shared task (Carreras & M`arquez, 2005).

Original: These results, averaged over ﬁve trials, are presented in Table 2.
Revised: These results, averaged over ﬁve trials, are presented in Table 2.

Original: All of our models outperform the existing state-of-the-art ensemble model Ouchi et al.
Revised: Excepting CRF-reduced , all of our models outperform the existing state-of-the-art ensemble model Ouchi et al.

Original: We ascribe this improvement over the existing literature to our use of RoBERTa – prior work in SRL tends to rely on ELMo (Peters et al., 2018), which tends to do worse than transformer-based models on downstream tasks (Devlin et al., 2019). 
Revised: We ascribe this improvement over the existing literature to our use of RoBERTa – prior work in SRL relies on ELMo (Peters et al., 2018), which tends to underperform transformer-based models on downstream tasks (Devlin et al., 2019). [Table] Ourexperiments [Table]

Revised: Resultsfromliterature

Original:  Of our models, CT signiﬁcantly 3 outperforms CD and CRF in F 1 score, yielding a new SOTA for SRL on the OntoNotes corpus.
Revised: [Table] Of our models [Table] ConstrTr signiﬁcantly 3 outperforms the others in F 1 score and yields a new SOTA for SRL on OntoNotes, in line with expectations from theoretical analysis and on synthetic data.

Original: Our other two models, while attaining similar F1 scores, show a trade-off between precision and recall, with CRF attaining signiﬁcantly higher recall and the CD showing signiﬁcantly higher precision.
Revised: Our other three models show trade-offs between precision and recall, wherein CRF-full outperforms the other two in recall and underperforms them in precision.

Original: This is in line with what would be expected – as our RegCCRF models are unable to predict rare role types, they are intrinsically limited in their capabilities for recall, with a theoretical maximum of 99% recall.
Revised: This is not surprising, as CRF-full is the only model capable of predicting rare role types. The other models, which use the reduced tag sets, have a theoretical maximum of 99% recall.

Original: Conversely, their constraints ensure that predictions are well-formed, and should increase precision by excluding ill-formed outputs. Indeed, CT achieves its high F 1 score by combining the high recall of CRF with the high precision of CD.
Revised: Interestingly, when comparing between the three models that use the reduced tag sets, we ﬁnd a statistically signiﬁcant interaction between the constraint setting and model recall, but not between constraints and model precision: ConstrTr has signiﬁcantly higher recall than ConstrDec , which in turn signiﬁcantly beats CRF-reduced in recall, but there are no statistically signiﬁcant differences between these models’ precisions.

Original: The process of training with constraints improves the model’s recall, bringing CT’s recall to a similar level as CRF’s without sacriﬁcing the boost in precision provided by the  constraints.
Revised: For our unconstrained models, CRF-full and CRF-reduced , the constraints speciﬁed in our automaton are violated in 0.81% and 0.84% of all output sequences respectively. 4 While this number is small, it should not be interpreted to mean that constraints are useless for almost all instances – as shown in Section 6.2, constraints during training can affect MAP inference even when none of the alternatives violate constraints.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We have presented a method to constrain the output of CRFs to a regular language.
Revised: We have presented a method to constrain the output of CRFs to a regular language.

Original: Our construction permits constraints to be used at training or prediction time; both theoretically and empirically, training-time constraining better captures the data distribution.
Revised: Our construction permits constraints to be used at training or prediction time; both theoretically and empirically, training-time constraining better captures the data distribution.

Original: Conceptually, our approach constitutes a novel bridge between constrained CRFs and neural-weighted FSTs.
Revised: Conceptually, our approach constitutes a novel bridge between constrained CRFs and neural-weighted FSTs.

Original: Future work could target enhancing the model’s expressibility, either by allowing constraints to depend explicitly on the input as regular relations, or by investigating non-binary constraints, i.e., regular language-based constraints with learnable weights.
Revised: Future work could target enhancing the model’s expressibility, either by allowing constraints to depend explicitly on the input as regular relations, or by investigating non-binary constraints, i.e., regular language-based constraints with learnable weights.

Original: Additionally, regular language induction (e.g. Dunay et al.
Revised: Additionally, regular language induction (e.g. Dunay et al.

Original: ; Bartoli et al.
Revised: ; Bartoli et al.

Original: (2016)) could be used to learn languages automatically, reducing manual speciﬁcation and identifying non-obvious constraints.
Revised: (2016)) could be used to learn languages automatically, reducing manual speciﬁcation and identifying non-obvious constraints.

Original: Another avenue for continuing research lies in identifying further applications for RegCCRFs.
Revised: Another avenue for continuing research lies in identifying further applications for RegCCRFs.

Original: The NLP task of relation extraction could be a fruitful target – RegCCRFs offer a mechanism to make the proposal of a relation conditional on the presence of the right number and type of arguments.
Revised: The NLP task of relation extraction could be a fruitful target – RegCCRFs offer a mechanism to make the proposal of a relation conditional on the presence of the right number and type of arguments.

Original: While our construction cannot be lifted directly to context-free languages due to the unbounded state space of the corresponding pushdown automata, context-free language can be approximated by regular languages (Mohri &
Revised: While our construction cannot be lifted directly to context-free languages due to the unbounded state space of the corresponding pushdown automata, context-free language can be approximated by regular languages (Mohri &

Original: Nederhof, 2001).
Revised: Nederhof, 2001).

Original: On this basis, for example, a RegCCRF backed by a regular language describing trees of a limited depth could also be applied to tasks with context-free constraints.
Revised: On this basis, for example, a RegCCRF backed by a regular language describing trees of a limited depth could also be applied to tasks with context-free constraints.

Original: To encourage the use of RegCCRFs, we provide an implementation as a Python library under the
Revised: To encourage the use of RegCCRFs, we provide an implementation as a Python library under the

Original: Apache 2.0 license which 
Revised: Apache 2.0 license which can be used as a drop-in replacement for standard CRFs in PyTorch.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: We have taken a number of steps to ensure the reproducibility of the models and experiments presented in this work.

Original: We have released all code for the RegCCRF model as an open-source Python 3 library under the Apache 2.0 license, which is included anonymized in the supplementary materials.
Revised: To ensure reproducibility, we have released all code for the RegCCRF model as an open-source Python 3 library under the Apache 2.0 license, which is included anonymized in the supplementary materials.

Original: Additionally, we include Python scripts for reproducing all experiments presented in the paper, detailed descriptions of our datasets and preprocessing steps, and training logs.
Revised: Additionally, we include Python scripts for reproducing all experiments presented in the paper, detailed descriptions of our datasets and preprocessing steps, and training logs.

Original: Furthermore, Appendix A lists all model hyperparameters, details the preprocessing steps taken for our experiments, and speciﬁes the hardware used for our experiments along with average training  times for each experiment.
Revised: Furthermore, Appendix A lists all model hyperparameters, details the preprocessing steps taken for our experiments and speciﬁes the hardware used for our experiments along with average training and inference times for each experiment.

Original: E THICS STATEMENT
Revised: E THICS STATEMENT

Revised: The research in this paper is fundamental in the sense that it enables machine learning models to better represent data and limit the search space at inference and learning time.

Original: We do notbelieve that there are ethically relevant aspects in this research.
Revised: It therefore does not in and of itself represent additional ethical risks on top of the previous work we build upon.

Original: A E XPERIMENTAL D ESIGN
Revised: A E XPERIMENTAL D ESIGN

Original: This appendix details the training procedures and hyperparameter choices for our experiments.
Revised: This appendix details the training procedures and hyperparameter choices for our experiments.

Original: These are summarized in Table 3.
Revised: These are summarized in Table 3.

Original: Full code for all experiments along with training logs, are also included in the supplementary materials.
Revised: Full code for all experiments, along with training logs, are also included in the supplementary materials.

Original: A. CRF S
Revised: A. CRF S

Original: For all CRFs and RegCCRFs, transition potentials were initialized randomly from a normal distribution with mean zero and standard deviation 0.1.
Revised: For all CRFs and RegCCRFs, transition potentials were initialized randomly from a normal distribution with mean zero and standard deviation 0.1.

Original: No CRFs or RegCCRFs employed special start- or end-transitions – that is, we did not insert any additional beginning-of-sequence or end-of-sequence tags for the Viterbi or forward algorithms. 
Revised: No CRFs or RegCCRFs employed special start- or end-transitions – that is, we did not insert any additional beginning-of-sequence or end-of-sequence tags for the Viterbi or forward algorithms. [Table]

Original: A. S YNTHETIC DATA EXPERIMENTS – TRAINING PROCEDURE
Revised: A. S YNTHETIC DATA EXPERIMENTS – TRAINING PROCEDURE

Original: For both synthetic data experiments, the emission potentials were represented explicitly for each position as trainable parameters – since the observation sequence was constant in all experiments, these did not depend on x .
Revised: For both synthetic data experiments, the emission potentials were represented explicitly for each position as trainable parameters – since the observation sequence was constant in all experiments, these did not depend on x .

Original: Parameters were initialized randomly using PyTorch default initialization, and optimized using stochastic gradient descent.
Revised: Parameters were initialized randomly using PyTorch default initialization, and optimized using stochastic gradient descent.

Original: To ensure fast convergence to a stable distribution, we employed learning rate decay – learning rate was initially set to 1.0, and reduced by 10% every 100 training steps.
Revised: To ensure fast convergence to a stable distribution, we employed learning rate decay – learning rate was initially set to 1.0, and reduced by 10% every 100 training steps.

Original: We trained all models for a total of 5000 steps with a batch size of 50.
Revised: We trained all models for a total of 5000 steps with a batch size of 50.

Original: All models were trained on
Revised: All models were trained on

Original: CPUs.
Revised: CPUs.

Original: For the experiment described in Section 6.1, we trained separate models for each k – the total training time for this experiment was approximately 35 minutes.
Revised: For the experiment described in Section 6.1, we trained separate models for each k – the total training time for this experiment was approximately 35 minutes.

Original: The experiment described in
Revised: The experiment described in

Original: Section 6.2 completed training in approximately 30 seconds.
Revised: Section 6.2 completed training in approximately 30 seconds.

Original: A. S EMANTIC ROLE LABELING – TRAINING PROCEDURE
Revised: A. S EMANTIC ROLE LABELING – TRAINING PROCEDURE

Original: In the semantic role labeling (SRL) experiments, we incorporated a pretrained RoBERTa network (Liu et al., 2019) – the implementation and weights for this model were obtained using the roberta-base model from the Hugging Face transformers library (Wolf et al., 2020).
Revised: In the semantic role labeling (SRL) experiments, we incorporated a pretrained RoBERTa network (Liu et al., 2019) – the implementation and weights for this model were obtained using the roberta-base model from the Hugging Face transformers library (Wolf et al., 2020).

Original: RoBERTa embeddings were projected down to transmission scores using a linear layer with a bias – projection weights and biases were initialized using the PyTorch default initialization.
Revised: RoBERTa embeddings were projected down to transmission scores using a linear layer with a bias – projection weights and biases were initialized using the PyTorch default initialization.

Original: Input tokens were sub-tokenized using RoBERTa’s tokenizer.
Revised: Input tokens were sub-tokenized using RoBERTa’s tokenizer.

Original: The marked predicate in each sentence was preﬁxed by a special <unk> token.
Revised: The marked predicate in each sentence was preﬁxed by a special <unk> token.

Original: During training, for efﬁciency reasons, we excluded all sentences with 120 or more subtokens – this amounted to 0.23% of all training instances.
Revised: During training, for efﬁciency reasons, we excluded all sentences with 120 or more subtokens – this amounted to 0.23% of all training instances.

Original: We nonetheless predicted on all instances, regardless of length.
Revised: We nonetheless predicted on all instances, regardless of length.

Original: We optimized models using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 10 − 5 .
Revised: We optimized models using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 10 − 5 .

Original: We ﬁne-tune RoBERTa parameters and learn the projection and RegCCRF weights jointly.
Revised: We ﬁne-tune RoBERTa parameters and learn the projection and RegCCRF weights jointly.

Original: For performance reasons, batch size was set to 2, but we utilized gradient accumulation over groups of 4 batches to simulate a batch size of 8.
Revised: For performance reasons, batch size was set to 2, but we utilized gradient accumulation over groups of 4 batches to simulate a batch size of 8.

Original: We utilized early stopping to avoid overﬁtting.
Revised: We utilized early stopping to avoid overﬁtting.

Original: Every 5000 training steps, we approximated our model’s F 1 score against a subset of the provided development partition, using a simpliﬁed reimplementation of the ofﬁcial evaluation script.
Revised: Every 5000 training steps, we approximated our model’s F 1 score against a subset of the provided development partition, using a simpliﬁed reimplementation of the ofﬁcial evaluation script.

Original: Each time we exceeded the previous best F 1 score for a model, we saved all model weights to disk.
Revised: Each time we exceeded the previous best F 1 score for a model, we saved all model weights to disk.

Original: After 50 such evaluations with no improvement, we terminated training, and used the last saved copy of model weights for ﬁnal evaluation.
Revised: After 50 such evaluations with no improvement, we terminated training, and used the last saved copy of model weights for ﬁnal evaluation.

Original: We performed all SRL experiments on GeForce GTX 1080 Ti GPUs.
Revised: We performed all SRL experiments on GeForce GTX 1080 Ti GPUs.

Original: Each experiment used a single
Revised: Each experiment used a single

Original: GPU.
Revised: GPU.

Original: Training took an average of 88 hours for RegCCRF models with constrained training, 23 hours for RegCCRF with constrained decoding, and 24 hours for CRF baseline models.
Revised: Training took an average of 88 hours for RegCCRF models with constrained training,hours for RegCCRF with constrained decoding, and 24 hours for CRF baseline models.

Revised: Inference on the complete test set took an average of 18 minutes 55 seconds for CT and CD, and an average of 55 seconds for CRF-full and CRF-reduced.

Original: All training logs with timestamps are included in the supplementary materials.
Revised: All training logs with timestamps are included in the supplementary materials.

Original: B C ONSTRUCTION AS WEIGHTED FST
Revised: B P ROOF OF CONSTRAINED TRAINING INEQUALITY

Revised: In this appendix, we prove the inequality presented in section 5: when compared by NLL against the data distribution, L unconstrained ≥ L constrained decoding ≥ L constrained training , with each L corresponding to that model’s negative log-likelihood.

Revised: We ﬁrst prove the left side of this inequality, comparing an unconstrained CRF to constrained decoding, and then prove the right side, comparing constrained decoding to constrained training.

Revised: We use the notation introduced in Section 5.

Revised: Theorem 1.

Revised: For

Revised: [Equation]

Revised: Here we compare the distributions P θ ( y | x ) and P θ ( y | x , L ) .

Revised: We wish to demonstrate that

Revised: P θ ( y | x ) can never achieve lower NLL than P θ ( y | x , L ) , and that the two distributions achieve identical NLL only when P θ ( y | x ) =

Revised: P θ ( y | x , L ) i.e. when constraints have no effect.

Revised: Of note, this proof is valid for all parameterizations θ , and not just for θ u .

Revised: Proof.

Revised: Since every y in (cid:101) P is in L ,

Revised: [Equation]

Revised: with α ≥ 1 .

Revised: Thus, the NLL of the regular-constrained CRF is

Revised: [Equation]

Revised: This differs from the NLL of the unconstrained CRF only by the term − ln α .

Revised: As α ≥ 1 , the regularconstrained CRF’s NLL is less than or equal to that of the unconstrained CRF, with equality only when α = 1 and therefore P θ ( y | x ) =

Revised: P θ ( y | x , L ) .

Revised: (cid:4)

Revised: Theorem 2.

Revised: E x , y ∼ (cid:101) P [ − ln P θ u ( y | x , L )] ≥ E x , y ∼ (cid:101) P [ − ln P θ c ( y | x , L )]

Revised: In this case, we compare the distributions P θ u ( y | x , L ) and P θ c ( y | x , L ) .

Revised: We will demonstrate that the former cannot achieve a lower NLL against the data distribution than the latter.

Revised: Proof.

Revised: This follows directly from our deﬁnitions, as we deﬁne θ c to minimize the NLL of

Revised: P θ ( y | x , L ) against the data distribution.

Revised: Thus, P θ u ( y | x , L ) could never yield a lower NLL than

Revised: P θ c ( y | x , L ) , as that would contradict our deﬁnition of θ c .

Original:  In this appendix, we present a construction of the RegCCRF as a weighted ﬁnite-state transducer with weight sharing.
Revised: (cid:4) C C ONSTRUCTION AS WEIGHTED FST In this appendix, we present a construction of the RegCCRF as a weighted ﬁnite-state transducer with weight sharing.

Original: We do this by ﬁrst specifying the transducer topology used, and then specifying how edge weights are parameterized in terms of θ .
Revised: We do this by ﬁrst specifying the transducer topology used, and then specifying how edge weights are parameterized in terms of θ .

Original: The resulting transducer yields an identical distribution to that of the CRF-based construction, P θ ( y | x , L ) .
Revised: The resulting transducer yields an identical distribution to that of the CRF-based construction, P θ ( y | x , L ) .

Original: B. T RANSDUCER TOPOLOGY
Revised: C. T RANSDUCER TOPOLOGY

Original: Starting from L , we deﬁne ¨ L to be the regular language of bigram sequences for the strings in L , i.e.,
Revised: Starting from L , we deﬁne ¨ L to be the regular language of bigram sequences for the strings in L , i.e.,

Original: [Equation]
Revised: [Equation]

Original: ) with $ acting as a end-of-string symbol.
Revised: (12) with $ acting as a end-of-string symbol.

Original: We let ¨ M be an unambiguous FSA for the language ¨ L , and choose to interpret this automaton as a ﬁnite-state transducer by stipulating that all edges should accept any symbol in the input language (but only one symbol per transition, and without allowing epsilon transitions).
Revised: We let ¨ M be an unambiguous FSA for the language ¨ L , and choose to interpret this automaton as a ﬁnite-state transducer by stipulating that all edges should accept any symbol in the input language (but only one symbol per transition, and without allowing epsilon transitions).

Original: This unweighted transducer will be used as the topology for our weighted ﬁnitestate transducer.
Revised: This unweighted transducer will be used as the topology for our weighted ﬁnitestate transducer.

Original: B. E DGE WEIGHTS
Revised: C. E DGE WEIGHTS

Original: In line with Rastogi et al. (2016), we would like to assign weights to the edges of our transducer ¨ M with a neural network.
Revised: In line with Rastogi et al. (2016), we would like to assign weights to the edges of our transducer ¨ M with a neural network.

Original: In order to obtain the same distribution as from our CRF-based construction, these weights must be parameterized in terms of our transition function g θ and emission function h θ .
Revised: In order to obtain the same distribution as from our CRF-based construction, these weights must be parameterized in terms of our transition function g θ and emission function h θ .

Original: For each edge in ¨ M , the weight depends only on the emitted bigram, the input sequence, and the index of the current input symbol – the weight does not depend on the FST states.
Revised: For each edge in ¨ M , the weight depends only on the emitted bigram, the input sequence, and the index of the current input symbol – the weight does not depend on the FST states.

Original: For a symbol bigram ( a, b ) , input sequence x , and index i , the edge weight is equal to
Revised: For a symbol bigram ( a, b ) , input sequence x , and index i , the edge weight is equal to

Original: [Equation]
Revised: [Equation]

Original: Each string in L corresponds bijectively to exactly one bigram sequence in ¨ L , which corresponds bijectively to exactly one accepting path in ¨ M – this path’s weight (in the Log semiring) is equal to the unscaled probability produced by our CRF construction, and so the weighted FST, interpreted as a probability distribution, yields the distribution P θ ( y | x , L ) . 
Revised: Each string in L corresponds bijectively to exactly one bigram sequence in ¨ L , which corresponds bijectively to exactly one accepting path in ¨ M – this path’s weight (in the Log semiring) is equal to the unscaled probability produced by our CRF construction, and so the weighted FST, interpreted as a probability distribution, yields the distribution P θ ( y | x , L ) . D A UTOMATON CONSTRUCTION FOR SEMANTIC ROLE LABELING

Revised: In this appendix, we describe how we generate the automaton architecture for our semantic role labeling experiments.

Revised: While our experiments used 5 core-roles, 17 non-core roles, and one continuation role, we discuss here a generalized setting with arbitrary sets of core, noncore, and continuations of core roles.

Revised: Algorithm 1 provides pseudocode for our construction.

Revised: The core idea is to use subsets of core roles as NFA states, so that we can keep track of which core roles have already ocurred.

Revised: Additional states are used in order to ensure all strings are valid BIO sequences.

Revised: Data: Sets R core , R noncore , and R continuation , of core, noncore, and continuation roles, respectively

Revised: Result:

Revised: A ﬁnite-state automaton M = (Σ , Q, q 1 , F, E ) , parameterized as described in SectionΣ ← { O UTSIDE } ∪ ( { B EGIN , I NSIDE } ×

Revised: ( R core ∪ R noncore ∪ R continuation )) ;

Revised: Q ← ∅ ; q 1 ← ∅ ;

Revised: F ← ∅ ;

Revised: E ← ∅ ; for p ∈ 2 R core do

Revised: Q ← Q ∪ { p } ;

Revised: F ← F ∪ { p } ;

Revised: E ← E ∪ { ( p, O UTSIDE , p ) } ; for r ∈ R noncore do s ← ( r, p ) ;

Revised: Q ← Q ∪ { s } ;

Revised: E ← E ∪ { ( p, ( B EGIN , r ) , p ) , ( p, ( B EGIN , r ) , s ) } ;

Revised: E ← E ∪ { ( r, ( I NSIDE , r ) , s ) , ( r, ( I NSIDE , r ) , p ) } ; end for r ∈ R continuation do if The core role corresponding to r is in p then s ← ( r, p ) ;

Revised: Q ← Q ∪ { s } ;

Revised: E ← E ∪ { ( p, ( B EGIN , r ) , p ) , ( p, ( B EGIN , r ) , s ) } ;

Revised: E ← E ∪ { ( r, ( I NSIDE , r ) , s ) , ( r, ( I NSIDE , r ) , p ) } ; end endfor r ∈

Revised: ( R core \ p ) do s ← ( r, p ) ; t ← p ∪ { r } ;

Revised: Q ← Q ∪ { s } ;

Revised: E ← E ∪ { ( p, ( B EGIN , r ) , s ) , ( p, ( B EGIN , r ) , t ) } ;

Revised: E ← E ∪ { ( s, ( I NSIDE , r ) , s ) , ( s, ( I NSIDE , r ) , t ) } ; end endreturn (Σ , Q, q 1 , F, E ) from given sets of core, noncore, and continuation roles.

Revised: To represent BIO labels, we use tuples of the form ( B EGIN , <roleType> ) for B labels, tuples of the form ( I NSIDE , <roleType> ) for I labels, and the symbol O UTSIDE for the sole O label.