Don’t Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation
Abstract
We propose Differentiable Temporal Logic (DTL), a model-agnostic framework that introduces temporal constraints to deep networks.
DTL treats the outputs of a network as a truth assignment of a temporal logic formula, and computes a temporal logic loss reﬂecting the consistency between the output and the constraints.
We propose a comprehensive set of constraints, which are implicit in data annotations, and incorporate them with deep networks via DTL.
We evaluate the effectiveness of DTL on the temporal action segmentation task and observe improved performance and reduced logical errors in the output of different task models.
Furthermore, we provide an extensive analysis to visualize the desirable effects of DTL.
[Figure]
Introduction
Recent years have witnessed signiﬁcant advances in video action analysis tasks such as action recognition [29, 55], action detection [49], and temporal action segmentation [15, 1].
This can be credited to the availability of large-scale datasets [47, 23] and the development of effective deep visual backbones [4, 50, 17].
Although data-driven backbones can capture useful spatio-temporal features, learning a large number of highly diverse temporal dependencies and correlations over long time spans can be very challenging.
In existing approaches, these temporal dependencies are not explicit to the model: it is possible to provide framewise annotation during training, however, temporal constraints like “event X has to occur after event Y ” are still implicit in the annotations and not explicitly enforced.
Even though these constraints could be statistically learned from a large amount of data, a pure vision model could still be confused by different actions with similar visual appearances.
An example of this is shown in Fig.
1, where the model confuses pour_cereal with pour_milk because both actions involve holding a carton.
In this paper, we propose a solution to this problem by employing temporal logic to apply declarative temporal constraints to the output of deep networks.
Linear Temporal Logic (LTL) [43], for example, deﬁnes operators that describe necessities, possibilities, and dependencies of actions in a series of actions.
Checking a network’s output against those constraints (a.k.a. model checking) tells us if the predicted actions are logically correct.
Logical correctness can be used as an additional training objective.
According to prior works in the neuro-symbolic community [54, 52, 18], applying logic constraints improves the performance of deep networks in tasks for graph data [54] and images [52] by reducing logical errors in the output.
Although one would anticipate similar beneﬁts when using the temporal logic for action analysis, most of the focus in literature is still on non-sequential data.
In this work, we focus on the temporal aspect and consider temporal constraints in videos.
Inspired by the foundational works in logic-based losses [54, 18], we propose a Differentiable Temporal Logic (DTL) framework, which uses an extended deﬁnition of linear temporal logic to constrain the output of action analysis models.
At a high level, DTL treats the model output as a truth assignment of variables in LTL formulae.
A differentiable evaluator performs model checking on the outputs and yields a satisfaction score, which measures the consistency between the constraints and the outputs.
As we optimize the satisfaction score through standard optimization methods, the constraints are enforced on the deep network.
Different from existing work [53], in DTL the relation between formula evaluation and formula satisfaction is deterministic, which means that DTL is logically sound.
Moreover, we propose a comprehensive set of constraints covering both temporal and nontemporal dependencies between actions, and show how they are represented using DTL.
We evaluate DTL on the challenging temporal action segmentation task, where modeling temporal dependencies between actions is crucial.
The efﬁcacy of our method is shown by the improved performance of different task models when constraints are enforced through DTL.
The contributions of this work are as follows:
• We present Differentiable Temporal Logic (DTL), a framework providing a model-agnostic manner of introducing temporal logic constraints to deep networks (cf.
Section 3.2).
• We propose DTL constraints to describe a wide range of relations between actions, which can be automatically procured from dataset annotation (cf.
Section 3.3).
• Experiments with different task models on the temporal action segmentation task show the efﬁcacy of DTL.
In addition, we provide an extensive study to show the effect of DTL constraints on task models.
Related Works 2.1 Temporal Action Segmentation
A temporal action segmentation model takes a set of video frames as input and predict the action category for each frame.
The model needs to capture both short and long term dependencies between action categories.
Earlier methods [3, 16] use sliding windows to model changes in visual appearance in a short time.
Long-term dependencies are captured from those short-term information by sequential models like hidden Markov model [33] and recurrent networks [34, 45].
Despite the remarkable performance, when these models are used to process long videos they still face forgetting issues and heavy computation burden.
Temporal Convolutional Network (TCN) [36, 37, 15, 38] enables efﬁcient modelling of temporal dependency in variable time spans with its ﬂexible receptive ﬁeld.
Many advancements are made based on MSTCN [15], which used stacked dilated temporal convolution layers for temporal modelling.
For example, ASRF [28] uses a boundary regression task on top of MSTCN to improve the quality of segmentation.
Huang et al.
[26] proposed a graph-based dependency model to improve the modeling of relations between actions.
Finally, Transformer [51] is introduced to this task and shows superior performance in [56].
Our view is that these methods can be complemented by DTL, since many declarative constraints are difﬁcult to learn from the data.
On the other hand, DTL can explicitly enforce these constraints on the task model during training.
[Figure]
[Figure]
2.2 Temporal Logic and Action Analysis
Temporal logic is a large family of logic systems that specify the relations between timed events.
Linear temporal logic (LTL) [43] is one of the earliest temporal logic.
It is a propositional modal logic that states the necessity and possibility about events in a linear discrete succession of time steps.
Later variants of LTL include Metric Temporal Logic (MTL) [40], which introduces time units as extra parameters to temporal operators, and Signal Temporal Logic [14], which speciﬁes temporal properties for continuous signals.
Allen’s interval algebra [2] and Interval Temporal Logic (ITL)
[22] specify the relations between intervals and their composition.
Temporal logic has been used to specify constraints for program veriﬁcation [43], robot control [13, 27, 44], and linguistics [41].
However, its application in action analysis remains scarce.
CASE E [21] uses ITL as a structural language to describe atomic actions for event classiﬁcation.
T-LEAF [53] is the ﬁrst (and so far the only, to the best of our knowledge) attempt to apply temporal logic to action analysis.
It uses an embedding model to incorporate LTL formulae when training deep networks for action classiﬁcation, which has little guarantee on logical soundness and only handles a small set of constraints.
DTL employs LTL for temporal constraints because LTL is: (1) more feasible than its successors with more operators and (2) expressive enough to beneﬁt action analysis.
In principle, DTL can be used with any logic system that is sound and whose formulae can be evaluated in a differentiable manner.
2.3 Logic Constraints on Neural Networks
Applying symbolic constraints to deep networks has been drawing increasing interest [25, 11, 39, 46, 20], yet it remains an open problem.
An important challenge is that logic propositions generally take discrete truth values, while end-to-end differentiability is usually required to optimize a network.
A workaround for this challenge is to ﬁnd a continuous consistency measurement between network outputs and constraints.
Existing methods can be categorized into two branches based on whether constraints are treated as data or procedures.
The ﬁrst branch is embedding-based [35, 30, 52, 53, 42], which treats constraints and network outputs as data and project them into a continuous space, where the distance between the two is used as a measurement.
For example, LENSR [52] represents logical constraints in Deterministic Decomposable Negation Normal Form (d-DNNF)
[8] and uses
GCN [31] as the embedder.
The second branch is procedure-based [54, 18, 19, 12, 24], which compiles constraints into computational procedures over network outputs and uses the result as consistency measurement.
For example, semantic loss [54] compiles formulae into logic circuits [9], along which the probability of the output satisfying the formula is computed as the measurement.
We design DTL as a procedure-based framework instead of an embedding-based one, since the latter generally lacks a soundness guarantee: high similarities in the embedding space does not guarantee logical correctness (or satisfaction).
Method
We consider the action segmentation task in a video clip with L frames and N candidate actions
A = { a 1 , . . . , a N } .
As shown in Fig.
2, we assume that there is a deep task model g parameterized by Θ , which takes frames x 0 , . . .
, x L − 1 as input, and outputs ˆ y a 1 , . . .
, ˆ y a N , where ˆ y a i ∈ R L is the unnormalized score for the presence of action a i throughout the L frames.
We assume ˆ y a i ,t >indicates a i presents at time t , and ˆ y a i ,t ≤ 0 indicates the opposite.
Apart from the ground truth label y a 1 , . .
, y a N , there is a set of constraints that describe the relation between actions.
The design goal of DTL is to incorporate these constraints into the task model g , so that ˆ y complies with both the ground truth y and the constraints.
In order to do so, we introduce a temporal logic representation Ψ , and an evaluator f that enforces the constraints in formula ψ ∈ Ψ on g , through its outputs ˆ y .
3.1 Syntax of Formulae
The deﬁnition of Ψ is an extension of Linear Temporal Logic (LTL) [43].
A formula ψ ∈ Ψ takes any of the forms separated by “ | ” below:
[Equation]
where a ∈ A is the atomic proposition, and ψ 1 , ψ 2 ∈ Ψ .
The connectives X ( NEXT ), F ( EVENTUAL ),
W ( WEAK_UNTIL ), and S ( SINCE ) are modal operators that form the temporal relations between propositions.
In our context, atomic propositions a represents action a .
Note that the deﬁnition in Eqn.
(1) is recursive.
For example, if a 1 ∈ Ψ , then X . . .
XF a 1 ∈ Ψ .
Semantically, X ψ is satisﬁed when proposition ψ is satisﬁed in the next time step.
F ψ is satisﬁed when ψ is satisﬁed by the end of the sequence.
ψ 1 W ψ 2 being satisﬁed means that ψ 1 must always be satisﬁed until ψ 2 is satisﬁed (and ψ 2 might never be satisﬁed).
ψ 1 S ψ 2 being satisﬁed means ψ 1 is always satisﬁed after ψ 2 is satisﬁed.
Section D.1 provides a more formal deﬁnition of these operators.
3.2 Formula Evaluator
A formula is said to be satisﬁed by a truth value assignment if the assignment is semantically compliant with the constraints.
In the context of this paper, each atomic proposition a i is assigned ˆ y a i .
Satisﬁability is determined by evaluation, which is a function f of a formula ψ and ˆ y :
[Equation]
where × refers to the Cartesian product.
The two arguments of f are formula ψ and model output
ˆ y a 1 : a N .
Parameter t ∈ [0 , L − 1] is the start time of the evaluation.
For example, t = 2 means that the evaluation is between ψ and the prediction starting from the third frame, i.e. ˆ y a 1 : a N , 2: L − 1 .
The result of f is a satisfaction score that measures the consistency between ˆ y and ψ .
Eqn.
(2) is abstract and must be detailed for all possible forms of a formula ψ can take in Eqn.
We aim to expand the deﬁnition so that Ψ is logically sound, i.e. there is a determined relation between f t ( ψ, ˆ y ) and the satisfaction of ψ given ˆ y .
Speciﬁcally, we would like f t ( ψ, ˆ y ) > 0 to imply that ψ is satisﬁed by ˆ y at time t .
Moreover, Ψ must be differentiable to be incorporated with task models.
Towards these goals, we ﬁrst deﬁne the evaluation for constants and atomic propositions:
[Equation]
Indeed, the evaluation result for True and False will always be positive and negative, because the former is always satisﬁed and the latter is never satisﬁed.
Note that if ψ = a , it is satisﬁed at time t when ˆ y a,t > 0 , i.e. action a happens at time t .
Next, we deﬁne the evaluation for operators “ ¬ ” (negation), “ ∧ ” (logical and), and “ ∨ ” (logical or):
[Equation]
pour_water (a 1 ) must after take_cup (a 2 ).
Declarative Constraints 𝜓 = ¬𝑎 !
𝑊𝑎 "
Temporal Logic Formula
DAG 𝑎 !
¬ 𝑊 𝑎 " 𝑎 !
¬ 𝑊 𝑎 " … !
𝑦 ! ! !
𝑦 ! "
𝑓 ":$%& 𝑎 & , !
𝑦 = !
𝑦 ! !
,":$%& = 𝑓 ":$%& 𝑎 ( , !
𝑦 = !
𝑦 ! " ,":$%& = 𝑓 ":$%& ¬𝑎 & , !
𝑦 ! !
= −𝑓 ":$%& 𝑎 & , !
𝑦 ! !
…𝐿− … #𝑦 … … 𝑓 " ¬𝑎 & 𝑊𝑎 ( , !
𝑦 = min ) 𝑓 ":* ¬𝑎 & , !
𝑦 ! !
= min ) 𝑓 ":( ¬𝑎 & , !
𝑦 ! !
= min ) =
(False) =
[Figure]
is
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
where γ is a parameter of function
[Equation] which approximates the minimum value of { x 1: L − 1 } [6], and max γ { x 1: L − 1 } = − min γ {− x 1: L − 1 } . It can be shown [6] that lim γ →∞ min γ { x 1: L − 1 } = min { x 1: L − 1 } .
In Eqn.
(4), the operator ¬ ﬂips the sign of f t ( ψ 1 , ˆ y ) , reﬂecting the negation semantics of ¬ .
In Eqn.
(5), min γ { f t ( ψ 1 , ˆ y ) , f t ( ψ 2 , ˆ y ) } will be negative ( False ) if ψ 1 or ψ 2 or both are False , and will be positive ( True ) iff.
both ψ 1 and ψ 2 are True .
This is consistent with the semantics of ∧ .
The same rationale applies to max γ for ∨ in Eqn.
Finally, we deﬁne the modal operators X, F, W, and S that are unique in Ψ .
Intuitively, evaluating
X ψ 1 is equivalent to evaluating ψ 1 at the next time step.
F ψ 1 means there is at least one time step at which ψ 1 is satisﬁed.
ψ 1 W ψ 2 and ψ 1 S ψ 2 require ψ 1 to be always satisﬁed in the time period speciﬁed by ψ 2 .
We formally deﬁne them as follows:
[Equation] [Equation]
{ f t [Equation] k ( [Equation]
[Equation]
ˆ y ) } , where k ≥ t is the min.
integer s.t.
f k [Equation] ψ
[Equation]
ˆ y
[Equation]
>
[Equation]
f t ( ψ 1 S ψ
[Equation]
ˆ y )
= min γ { f k : L − 1 ( ψ 1 , ˆ y ) } , where k ≥ t is the min. integer s.t. f k ( ψ
[Equation]
, ˆ y ) > 0 .
In Eqn.
(9), min γ { f t : k ( ψ 1 , ˆ y ) } is positive ( True ) iff.
all elements of { f t : k ( ψ 1 , ˆ y ) } are positive, which means that ψ 1 stays True from time t to time k .
This is consistent with the semantics of W. We use min γ for S in Eqn.
(10) for the same reason.
The deﬁnitions in Eqn.
(3)-(10) provide a soundness guarantee for Ψ , allowing us to use f t ( ψ, ˆ y ) > 0 as an optimization objective to enforce the constraints in ψ on a task model.
Formally, when γ → ∞ , the approximated evaluation (because of min γ ) becomes exact, and the following theorem is true by construction (a proof is provided in Section D.3):
Theorem 1.
(Soundness) With ψ ∈ Ψ , γ → ∞ : if f t ( ψ, ˆ y ) > 0 , then ψ is satisﬁed by ˆ y at time t .
In essence, the evaluation process propagates network predictions from atomic propositions to logic operators and ﬁnally to the formula.
Equivalently, we can represent a formula as a directed acyclic graph (DAG) where each leaf node is labeled with True , False , or action a ; and each internal node is labeled with logic operators.
The edges of the DAG point from child nodes to their parents, along which the truth value propagates following Eqn.
Fig.
3 illustrates how a constraint “a person cannot pour water before taking a cup” is converted to a DAG and evaluated.
3.3 Constraints
This section discusses different types of constraints we ﬁnd useful for action analysis and the way they are represented using Ψ .
A constraint can be categorized into either a temporal or a nontemporal constraint based on whether it states the possibility and necessity of an action in a speciﬁc time period.
We propose two temporal constraints, namely Backward Dependency (BD) and Forward Cancellation (FC), and two non-temporal constraints, namely Implication (Ip), and Exclusivity (Ex).
We also introduce how constraints of these types are curated from the training annotations.
Backward Dependency (BD)
It is important to specify the proper order of actions because it usually determines the semantics of an activity.
An order is generally related to temporal dependency:
An action cannot be performed if its prerequisite actions did not occur.
A way to describe this dependency is “one action must occur before another”.
We write this constraint as
[Equation]
where ( X → Y ) means “ X implies Y ” and is equivalent to ( ¬ X ∨ Y ) .
This logic expression means the following: if a i and a j occurs in the same video, a i must not occur until a j occurs.
Set B contains action pairs that satisfy this constraint.
In practice, ( a 1 , a 2 ) ∈ B if a 2 always occurs before a 1 for any co-occurrence of a 1 and a 2 in the annotation.
Forward Cancellation (FC)
Apart from backward dependence, actions can make some other actions impossible in all future time steps.
For example, in a video of salad preparation, the action serve_salad_to_plate marks the end of the preparation.
Once this action occurs, actions like cut_lettuce or cut_tomato should not occur thereafter.
We write this constraint as
[Equation]
which reads “if a i and a j occur in the same video, a i cannot occur after the occurrence of a j ”
Set F contains action pairs that satisfy this constraint.
Implication (Ip) There could be some actions that are semantically dependent but temporally independent: these actions are necessary to complete an activity, but the order is not crucial.
For example, in a video on juice preparation, the person must take_cup and peel_orange , but these two actions are not temporally correlated.
We write this constraint as
[Equation]
Set I contains action pairs that satisfy this constraint.
In practice, ( a 1 , a 2 ) ∈ I if a 2 always occurs if a 1 occurs in the annotation.
Exclusivity (Ex) As opposed to what implication constraints describe, some actions are mutually exclusive: they will never co-occur in the same video.
For example, if we know that a video contains a single activity that is “coffee” or “frying eggs”, then pour_coffee cannot happen if put_egg_into_plate occurs at any time in the video.
This constraint is written as
[Equation]
Set X contains pairs of actions that never occur in the same video.
Finally, we connect all constraints using ∧ as ψ = ψ BD ∧ ψ FC ∧ ψ Ip ∧ ψ Ex .
3.4 Training with Constraints
During training, we hope that the output ˆ y is constrained by both ground truth y and logic constraints described by ψ .
The constraints from ground truth are enforced by a task-speciﬁc task loss L T (ˆ y , y ) , for example, framewise cross-entropy loss for the temporal action segmentation task.
For logic constraints, we treat ˆ y as an assignment of ψ and from Theorem 1 we know f t ( ψ, ˆ y ) >
if ˆ y satisﬁes ψ from time t .
Therefore, for any g we can minimize the following objective:
[Equation]
[Equation] where L T is the loss term for the target task, λ is the weight of the loss of logic L TL , and σ ( x ) =
log(1+ e − x ) penalizes negative evaluation results.
We set t = 0 in f t ( ψ, ˆ y ) in our experiments, since we require the prediction to satisfy the constraints from the ﬁrst frame.
Note that it is possible to set t to different values so that the constraints can be applied ﬂexibly at different temporal locations. [Table] [Table] [Table]
[Table]
Experiments
In this section [Table] we ﬁrst assess the proposed DTL using the temporal action segmentation task. Through this task
[Table] we show that DTL can provide dependency information to improve the performance of action analysis models, in both quantitative and qualitative manners.
Then, we perform an ablation study to show the effects of different types of constraints.
Finally, with a gradient-based analysis, we explain how DTL affects the task model.
All experiments are run with PyTorch 1.
on an NVIDIA A6000 GPU.
More details are covered in the appendix.
4.1 Temporal Action Segmentation
Datasets We use 50Salads [48] and Breakfast [32] for this task.
In these datasets, each video spans at least 200 seconds and contains at least ﬁve different actions.
50Salads contains 50 videos of salad preparation with frame-level action annotations of 19 actions.
We generated a total ofconstraints from its annotation.
Breakfast consists of 1,712 videos with 18 video-level activities, and each frame has one of the 47 actions.
We use only its action-level annotations for this task.
There are a total of 2,145 constraints for this dataset.
The details of the constraints are provided in Section E.
Task Models We use three task models in evaluation: a single-layer bidirectional Gated Recurrent Unit (GRU) [5], a temporal convolution model MSTCN [15], and a transformer model ASFormer [56].
We use all three task models to examine the performance gain brought by DTL.
For the ablation study and further discussion, we use GRU and MSTCN for their ease of training.
The architecture of GRU is detailed in the appendix.
For experiments on MSTCN and ASFormer, we retrain the corresponding models using their released source codes.
The task models are trained and assessed using the protocol in [15], where the inputs to the task models are the 2048-dimension [Table] features extracted using I3D [4] pre-trained on ImageNet [10].
Frame-wise cross-entropy is used as the task loss for all the task models.
Levenshtein distance (Edit), F1 score with thresholds 0.1, 0.25, and 0.5 (F1@{10,25,50}), and frame-wise accuracy (Acc) are used to measure the quality of the outputs.
The results are from a k -fold cross-validation, where k
= 5 for 50Salads and k = 4 for Breakfast.
Performance Table 1 and 2 show the performance of task models trained without (Base) and with (Base + DTL) temporal logic objective L TL , and the performance difference (Gain), on 50Salads and Breakfast respectively.
For completeness, we also include the base performance published in the original papers where applicable.
A prominent observation is that the performance gain is consistently positive.
This indicates the applicability and efﬁcacy of DTL on task models with different architectures and on datasets of different scales. Another observation is that the gains on different metrics are not even: there are noticeable improvements on Levenshtein distance and F1 scores while the improvement on frame-wise accuracy is not as signiﬁcant. We conjecture the reason to be that frame-wise accuracy treats frames independently and does not distinguish well between models with and without the over-segmentation problem. Since DTL constrains the relations between segments of actions instead of frames, metrics based on segment differences like Levenshtein distance and F1-scores are therefore comparably better reﬂections of performance improvement.
4.2 Ablation Study
To understand the effects of each constraint type, we perform an ablation study using GRU and MSTCN on both datasets. In each experiment, we apply only one type of constraint and retrain the task model. The results are shown in Table 3.
The ﬁrst observation is that while applying constraints helps the task model, the extent of improvement differs for different models on different datasets.
On 50Salads, nearly all the action classes are present in each video sample, making Ip constraints less helpful for GRU.
However, Ip is the most helpful constraint type for MSTCN, which indicates that MSTCN is comparably weaker in modelling action co-occurrence and beneﬁts more from this constraint.
On the other hand, the more complicated nature of Breakfast makes both models beneﬁt from all four types of constraints.
Another important observation is that more constraints do not guarantee better performance.
For example, GRU enjoys the highest improvement with only FC constraints — higher than when BD and Ip are added.
This reveals a trade-off between precision and completeness of constraints: Because the min γ and max γ functions used in Eqn.
(5)-(10) are approximations, the evaluation becomes less accurate when the number of constraints increases.
For
MSTCN on 50Salads and both models on Breakfast, the beneﬁt of more complete constraints outweighs the increased evaluation errors, resulting in more improvements when using all constraints.
Understanding the ﬁne-grained interactions between the constraints and different backbone models remains an open question.
4.3 Qualitative Results
We qualitatively compare predictions from task models trained with and without DTL in Fig.
In Fig.
4a, we show the output of MSTCN on 50Salads.
Note that the baseline MSTCN
[Figure]
4.2 Ablation Study
To understand the effects of each constraint type, we perform an ablation study using GRU and
MSTCN on both datasets.
In each experiment, we apply only one type of rules and retrain the taskmodel.
The results are shown in Table 3.
The first observation is that while applying constraintshelps the task model, the extent of improvement differs for different model on different datasets.
On 50Salads, nearly all the action classes presents in each individual video samples, making Ipconstraints less helpful for GRU.
However, Ip is the most helpful constraint type for MSTCN, whichindicates that MSTCN is comparably weaker in modelling action co-occurrence and benefit morefrom this constraint.
On the other hand, the more complicated nature of Breakfast makes both modelsbenefit from all four types of constraints.
Another important observation is that more constraintsdoes not guarantee better performance.
For example, GRU enjoys the highest improvement withonly FC constraints — higher than the improvement when BD and Ip are added.
This reveals atradeoff between precision and completeness of constraints: because the min γ and max γ functionsused in Equation (5)-(10) are approximations, evaluation becomes less accurate when the number ofconstraints increases.
For MSTCN on 50Salads and both models on Breakfast dataset, the benefit ofmore complete constraints outweighs the increased evaluation errors, resulting more improvementswhen using all constraints.
4.3 Qualitative Results
[Figure]
We qualitatively compare predictions from task models trained with and without DTL in
Fig.
In Fig.
4a, we show the output of MSTCN on 50Salads.
Note that the baseline
MSTCN predicts a mix_dressing action after add_dressing , which is in conflict with a FC rule
FC ( add_dressing , mix_dressing ) because dressing cannot be mixed after it has been added to thesalad.
A similar example about GRU on Breakfast is shown in Fig.
4b, where the action pour_coffeeerroneously presents in a video for tea preparation.
The baseline output violates an exclusivity rule
F add_teabag → ¬ F pour_coffee because except special cases coffee should not be added to tea.[-
-]
Both errors are fixed in the same models trained with DTL.
Moreover, we notice that correcting anindividual wrong segment improves the quality of the whole output sequence as the output near thefixed segment are also corrected.
This is because the task models we consider are: (a) recurrentmodels that depend current output on past inputs, or (b) convolutional/transformer models that predictbased on receptive field/attention information.
In either case, correcting an error fixes the cascadedeffect it has on the model states, which improves the quality of all outputs nearby.
4.4 Understand the Effects of DTL
One of the biggest benefit of DTL, as discussed in Section 1, is that it provides additional supervisorysignals for temporal constraints which are implicit in training data.
We would like to understand howthose signals affect the task model.
Formally, given a model g parameterized by Θ and an input x ,we would like to know if a constraint ψ has encouraged or suppressed the model’s output about actiona at time step t .
This is possible by assuming that the output of the model is continuous with respectto its parameters, or lim ∥ δ Θ ∥→
g Θ + δ Θ ( x ) =
g Θ ( x ) .
With this assumption, the effect of ψ on Θcan be approximated by the changes in the output as we update Θ based on f 0 ( ψ, ˆ y ) .
Specifically,we first calculate δ ψ Θ = ∂σ ( f 0 ( ψ, ˆ y )) /∂
Θ , where σ ( x ) = log(1 +
e − x ) as in Equation (11).
Then, we perform a step of gradient descend and get an updated model as Θ ′ =
Θ − γδ ψ Θ , whereγ = 10 − 4 is a small update step.
Finally, the output difference caused by ψ with input x is obtained as∆ ψ, x = g Θ ′ ( x ) − g Θ ( x ) , where ∆ ψ, x ∈ R N × L .
A positive (negative) ∆ ψ, x a,t indicates a promotive(suppressive) effect of ψ on ˆ y a,t as it increases (decreases) the score for action a at time t .
[Figure]
[Figure]
predicts a mix_dressing action after add_dressing , which is in conﬂict with
FC ( add_dressing , mix_dressing ) because dressing cannot be mixed after it has been added to the salad.
A similar example of GRU on Breakfast is shown in Fig.
4b, where the action pour_coffee is erroneously predicted in a video for “tea preparation”.
The baseline output violates an exclusivity constraint F add_teabag → ¬ F pour_coffee because, except in special cases, coffee should not be added to tea.
Both errors are ﬁxed in the models trained with DTL.
Moreover, we notice that correcting an individual wrong segment improves the quality of the whole output sequence.
This is because the task models we consider are: (a) recurrent models that condition current output on past inputs, or (b) convolutional/transformer models that predict based on receptive ﬁeld/attention information.
In either case, correcting an error ﬁxes the cascade effect it has on the model states, which improves the quality of all nearby outputs.
4.4 Analyzing the Effects of DTL
One of the greatest beneﬁts of DTL is that it provides additional supervisory signals for temporal constraints, which are implicit in training data.
We would like to understand how those signals affect the task model.
Formally, given a model g parameterized by Θ and an input x , we would like to know if a constraint ψ i promotes or suppresses the model’s output about action a
at time t .
We assume that the output of the model is continuous with respect to its parameters, or lim ∥ δ Θ ∥→
g Θ + δ Θ ( x ) =
g Θ ( x ) .
With this assumption, the effect of ψ i on Θ can be approximated by the changes in the output as we update Θ based on f 0 ( ψ i , ˆ y ) .
Speciﬁcally, we ﬁrst compute δ ψ i Θ = ∂σ ( f 0 ( ψ i , ˆ y )) /∂
Θ , where σ ( x ) = log(1 +
e − x ) as in Eqn.
Then, we update Θ as Θ ′ =
Θ − γδ ψ i Θ , where γ = 10 − 4 is a small update step.
Finally, the difference in output caused by ψ i is obtained as ∆ ψ i , x = g Θ ′ ( x ) − g Θ ( x ) , where ∆ ψ i , x ∈ R N × L .
A positive (negative) ∆ ψ i , x a,t indicates a promotive (suppressive) effect of ψ i on ˆ y a,t as it increases (decreases) the score for action a at time t .
We use ∆ ψ i , x to pinpoint the constraint that makes a model predict an action a at time t .
This can be done by calculating ∆ ψ i , x a,t for all ψ i in ψ and ﬁnd the most positively inﬂuential constraint as arg max ψ i ∆ ψ i , x a,t .
For example, Fig. 5 shows a snippet of prediction from MSTCN on 50Salads and the constraints we found contributed the most to each predicted segment.
One important observation from this is that actions in earlier time steps are generally promoted by later actions, as the former could serve as the prerequisites of the latter.
We can also summarize the total effect of ψ i on all time steps and samples as ∆ ψ i a = 1 |X| [Equation] x [Equation]
P [Equation] − [Equation]
=
[Equation]
∆ ψ i , x a,t .
∆ ψ i a provides two pieces of information.
The ﬁrst is the effect of ψ i on all actions.
As illustrated as a heatmap in Fig. 6, ∆ ψ i a suggest that the Ip constraints play mainly promotional roles, where the FC, BD, and Ex constraints are mostly suppressive.
This is consistent with our design intuitions:
Ip encourages co-occurrence of actions, FC and Ex constraints are intrinsically prohibitive, and BD constraints suppress an action if its prerequisites are missing. [Table]
The second information is how an action a is affected by other actions.
This can be determined by selecting the most promotive and suppressive constraints for action a , and checking the actions involved in these constraints.
Table 4 shows the relations between example actions in the Breakfast dataset.
An interesting observation is that for put_ToppingOnTop the most promotive and suppressive action is the same put_PutBunTogether , meaning that relation between two actions could largely change depending on the context.
This also indicates that DTL can help the model learn different dependencies between the same pair of actions.
Conclusion
We propose DTL, a framework that uses temporal logic to constrain the training of action analysis models.
Experimental results on the action segmentation task show that DTL effectively improves the performance of task models with different architectures.
An ablation study reveals the divergent effect of different types of rules on different task models.
Our work suggests that temporal constraints can be explicitly provided to a deep network and reduce logical errors in its output.
The source code of our work is accessible at https://diff-tl.github.io/ .
Limitations There remain some limitations in this work.
First, in this work, we only explored a subset of temporal knowledge that is expressible as frequency matrices.
When such temporal correlation is sparse, DTL is less effective.
This calls for more ﬂexible knowledge curation methods (e.g. with human involvement).
Besides, non-temporal knowledge about actions, such as object affordance, can be exploited to handle more complicated actions like the verb-object compositions in Epic-Kitchens [7].
Second, in terms of knowledge type, DTL can beneﬁt from more expressive logic languages, like Allen’s Interval Algebra [2], which supports ﬁrst-order temporal constraints beyond necessities and possibilities.
Moreover, DTL formulae are evaluated in their raw forms, which means that the evaluation efﬁciency could be further improved by better formula compilation [9] and optimized message passing on directed acyclic graphs.
In addition, the constraints we collected from dataset annotations only form a partial and clean view of real-world scenarios.
The real-world constraints are more comprehensive and complex, which poses a higher requirement on generalization and robustness against uncertainties.
Acknowledgments and Disclosure of Funding
This research is partially supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative.
Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the author(s) and do not reﬂect the views of
National Research Foundation, Singapore.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] While Theorem 1 is true by construction, we provide a sketchy proof in the appendix.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Training details are mostly provided in Section 4 while some extra information is in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] With an exception that error bars are omitted for clarity in Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We used publically available 50Salads and Breakfast dataset, the original papers have been cited.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The code and data are publicly available and the consent has been described in the cited papers.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
No such information was found in the dataset we use.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-This is necessary as − 0 = 
-]
[N/A]