Don’t Pour Cereal into Coffee: Differentiable Temporal Logic for Temporal Action Segmentation
Abstract
We propose Differentiable Temporal Logic (DTL), a model-agnostic framework that introduces temporal constraints to deep networks.
DTL treats the outputs of a network as a truth assignment of a temporal logic formula, and computes a temporal logic loss reflecting the consistency between the output and the constraints.
We propose a comprehensive set of constraints, which are implicit in data annotations, and incorporate them with deep networks via DTL.
We evaluate the effectiveness of
DTL on the temporal action segmentation task and observe improved performance and reduced logical errors in the output of different task models.
Furthermore, we provide an extensive analysis to visualize the desirable effects of DTL.
[Figure]
Introduction
Recent years have witnessed significant advances in video action analysis tasks such as action recognition [29, 58], action detection [52], and temporal action segmentation [15, 1].
This can be credited to the availability of large-scale datasets [50, 23] and the development of effective deep visual backbones [4, 53, 17].
Although data-driven backbones can capture useful spatio-temporal features, learning a large number of highly diverse temporal dependencies and correlations over long time spans can be very challenging.
In existing approaches, these temporal dependencies are not explicit to the model: it is possible to provide framewise annotation during training, however, temporal constraints like “event X has to occur after event Y ” are still implicit in the annotations and not explicitly enforced.
Even though these constraints could be statistically learned from a large amount of data, a pure vision model could still be confused by different actions with similar visual appearances.
An example of this is shown in Fig.
1, where the model confuses pour_cereal with pour_milk because both actions involve holding a carton.
In this paper, we propose a solution to this problem by employing temporal logic to apply declarative temporal constraints to the output of deep networks.
Linear Temporal Logic (LTL) [45], for example, defines operators that describe necessities, possibilities, and dependencies of actions in a series of actions.
Checking a network’s output against those constraints (a.k.a. model checking) tells us if the predicted actions are logically correct.
Logical correctness can be used as an additional training objective.
According to prior works in the neuro-symbolic community [57, 55, 18], applying logic constraints improves the performance of deep networks in tasks for graph data [57] and images [55] by reducing logical errors in the output.
Although one would anticipate similar benefits when using the temporal logic for action analysis, most of the focus in literature is still on non-sequential data.
In this work, we focus on the temporal aspect and consider temporal constraints in videos.
Inspired by the foundational works in logic-based losses [57, 18], we propose a Differentiable Temporal Logic (DTL) framework, which uses an extended definition of linear temporal logic to constrain the output of action analysis models.
At a high level, DTL treats the model output as a truth assignment of variables in LTL formulae.
A differentiable evaluator performs model checking on the outputs and yields a satisfaction score, which measures the consistency between the constraints and the outputs.
As we optimize the satisfaction score through standard optimization methods, the constraints are enforced on the deep network.
Different from existing work [56], in DTL the relation between formula evaluation and formula satisfaction is determined, which means that DTL is logically sound.
Moreover, we propose a comprehensive set of constraints covering both temporal and non-temporal dependencies between actions, and show how they are represented using DTL.
We evaluate DTL on the challenging temporal action segmentation task, where modeling temporal dependencies between actions is crucial.
The efficacy of our method is shown by the improved performance of different task models when constraints are enforced through DTL.
The contributions of this work are as follows:
• We present Differentiable Temporal Logic (DTL), a framework providing a model-agnostic manner of introducing temporal logic constraints to deep networks (cf.
Section 3.2).
• We propose DTL constraints to describe a wide range of relations between actions, which can be automatically procured from dataset annotation (cf.
Section 3.3).
• Experiments with different task models on the temporal action segmentation task show the efficacy of DTL.
In addition, we provide an extensive study to show the effect of DTL constraints on task models.
Related Works 2.1 Temporal Action Segmentation
A temporal action segmentation model takes a set of video frames as input and predict the action category for each frame.
The model needs to capture both short and long term dependencies between action categories.
Earlier methods [3, 16] use sliding windows to model changes in visual appearance in a short time.
Long-term dependencies are captured from those short-term information by sequential models like hidden Markov model [34] and recurrent networks [35, 47].
Despite the remarkable performance, when these models are used to process long videos they still face forgetting issues and heavy computation burden.
Temporal Convolutional Network (TCN) [37, 38, 15, 39] enables efficient modelling of temporal dependency in variable time spans with its flexible receptive field.
Many advancements are made based on MSTCN [15], which used stacked dilated temporal convolution layers for temporal modelling.
For example, ASRF [28] uses a boundary regression task on top of MSTCN to improve the quality of segmentation.
Huang et al.
[26] proposed a graph-based dependency model to improve the modeling of relations between actions.
Finally, Transformer [54] is introduced to this task and shows superior performance in [59].
Our view is that these methods can be complemented by DTL, since many declarative constraints are difficult to learn from the data.
On the other hand, DTL can explicitly enforce these constraints on the task model during training.
[Figure]
2.2 Temporal Logic and Action Analysis
Temporal logic is a large family of logic systems that specify the relations between timed events.
Linear temporal logic (LTL) [45] is one of the earliest temporal logic.
It is a propositional modal logic that states necessity and possibility about events in a linear discrete succession of time steps.
Later variants of LTL include Metric Temporal Logic (MTL)
[41], which introduces time units as extra parameters to temporal operators, and Signal Temporal Logic [14], which specifies temporal properties for continuous signals.
Allen’s interval algebra [2] and Interval Temporal Logic (ITL)
[22] specify the relations between intervals and their composition.
Temporal logic has been used to specify constraints for program verification [45], robot control [13, 27, 46], and linguistics [42].
However, its application in action analysis remains scarce.
CASE E [21] uses ITL as a structural language to describe atomic actions for event classification.
T-LEAF [56] is the first (and so far the only, to the best of our knowledge) attempt to apply temporal logic to action analysis.
It uses an embedding model to incorporate LTL formulae when training deep networks for action classification, which has little guarantee on logical soundness and only handles a small set of constraints.
DTL adopts LTL for temporal constraints because LTL is: (1) more feasible than its successors with more operators and (2) expressive enough to benefit action analysis.
In principle, DTL can be used with any logic system that is sound and whose formulae can be evaluated in a differentiable manner.
2.3 Logic Constraints on Neural Networks
Applying symbolic constraints to deep networks has been drawing increasing interest [25, 11, 40, 48, 20], yet it remains an open problem.
An important challenge is that logic propositions generally take discrete truth values, while end-to-end differentiability is usually required to optimize a network.
A workaround for this challenge is to find a continuous consistency measurement between network outputs and constraints.
Existing methods can be categorized into two branches based on whether constraints are treated as data or procedures.
The first branch is embedding-based [36, 30, 55, 56, 43], which treats constraints and network outputs as data and project them into a continuous space, where the distance between the two is used as a measurement.
For example, LENSR [55] represents logical constraints in Deterministic Decomposable Negation Normal Form (d-DNNF)
[8] and uses
GCN [32] as the embedder.
The second branch is procedure-based [57, 18, 19, 12, 24], which compiles constraints into computational procedures over network outputs and uses the result as consistency measurement.
For example, semantic loss [57] compiles formulae into logic circuits [9], along which the probability of the output satisfying the formula is computed as the measurement.
We design DTL as a procedure-based framework instead of an embedding-based one, since the latter generally lacks a soundness guarantee: high similarities in the embedding space does not guarantee logical correctness (or satisfaction).
Method
We consider the action segmentation task in a video clip with L frames and N candidate actions
A = { a 1 , . . . , a N } .
As shown in Fig.
2, we assume that there is a deep task model g parameterized by Θ , which takes frames x 0 , . . .
, x L − 1 as input, and outputs ˆ y a 1 , . . .
, ˆ y a N , where ˆ y a i ∈ R L is the unnormalized score for the presence of action a i throughout the L frames.
We assume ˆ y a i ,t >indicates a i presents at time t , and ˆ y a i ,t ≤ 0 indicates the opposite.
Apart from the ground truth label y a 1 , . .
, y a N , there is a set of constraints that describe the relation between actions.
The design goal of DTL is to incorporate these constraints into the task model g , so that ˆ y complies with both the ground truth y and the constraints.
In order to do so, we introduce a temporal logic representation Ψ , and an evaluator f that enforces the constraints in formula ψ ∈ Ψ on g , through its outputs ˆ y .
3.1 Syntax of Formulae
The definition of Ψ is an extension of Linear Temporal Logic (LTL) [45].
A formula ψ ∈ Ψ takes any of the forms separated by “ | ” below:
[Equation]
where a ∈ A is the atomic proposition, and ψ 1 , ψ 2 ∈ Ψ .
The connectives X ( NEXT ), F ( EVENTUAL ),
W ( WEAK_UNTIL ), and S ( SINCE ) are modal operators that form the temporal relations between propositions.
In our context, atomic propositions a represents action a .
Note that the definition in Eqn.
(A1) is recursive.
For example, if a 1 ∈ Ψ , then X . . .
XF a 1 ∈ Ψ .
Semantically, X ψ is satisfied when proposition ψ is satisfied in the next time step.
F ψ is satisfied when ψ is satisfied by the end of the sequence.
ψ 1 W ψ 2 being satisfied means that ψ 1 must always be satisfied until ψ 2 is satisfied (and ψ 2 might never be satisfied).
ψ 1 S ψ 2 being satisfied means ψ 1 is always satisfied after ψ 2 is satisfied.
Section D.1 provides a more formal definition of these operators.
3.2 Formula Evaluator
A formula is said to be satisfied by a truth value assignment if the assignment is semantically compliant with the constraints.
In the context of this paper, each atomic proposition a i is assigned ˆ y a i .
Satisfiability is determined by evaluation, which is a function f of a formula ψ and ˆ y :
[Equation]
where × refers to the Cartesian product.
The two arguments of f are formula ψ and model output
ˆ y a 1 : a N .
Parameter t ∈ [0 , L − 1] is the start time of the evaluation.
For example, t = 2 means that the evaluation is between ψ and the prediction starting from the third frame, i.e. ˆ y a 1 : a N , 2: L − 1 .
The result of f is a satisfaction score that measures the consistency between ˆ y and ψ .
Eqn.
(2) is abstract and must be detailed for all possible forms of a formula ψ can take in Eqn.
(A1).
We aim to expand the definition so that Ψ is logically sound, i.e. there is a determined relation between f t ( ψ, ˆ y ) and the satisfaction of ψ given ˆ y .
Specifically, we would like f t ( ψ, ˆ y ) > 0 to imply that ψ is satisfied by ˆ y at time t .
Moreover, Ψ must be differentiable to be incorporated with task models.
Towards these goals, we first define the evaluation for constants and atomic propositions:
[Equation]
Indeed, the evaluation result for True and False will always be positive and negative, because the former is always satisfied and the latter is never satisfied.
Note that if ψ = a , it is satisfied at time t when ˆ y a,t > 0 , i.e. action a happens at time t .
Next, we define the evaluation for operators “ ¬ ” (negation), “ ∧ ” (logical and), and “ ∨ ” (logical or):
[Equation]
where γ is a parameter of function min γ { x 1: L − 1 } = − 1 γ
ln (cid:80) L − 1 i =1 e − γx i , which approximates the minimum value of { x 1: L − 1 } [6], and max γ { x 1: L − 1 } = − min γ {− x 1: L − 1 } .
It can be shown [6] pour_water (a 1 ) must after take_cup (a 2 ).
Declarative Constraints 𝜓 = ¬𝑎 !
𝑊𝑎 "
Temporal Logic Formula
DAG 𝑎 !
¬ 𝑊 𝑎 " 𝑎 !
¬ 𝑊 𝑎 " … !
𝑦 ! ! !
𝑦 ! "
𝑓 ":$%& 𝑎 & , !
𝑦 = !
𝑦 ! !
,":$%& = 𝑓 ":$%& 𝑎 ( , !
𝑦 = !
𝑦 ! " ,":$%& = 𝑓 ":$%& ¬𝑎 & , !
𝑦 ! !
= −𝑓 ":$%& 𝑎 & , !
𝑦 ! !
…𝐿− … #𝑦 … … 𝑓 " ¬𝑎 & 𝑊𝑎 ( , !
𝑦 = min ) 𝑓 ":* ¬𝑎 & , !
𝑦 ! !
= min ) 𝑓 ":( ¬𝑎 & , !
𝑦 ! !
= min ) =
(False) =
𝑘 = 2 is the min.
value s.t.
𝑓 * 𝑎 ( , !
𝑦
=
True 1 …     (1) (2) (3) (4) (5)
that lim γ →∞ min γ { x 1: L − 1 } =
min { x 1: L − 1 } .
In Eqn.
(4), the operator ¬ flips the sign of f t ( ψ 1 , ˆ y ) , reflecting the negation semantics of ¬ .
In Eqn.
(5), min γ { f t ( ψ 1 , ˆ y ) , f t ( ψ 2 , ˆ y ) } will be negative ( False ) if ψ 1 or ψ 2 or both are False , and will be positive ( True ) iff.
both ψ 1 and ψ 2 are True .
This is consistent with the semantics of ∧ .
The same rationale applies to max γ for ∨ in Eqn.
Finally, we define the modal operators X, F, W, and S that are unique in Ψ .
Intuitively, evaluating
X ψ 1 is equivalent to evaluating ψ 1 at the next time step.
F ψ 1 means there is at least one time step at which ψ 1 is satisfied.
ψ 1 W ψ 2 and ψ 1 S ψ 2 require ψ 1 to be always satisfied in the time period specified by ψ 2 .
We formally define them as follows:
[Equation] [Equation]
t
[Equation]
ψ 1 , ˆ y ) } , where k ≥ t is the min.
integer s.t.
f k ( ψ
[Equation]
, ˆ y ) >
[Equation] [Equation]
y ) } , where k ≥ t is the min.
integer s
[Equation]
t
[Equation]
In Eqn.
(9), min γ { f t : k ( ψ 1 , ˆ y ) } is positive ( True ) iff.
all elements of { f t : k ( ψ 1 , ˆ y ) } are positive, which means that ψ 1 stays True from time t to time k .
This is consistent with the semantics of W. We use min γ for S in Eqn.
(10) for the same reason.
The definitions in Eqn.
(3)-(10) provide a soundness guarantee for Ψ , allowing us to use f t ( ψ, ˆ y ) >as an optimization objective to enforce the constraints in ψ on a task model.
Formally, when γ → ∞ , the approximated evaluation (because of min γ ) becomes exact, and the following theorem is true by construction (a proof is provided in Section D.2):
Theorem 1.
(Soundness) With ψ ∈ Ψ , γ → ∞ : if f t ( ψ, ˆ y ) > 0 , then ψ is satisfied by ˆ y at time t .
In essence, the evaluation process propagates network predictions from atomic propositions to logic operators and finally to the formula.
Equivalently, we can represent a formula as a directed acyclic graph (DAG) where each leaf node is labeled with True , False , or action a ; and each internal node is labeled with logic operators.
The edges of the DAG point from child nodes to their parents, along which the truth value propagates following Eqn.
Fig.
3 illustrates how a constraint “a person cannot pour water before taking a cup” is converted to a DAG and evaluated.
3.3 Constraints
This section discusses different types of constraints we find useful for action analysis and the way they are represented using Ψ .
A constraint can be categorized into either a temporal or a non-temporal constraint based on whether it states the possibility and necessity of an action in a specific time period.
We propose two temporal constraints, namely Backward Dependency (BD) and Forward
Cancellation (FC), and two non-temporal constraints, namely Implication (Ip), and Exclusivity (Ex).
We also introduce how constraints of these types are curated from the training annotations.
Backward Dependency (BD)
It is important to specify the proper order of actions because it usually determines the semantics of an activity.
An order is generally related to temporal dependency:
An action cannot be performed if its prerequisite actions did not occur.
A way to describe this dependency is “one action must occur before another”.
We write this constraint as
[Equation]
where ( X → Y ) means “ X implies Y ” and is equivalent to ( ¬ X ∨ Y ) .
This logic expression means the following: if a i and a j occurs in the same video, a i must not occur until a j occurs.
Set B contains action pairs that satisfy this constraint.
In practice, ( a 1 , a 2 ) ∈ B if a 2 always occurs before a 1 for any co-occurrence of a 1 and a 2 in the annotation.
Forward Cancellation (FC)
Apart from backward dependence, actions can make some other actions impossible in all future time steps.
For example, in a video of salad preparation, the action serve_salad_to_plate marks the end of the preparation.
Once this action occurs, actions like cut_lettuce or cut_tomato should not occur thereafter.
We write this constraint as
[Equation]
which reads “if a i and a j occur in the same video, a i cannot occur after the occurrence of a j ”
Set F contains action pairs that satisfy this constraint.
Implication (Ip) There could be some actions that are semantically dependent but temporally independent: these actions are necessary to complete an activity, but the order is not crucial.
For example, in a video on juice preparation, the person must take_cup and peel_orange , but these two actions are not temporally correlated.
We write this constraint as
[Equation]
F a i → F a j ) .
Set I contains action pairs that satisfy this constraint.
In practice, ( a 1 , a 2 ) ∈ I if a 2 always occurs if a 1 occurs in the annotation.
Exclusivity (Ex) As opposed to what implication constraints describe, some actions are mutually exclusive: they will never co-occur in the same video.
For example, if we know that a video contains a single activity that is “coffee” or “frying eggs”, then pour_coffee cannot happen if put_egg_into_plate occurs at any time in the video.
This constraint is written as
[Equation]
Set X contains pairs of actions that never occur in the same video.
Finally, we connect all constraints using ∧ as ψ = ψ BD ∧ ψ FC ∧ ψ Ip ∧ ψ Ex .
3.4 Training with Constraints
During training, we hope that the output ˆ y is constrained by both ground truth y and logic constraints described by ψ .
The constraints from ground truth are enforced by a task-specific task loss L T (ˆ y , y ) , for example, framewise cross-entropy loss for the temporal action segmentation task.
For logic constraints, we treat ˆ y as an assignment of ψ and from Theorem 1 we know f t ( ψ, ˆ y ) >
if ˆ y satisfies ψ from time t .
Therefore, for any g we can minimize the following objective:
[Equation]
(11) where L T is the loss term for the target task, λ is the weight of the loss of logic L TL , and σ ( x ) =
log(1 + e − x ) penalizes negative evaluation results.
We set t = 0 in f t ( ψ, ˆ y ) in our experiments, since we require the prediction to satisfy the constraints from the first frame.
Note that it is possible to set t to different values so that the constraints can be applied flexibly at different temporal locations.
Experiments
In this section, we first assess the proposed DTL using the temporal action segmentation task.
Through this task, we show that DTL can provide dependency information to improve the performance of action analysis models, in both quantitative and qualitative manners.
Then, we perform an ablation study to show the effects of different types of constraints.
Finally, with a gradient-based analysis, we explain how DTL affects the task model.
All experiments are run with PyTorch 1.10 on an NVIDIA
A6000 GPU.
More details are covered in the appendix.
[Table] [Table]
[Table]
Temporal Action Segmentation
Datasets We use 50Salads [51] and Breakfast [33] for this task.
In these datasets, each video spans at least 200 seconds and contains at least five different actions.
50Salads contains 50 videos of salad preparation with frame-level action annotations of 19 actions.
We generated a total of 313 constraints from its annotation.
Breakfast consists of 1,712 videos with 18 video-level activities, and each frame has one of the 47 actions.
We use only its action-level annotations for this task.
There are a total of 2,145 constraints for this dataset.
The details of the constraints are provided in Section E.
Task Models We use three task models in evaluation: a single-layer bidirectional Gated Recurrent
Unit (GRU) [5], a temporal convolution model MSTCN [15], and a transformer model ASFormer [59].
We use all three task models to examine the performance gain brought by DTL.
For the ablation study and further discussion, we use GRU and MSTCN for their ease of training.
The architecture of GRU is detailed in the appendix.
For experiments on MSTCN and ASFormer, we retrain the corresponding models using their released source codes.
The task models are trained and assessed using the protocol in [15], where the inputs to the task models are the 2048-dimension features extracted using I3D [4] pre-trained on ImageNet [10].
Frame-wise cross-entropy is used as the task loss for all the task models.
Levenshtein distance (Edit), F1 score with thresholds 0.1, 0.25, and 0.5 (F1@{10,25,50}), and frame-wise accuracy (Acc) are used to measure the quality of the outputs.
The results are from a k -fold cross-validation, where k
= 5 for 50Salads and k = 4 for Breakfast.
Performance Table 1 and 2 show the performance of task models trained without (Base) and with (Base + DTL) temporal logic objective L TL , and the performance difference (Gain), on 50Salads and Breakfast respectively.
For completeness, we also include the base performance published in the original papers where applicable.
A prominent observation is that the performance gain is
There are no exclusivity constraints in 50Salads.
[Table]
[Figure]
4.2 Ablation Study
To understand the effects of each constraint type, we perform an ablation study using GRU and
MSTCN on both datasets.
In each experiment, we apply only one type of rules and retrain the taskmodel.
The results are shown in Table 3.
The first observation is that while applying constraintshelps the task model, the extent of improvement differs for different model on different datasets.
On 50Salads, nearly all the action classes presents in each individual video samples, making Ipconstraints less helpful for GRU.
However, Ip is the most helpful constraint type for MSTCN, whichindicates that MSTCN is comparably weaker in modelling action co-occurrence and benefit morefrom this constraint.
On the other hand, the more complicated nature of Breakfast makes both modelsbenefit from all four types of constraints.
Another important observation is that more constraintsdoes not guarantee better performance.
For example, GRU enjoys the highest improvement withonly FC constraints — higher than the improvement when BD and Ip are added.
This reveals atradeoff between precision and completeness of constraints: because the min γ and max γ functionsused in Equation (5)-(10) are approximations, evaluation becomes less accurate when the number ofconstraints increases.
For MSTCN on 50Salads and both models on Breakfast dataset, the benefit ofmore complete constraints outweighs the increased evaluation errors, resulting more improvementswhen using all constraints.
4.3 Qualitative Results
We qualitatively compare predictions from task models trained with and without DTL in
Fig.
In Fig.
4a, we show the output of MSTCN on 50Salads.
Note that the baseline
MSTCN predicts a mix_dressing action after add_dressing , which is in conflict with a FC rule
FC ( add_dressing , mix_dressing ) because dressing cannot be mixed after it has been added to thesalad.
A similar example about GRU on Breakfast is shown in Fig.
4b, where the action pour_coffeeerroneously presents in a video for tea preparation.
The baseline output violates an exclusivity rule
F add_teabag → ¬ F pour_coffee because except special cases coffee should not be added to tea.
Both errors are fixed in the same models trained with DTL.
Moreover, we notice that correcting anindividual wrong segment improves the quality of the whole output sequence as the output near thefixed segment are also corrected.
This is because the task models we consider are: (a) recurrentmodels that depend current output on past inputs, or (b) convolutional/transformer models that predictbased on receptive field/attention information.
In either case, correcting an error fixes the cascadedeffect it has on the model states, which improves the quality of all outputs nearby.
4.4 Understand the Effects of DTL
One of the biggest benefit of DTL, as discussed in Section 1, is that it provides additional supervisorysignals for temporal constraints which are implicit in training data.
We would like to understand howthose signals affect the task model.
Formally, given a model g parameterized by Θ and an input x ,we would like to know if a constraint ψ has encouraged or suppressed the model’s output about actiona at time step t .
This is possible by assuming that the output of the model is continuous with respectto its parameters, or lim ∥ δ Θ ∥→
g Θ + δ Θ ( x ) =
g Θ ( x ) .
With this assumption, the effect of ψ on Θ 268 can be approximated by the changes in the output as we update Θ based on f 0 ( ψ, ˆ y ) .
Specifically, 269 we first calculate δ ψ Θ = ∂σ ( f 0 ( ψ, ˆ y )) /∂
Θ , where σ ( x ) = log(1 +
e − x ) as in Equation (11).
Then, we perform a step of gradient descend and get an updated model as Θ ′ =
Θ − γδ ψ Θ , where 271 γ = 10 − 4 is a small update step.
Finally, the output difference caused by ψ with input x is obtained as 272 ∆ ψ, x = g Θ ′ ( x ) − g Θ ( x ) , where ∆ ψ, x ∈ R N × L .
A positive (negative) ∆ ψ, x a,t indicates a promotive 273 (suppressive) effect of ψ on ˆ y a,t as it increases (decreases) the score for action a at time t .
[Figure]
consistently positive.
This indicates the applicability and efficacy of DTL on task models with different architectures and on datasets of different scales.
Another observation is that the gains on different metrics are not even: there are noticeable improvements on Levenshtein distance and F scores while the improvement on frame-wise accuracy is not as significant.
We conjecture the reason to be that frame-wise accuracy treats frames independently and does not distinguish well between models with and without the over-segmentation problem.
Since DTL constrains the relations between segments of actions instead of frames, metrics based on segment differences like Levenshtein distance and F1-scores are therefore comparably better reflections of performance improvement.
4.2 Ablation Study
To understand the effects of each constraint type, we perform an ablation study using GRU and
MSTCN on both datasets.
In each experiment, we apply only one type of constraint and retrain the task model.
The results are shown in Table 3.
The first observation is that while applying constraints helps the task model, the extent of improvement differs for different models on different datasets.
On 50Salads, nearly all the action classes are present in each video sample, making Ip constraints less helpful for GRU.
However, Ip is the most helpful constraint type for MSTCN, which indicates that MSTCN is comparably weaker in modelling action co-occurrence and benefits more from this constraint.
On the other hand, the more complicated nature of Breakfast makes both models benefit from all four types of constraints.
Another important observation is that more constraints do not guarantee better performance.
For example, GRU enjoys the highest improvement with only FC constraints — higher than when BD and Ip are added.
This reveals a trade-off between precision and completeness of constraints: Because the min γ and max γ functions used in Eqn.
(5)-(10) are approximations, the evaluation becomes less accurate when the number of constraints increases.
For MSTCN on 50Salads and both models on Breakfast, the benefit of more complete constraints outweighs the increased evaluation errors, resulting in more improvements when using all constraints.
Understanding the fine-grained interactions between the constraints and different backbone models remains an open question.
4.3 Qualitative Results
We qualitatively compare predictions from task models trained with and without DTL in Fig.
In Fig.
4a, we show the output of MSTCN on 50Salads.
Note that the baseline MSTCN
··· (Preceding actions omitted.)
mix_dressing peel_cucumber cut_cucumber mix_ingredients serve_salad_onto_plate add_dressing ··· (Succeeding actions omitted.)
Implication Backward Dependency
BD( add_addressing , mix_dressing ) predicts a mix_dressing action after add_dressing , which is in conflict with a FC constraint
FC ( add_dressing , mix_dressing ) because dressing cannot be mixed after it has been added to the salad.
A similar example of GRU on Breakfast is shown in Fig.
4b, where the action pour_coffee is erroneously predicted in a video for “tea preparation”.
The baseline output violates an exclusivity constraint F add_teabag → ¬ F pour_coffee because, except in special cases, coffee should not be added to tea.
Both errors are fixed in the models trained with DTL.
Moreover, we notice that correcting an individual wrong segment improves the quality of the whole output sequence.
This is because the task models we consider are: (a) recurrent models that condition current output on past inputs, or (b) convolutional/transformer models that predict based on receptive field/attention information.
In either case, correcting an error fixes the cascade effect it has on the model states, which improves the quality of all nearby outputs.
4.4 Analyzing the Effects of DTL
One of the greatest benefits of DTL is that it provides additional supervisory signals for temporal constraints, which are implicit in training data.
We would like to understand how those signals affect the task model.
Formally, given a model g parameterized by Θ and an input x , we would like to know if a constraint ψ i promotes or suppresses the model’s output about action a
at time t .
We assume that the output of the model is continuous with respect to its parameters, or lim ∥ δ Θ ∥→
g Θ + δ Θ ( x ) =
g Θ ( x ) .
With this assumption, the effect of ψ i on Θ can be approximated by the changes in the output as we update Θ based on f 0 ( ψ i , ˆ y ) .
Specifically, we first compute δ ψ i Θ = ∂σ ( f 0 ( ψ i , ˆ y )) /∂
Θ , where σ ( x ) = log(1 +
e − x ) as in Eqn.
Then, we update Θ as Θ ′ =
Θ − γδ ψ i Θ , where γ = 10 − 4 is a small update step.
Finally, the difference in output caused by ψ i is obtained as ∆ ψ i , x = g Θ ′ ( x ) − g Θ ( x ) , where ∆ ψ i , x ∈ R N × L .
A positive (negative) ∆ ψ i , x a,t indicates a promotive (suppressive) effect of ψ i on ˆ y a,t as it increases (decreases) the score for action a at time t .
We use ∆ ψ i , x to pinpoint the constraint that makes a model predict an action a at time t .
This can be done by calculating ∆ ψ i , x a,t for all ψ i in ψ and find the most positively influential constraint as arg max ψ i ∆ ψ i , x a,t .
For example, Fig. 5 shows a snippet of prediction from MSTCN on 50Salads and the constraints we found contributed the most to each predicted segment.
One important observation from this is that actions in earlier time steps are generally promoted by later actions, as the former could serve as the prerequisites of the latter.
We can also summarize the total effect of ψ i on all time steps and samples as ∆ ψ i a =
∆ i , x a,t .
∆ ψ i a provides two pieces of information.
The first is the effect of ψ i on all actions.
As illustrated as a heatmap in Fig. 6, ∆ ψ i a suggest that the Ip constraints play mainly promotional roles, where the FC, BD, and Ex constraints are mostly suppressive.
This is consistent with our design intuitions:
Ip encourages co-occurrence of actions, FC and Ex constraints are intrinsically prohibitive, and BD constraints suppress an action if its prerequisites are missing.
The second information is how an action a is affected by other actions.
This can be determined by selecting the most promotive and suppressive constraints for action a , and checking the actions involved in these constraints.
Table 4 shows the relations between example actions in the Breakfast
[Table]
dataset.
An interesting observation is that for put_ToppingOnTop the most promotive and suppressive action is the same put_PutBunTogether , meaning that relation between two actions could largely change depending on the context.
This also indicates that DTL can help the model learn different dependencies between the same pair of actions.
Conclusion
We propose DTL, a framework that uses temporal logic to constrain the training of action analysis models.
Experimental results on the action segmentation task show that DTL effectively improves the performance of task models with different architectures.
An ablation study reveals the divergent effect of different types of rules on different task models.
Our work suggests that temporal constraints can be explicitly provided to a deep network and reduce logical errors in its output.
The source code of our work is accessible at https://diff-tl.github.io/ .
Limitations There remain some limitations in this work.
First, in this work, we only explored a subset of temporal knowledge that is expressible as frequency matrices.
When such temporal correlation is sparse, DTL is less effective.
This calls for more flexible knowledge curation methods (e.g. with human involvement).
Besides, non-temporal knowledge about actions, such as object affordance, can be exploited to handle more complicated actions like the verb-object compositions in Epic-Kitchens [7].
Second, in terms of knowledge type, DTL can benefit from more expressive logic languages, like Allen’s Interval Algebra [2], which supports first-order temporal constraints beyond necessities and possibilities.
Moreover, DTL formulae are evaluated in their raw forms, which means that the evaluation efficiency could be further improved by better formula compilation [9] and optimized message passing on directed acyclic graphs.
In addition, the constraints we collected from dataset annotations only form a partial and clean view of real-world scenarios.
The real-world constraints are more comprehensive and complex, which poses a higher requirement on generalization and robustness against uncertainties
Acknowledgments and Disclosure of Funding
This research is partially supported by the National Research Foundation, Singapore under its
Strategic Capability Research Centres Funding Initiative.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of
National Research Foundation, Singapore.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] While Theorem 1 is true by construction, we provide a sketchy proof in Section D.2.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Training details are mostly provided in Section 4 while some extra information is in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] With an exception that error bars are omitted for clarity in Table 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We used publically available 50Salads and Breakfast dataset, the original papers have been cited.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] The code and data are publicly available and the consent has been described in the cited papers.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
No such information was found in the dataset we use.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
The table below shows the notations grouped by the modules.
[Table]
B Implementation Details
Task Models.
The implementation for MSTCN [15] and ASFormer [59] are from existing opensource code provided by corresponding authors.
The GRU is implemented as follows: a fullyconnected layer first transforms the 2048-dimension input into a 512-dimension vector.
The vector is then sent to a bi-directional Gated Recurrent Unit layer with a hidden size of 512, which yields a 1024-dimension vector.
The vector is finally transformed by a second fully-connected layer into an
N -dimension vector, where each dimension represents the un-normalized score of an action class.
Hyperparameters.
There are two hyperparameters in DTL: γ and λ .
γ is the parameter for min γ () and max γ () , and is set to 1 .
λ is the weight for temporal logic loss.
We set λ = 0 .
1 for both GRU and MSTCN, and λ = 0 .
005 for ASFormer.
All hyperparameters are determined empirically.
Training.
All the experiments are performed on an NVIDIA A6000 GPU with PyTorch 1.10.
The training for MSTCN and ASFormer follows the way defined in the corresponding source code.
We use Adam [31] to optimize the GRU with a learning rate of 5 × 10 − 4 for 50 epochs.
The model with the best validation performance is reported.
C Class-wise Performances
We provide class-wise performance (measured by F1@25) comparison between task models trained with and without DTL.
The result is shown in Fig.
A1 and Fig.
A2.
A general observation is that actions with lower performance on task models without DTL benefit more from DTL.
We conjecture that actions with higher performance reflect the part of knowledge which is better learned by the a d d _ p e p p
e
r a d d _ o il p l a c e _
c u c u m b e r _ i n t o _ b
o w l a d d _ s a l t p l a c e _ t o m a t
o _ i n t o _
b o w l c u t _ c u c u m b
e
r a d d _ v i n e g a r m i x _ i n g r e d i e n t s p l a c e _ l e tt u
c
e
_ i n t o _ b o w l a d d _ d r e s
s i n
g c u t _ c
h e e s e c u t _ t o m a t
o c u t _ l e tt u c e p l a c e _
c
h e e s e _ i n t o _ b o w l m i x _
d r e
s s i n g s e r v e _ s a l a d _ o n t
o _
p l a t e p e e l _ c u c u m b e r a c t i o n _ e n d a c t i o n _ s t a r t
[Figure]
Avg.
Improvement
[Figure]
c u t _ f r u i t c r a c k _ e gg b u tt e r _ p a n f r y _ e gg a dd _ s a l t n p e pp e r s il p u t _ f r u
i t 2 b o w l p ee l _ f r u i t t a k e _ p l a t e s t
i r f r y _ e gg
f r y _ p a n c a k e p o u r _ m il k s p oo n _ f l o u r p o u r _ o il s p oo n _ p o w d e r s t i r _ d o u g h p
o u r _ d o u g h 2 p a n p u t _ e gg 2 p l a t e p u t _ t
o
pp i n g o n t
o p t a k e _
b o w l
s m e a r _ b u tt e r t a k e _
c
u p p
o u r _ c
o ff ee a dd _ t e a b a g t a k e _ k n i f e s
q u ee z e _
o r a n g e s t i r _ e gg p o u r _ c e r e a l s p o u r _ e gg 2 p a n p u t _ p a n c a k e 2 p l a t e s t
i r _ m il k p u t _ b un t
o g e t h e r c u t _ b un p
o u r _ j u
i
c
e p o u r _ w a t e r p o u r _ f l o u r t a k e _ g l a ss s p oo n _ s u g a r s t
i r _
c
e r e a l s c u t _
o r a n g e t a k e _ t
o pp i n g s t
i r _ f r u
i t t a k e _ s q u ee z e r p
o u r _
s u g a r s t
i r _ c o ff ee t a k e _ e gg s t
a k e _ b u tt e r s t i r _ t e a
Avg.
Improvement: 0.2 Avg.
Improvement:
[Figure]
task model from the annotation.
Therefore, DTL is less effective for this subset of actions because currently its constraints are also procured from annotation.
On the other hand, this shows that DTL is able to help the model better learn constraints from annotation.
We anticipate more performance improvement with more general constraints that go beyond knowledge in the annotations in future works.
D Extended Discussion on DTL
Section 3.1 introduced logic operators and briefly introduced their semantics.
In Section 3.2, we introduced the evaluation of those operators in detail.
In this section, we extend the discussion on
DTL and answer the following two questions:
What is the formal definition to the semantics of DTL operators (cf.
Section D.1)?
What is the relation between the evaluation of DTL operators related and their semantics (cf.
Section D.2)?
D.1 Semantics of Logic Operators
This section formally defines the semantics of these logic operators, which is largely an extension of linear temporal logic.
It is recommended to compare the semantics defined in Eqn.
(A2)-(A9) below with the evaluation defined in Eqn.
Also recall that ψ ∈ Ψ takes the following forms:
[Equation]
Before defining the semantics, we first introduce a symbol ω , which is a truth assignment of ψ in L time steps.
ω is an L -long word ω 0:
L − 1 = ω 0 ω 1 . . .
ω
L − 1 , where ω t is the set of atomic propositions that are True at time t b .
We use ω t 1 : t 2 to denote a substring of ω from time t 1 to time t 2 and ω t 1 : as a shorthand for ω t 1 : L − 1 .
ω t : | = ψ means ω t : satisfies ψ under the semantics of Ψ .
Now we start to define the semantics of Ψ .
The semantics of constants are straight-forward:
[Equation]
which states that any truth assignment satisfies ψ if ψ = True , and no truth assignment will satisfy ψ if ψ = False .
If ψ = a , then a satisfying assignment must assign True to a (make a occur) at time 0.
For common propositional logic operators ¬ ( NEGATION ), ∧ ( AND ) and ∨ ( OR ), we define their semantics as
[Equation]
where ψ 1 , ψ 2 ∈ Ψ .
Finally, we define the semantics for modal operators X , F , W, and S: [Equation] t
[Equation]
Intuitively, X shifts the time to the next time step.
F states the necessity of ψ 1 .
Both W and S state the possibility of ψ 1 during some time intervals specified by the occurrence of ψ 2 .
D.2 A Proof of Soundness for DTL
Theorem 1 in the main paper states the soundness of DTL: if f t ( ψ, ˆ y ) > 0 , then ˆ y satisfies the constraints in formula ψ at time t .
In other words, it states that evaluated satisfaction entails semantic satisfaction .
In this section, we provide a sketchy proof for Theorem 1.
This proof starts with a basic assumption, where we formally establish the connection between ˆ y and a truth assignment ω :
[Equation]
We also have the following assumption that rules out value “0” throughout the evaluation.
This is necessary as − 0 =
can break the evaluation of logical negation.
Assumption A2.
[Equation]
Then we can prove the following Theorem A1, which directly leads to Theorem 1.
Theorem A1.
With ψ ∈ Ψ , γ → ∞ : f t ( ψ, ˆ y ) > 0 ⇔ ω t : | = ψ .
Proof.
Let ψ, a, ψ 1 , ψ 2 ∈ Ψ , where a is an atomic proposition.
We need to prove Theorem 1 for all forms in Eqn.
(A1).
We do this by induction.
Base Case When ψ
= a .
[Equation]
Therefore the base case is true.
[Table]
[Equation]
Case 1:
[Table]
• If f t ( ψ, ˆ y ) > 0 : f
[Equation]
ψ
[Equation]
ˆ y
[Equation]
• If ω t : | = ψ : ω t : | = ψ
[Table]
= ¬
[Table]
[Equation]
[Table]
=
[Equation]
[Table]
[Equation]
=
[Table] [Table]
[Table]
and
[Table]
[Equation]
[Table] [Table]
[Equation]
Case 3: ψ = ( ψ 1 ∨ ψ 2 ) .
The proof is similar to Case 2.
Case 4: ψ = X ψ 1 .
[Equation]
Case 5: ψ = F ψ 1 .
[Equation]
Case 6: ψ = ( ψ 1 W ψ 2 ) .
• If ∃ k ∈ [ t, L ) s.t. f k
( ψ 2 , ˆ y ) > 0 :
f t ( ψ, ˆ y ) > 0 ψ =( ψ 1 W ψ 2 ) ⇐ ==
==
===
⇒
f t ( ψ 1 W ψ 2 , ˆ y ) >Eqn.
(9) ⇐ ==
===
⇒ min { f k ′ :
t ( ψ 1 , ˆ y ) > 0 } , and k ′ ∈ [ t, L ) is the min.
integer s.t.
f k ′ ( ψ 2 , ˆ y ) >
⇐⇒ ∃ t 1 ∈ [ t, k ] s.t.
f t 1 ( ψ 1 , ˆ y ) > 0 and ∃ k ∈ [ t, L ) s.t.
f k ( ψ 2 , ˆ y ) >
I.H.
⇐ = ⇒ ∃ t 1 ∈ [ t, k ] s.t. ω t 1 : | = ψ 1 and ∃ k ∈ [ t, L ) s.t. ω
k : | = ψ
Eqn.
[Equation]
A8)
[Equation]
• If ∀ k ∈ [ t, L ) , f k ( ψ 2 , ˆ y ) < 0 :
This is a special case for W (and S as well).
When this happens, we directly set f t ( ψ, ˆ y ) =
min γ ( f t : L − 1 ( ψ 1 , ˆ y )) .
Then f t ( ψ, ˆ y ) > 0 ⇔ ω t : | = ψ 1 [-> 
 ⇐⇒-]⇔ ω t : | = ( ψ 1 W ψ 2 ) .
Case 7: ψ = ( ψ 1 S ψ 2 ) .
The proof is similar to Case 6.
E Constraints
This section provides a quick view of the constraints used in our experiment.
We first explain how constraints are collected and then provide samples for different types of constraints collected for the two datasets used in our experiment.
E.1 Collecting Constraints
The constraints are automatically generated from the existing annotations of datasets.
Algorithm A shows how statistics about co-occurrences between actions can be collected.
Then Algorithm A uses those statistics to generate the four types of constraints discussed in Section 3.3.
Algorithm A1:
Algorithm to collect the co-occurrence statistics from dataset annotation.
Input: Set of samples M = { m 1 , m 2 , . . .
, m M } , where m i =
[ y 0 , y 2 , ..., y L − 1 ] and y i ∈ A is the index of one of the N actions
▷ B [ a i , a j ] is the frequency of a i occurring before a j ▷ P [ a i , a j ] is the frequency of a i occurring after a j ▷ J [ a i , a j ] is the number of videos where a i occurs with a j ▷ C [ a i , a j ] is the number of videos where a i occurs 1 B , P ,
J ← zero matrices of size N × N ; vector of size N ; 3 foreach do of
of
N
N ;
., y L −
annotation of m ; foreach
do if
do
E.2 Samples of Constraints
For clarity, we show 10 entries for each type of constraint.
E.2.1 Breakfast
Breakfast contains 48 actions for ten different activities about breakfast preparation.
Each video contains a single activity, which could be making coffee, cereal, tea, fried egg, pancake, sandwich,
2 foreach i ∈ { 0 , 1 , . . .
, N − 1 } do foreach j ∈ { 0 , 1 , . . .
, N − 1 } do if i ̸
Constraints ← append_BD i juice, etc.
Breakfast is therefore different from 50Salads because its actions could be mutually exclusive.
Below is a list of actions:
• take_cup • pour_coffee • pour_milk • pour_sugar • stir_coffee •
spoon_sugar • add_teabag • pour_water • stir_tea • cut_bun • smear_butter • put_toppingOnTop • put_bunTogether • take_plate • take_knife • take_butter • take_topping • cut_orange • squeeze_orange •
take_glass • pour_juice • take_squeezer • take_bowl • pour_cereals • stir_cereals • spoon_powder • stir_milk • pour_oil • take_eggs • crack_egg • add_saltnpepper • fry_egg • put_egg2plate • butter_pan • cut_fruit • put_fruit2bowl • peel_fruit • stir_fruit • stirfry_egg • stir_egg • pour_egg2pan • spoon_flour • stir_dough • pour_dough2pan • fry_pancake • put_pancake2plate • pour_flour • SIL
Backward Dependency
The following shows a subset of the back dependency constraints, where
BD ( a i , a j ) =
( F a i ∧ F a j ) → ( ¬ a i W a j ) reads “action a i is backward dependent on action a j .”
• BD( pour_dough2pan , spoon_flour ), • BD( squeeze_orange , cut_orange ), • BD( add_saltnpepper , take_bowl ), • BD( put_egg2plate , pour_egg2pan ), • BD( stir_coffee , pour_coffee ), • BD( butter_pan , take_eggs ), • BD( pour_dough2pan , butter_pan ), • BD( stir_tea , take_cup ), • BD( pour_milk , pour_coffee ), • BD( pour_juice , take_plate ).
Forward Cancellation
The following shows a subset of the forward cancellation constraints, where
FC ( a i , a j ) =
( F a i ∧ F a j ) → ( ¬ a j S a i ) reads “action a i cancels the future occurrence of a j .”
• FC
pour_dough2pan , pour_flour ), • FC( take_squeezer , take_knife ), • FC( fry_pancake , pour_dough2pan ), • FC( put_buntogether , take_topping ), • FC( pour_dough2pan , take_bowl ), • FC( put_toppingontop , cut_bun ), • FC
stir_fruit , peel_fruit ), • FC
( pour_milk , pour_coffee ), • FC( put_buntogether , take_knife ), • FC
( stir_coffee , take_cup ).
Implication The following shows a subset of the implication constraints, where Ip ( a i , a j ) =
F a i → F a j reads “action a i implies the occurrence of a j .”
• Ip( fry_pancake , pour_dough2pan ), • Ip( pour_flour , stir_dough ), • Ip( butter_pan , crack_egg ), • Ip( stir_dough , pour_milk ), • Ip( pour_cereals , pour_milk ), • Ip( add_teabag , pour_water ), • Ip( spoon_powder , pour_milk ), • Ip( take_topping , cut_bun ),
• Ip( take_butter , smear_butter ), • Ip( stirfry_egg , crack_egg ).
Exclusivity The following shows a subset of the exclusivity constraints, where Ex ( a i , a j ) =
F a i → ¬ F a j reads “if action a i occurs, action a j will not occur in the same video”.
• Ex( butter_pan , cut_bun ), • Ex( smear_butter , spoon_sugar ), • Ex
( take_butter , pour_sugar ), •
Ex( stir_cereals , cut_bun ), • Ex( add_teabag , take_butter ), • Ex( take_butter , stir_cereals ), • Ex( put_fruit2bowl , put_pancake2plate ), • Ex( put_fruit2bowl , pour_sugar ), • Ex( cut_bun , spoon_sugar ), • Ex( spoon_flour , add_teabag ).
E.2.2 50Salads
50Salads contains 19 actions for a single activity “making salad”:
• cut_tomato • place_tomato_into_bowl • cut_cheese • place_cheese_into_bowl • cut_lettuce • place_lettuce_into_bowl • add_salt • add_vinegar • add_oil • add_pepper • mix_dressing • peel_cucumber • cut_cucumber • place_cucumber_into_bowl • add_dressing • mix_ingredients • serve_salad_onto_plate • action_start • action_end
Backward Dependency
• BD( serve_salad_onto_plate , peel_cucumber ), • BD( serve_salad_onto_plate , place_tomato_into_bowl ), • BD( add_dressing , cut_tomato ), • BD( serve_salad_onto_plate , cut_cucumber ), • BD( serve_salad_onto_plate , cut_lettuce ), • BD( add_dressing , add_pepper ), • BD( serve_salad_onto_plate , mix_ingredients ), • BD( serve_salad_onto_plate , add_salt ), • BD( serve_salad_onto_plate , place_cucumber_into_bowl ), • BD( add_dressing , cut_cheese ).
[Table]
• FC( add_dressing , peel_cucumber ), • FC( serve_salad_onto_plate , add_vinegar ), • FC( place_cheese_into_bowl , cut_cheese ), • FC( mix_ingredients , cut_cheese ), • FC( add_dressing , add_oil ), • FC( place_cucumber_into_bowl , peel_cucumber ), • FC( serve_salad_onto_plate , mix_ingredients ), • FC( serve_salad_onto_plate , cut_cheese ), • FC( serve_salad_onto_plate , peel_cucumber ), • FC( add_dressing , place_cucumber_into_bowl ).
Implication
• Ip( place_cheese_into_bowl , action_end ), • Ip( cut_cheese , cut_lettuce ), • Ip( add_oil , place_tomato_into_bowl ), • Ip( place_tomato_into_bowl , cut_tomato ), • Ip( mix_dressing , cut_cheese ), • Ip( add_salt , add_pepper ), • Ip( place_lettuce_into_bowl , add_oil ), • Ip( add_salt , place_tomato_into_bowl ), • Ip( place_tomato_into_bowl , action_end ), • Ip( add_
[Table]
Exclusivity There are no exclusivity constraints because all actions are from the same activity.
F Action Detection on Charades
[Table]
One might be curious about whether DTL works without the “one-action-per-frame” assumption used in the main paper.
In fact, this assumption emerges from the definition of the action segmentation task rather than DTL itself.
To examine the applicability of DTL without this assumption, we assess DTL using the action detection task on the Charades [49] dataset.
Different from the datasets used in the action segmentation task, every frame in Charades can be labeled with zero or multiple actions.
The dataset contains 9,848 videos with 157 frame-level action categories.
Each video spans about 30 seconds and contains six actions on average.
Following the same procedure introduced in Section 3.3, we collected a total of 9,668 constraints.
We sampled 2,000 constraints for this experiment.
Following the procedures in [44], we use I3D [4] pretrained on Kinetics-400 to extract the frame-level features and mean average precision (mAP) as the performance metric.
DTL is assessed using two task models, namely, a GRU with 512 hidden units (abbreviated as GRU), and a three-layer temporal convolutional network (abbreviated as TConv).
As shown in Table A2, improvements are observed for both task models when they are trained with DTL.
Note that since we aim to show that DTL works across different tasks, we do not use constraints unique to the action detection task.
For example, we do not collect constraints about co-occurrences of actions in the same frame.
This means further improvement is possible when those absent constraints are collected and enforced.
The definition of a more comprehensive set of constraints and their applications in various action analysis tasks are left as future works.