  Weakly-supervised temporal action localization aims to localize action
instances temporal boundary and identify the corresponding action category with
only video-level labels. Traditional methods mainly focus on foreground and
background frames separation with only a single attention branch and class
activation sequence. However, we argue that apart from the distinctive
foreground and background frames there are plenty of semantically ambiguous
action context frames. It does not make sense to group those context frames to
the same background class since they are semantically related to a specific
action category. Consequently, it is challenging to suppress action context
frames with only a single class activation sequence. To address this issue, in
this paper, we propose an action-context modeling network termed ACM-Net, which
integrates a three-branch attention module to measure the likelihood of each
temporal point being action instance, context, or non-action background,
simultaneously. Then based on the obtained three-branch attention values, we
construct three-branch class activation sequences to represent the action
instances, contexts, and non-action backgrounds, individually. To evaluate the
effectiveness of our ACM-Net, we conduct extensive experiments on two benchmark
datasets, THUMOS-14 and ActivityNet-1.3. The experiments show that our method
can outperform current state-of-the-art methods, and even achieve comparable
performance with fully-supervised methods. Code can be found at
https://github.com/ispc-lab/ACM-Net
