  Temporal receptive fields of models play an important role in action
segmentation. Large receptive fields facilitate the long-term relations among
video clips while small receptive fields help capture the local details.
Existing methods construct models with hand-designed receptive fields in
layers. Can we effectively search for receptive field combinations to replace
hand-designed patterns? To answer this question, we propose to find better
receptive field combinations through a global-to-local search scheme. Our
search scheme exploits both global search to find the coarse combinations and
local search to get the refined receptive field combination patterns further.
The global search finds possible coarse combinations other than human-designed
patterns. On top of the global search, we propose an expectation guided
iterative local search scheme to refine combinations effectively. Our
global-to-local search can be plugged into existing action segmentation methods
to achieve state-of-the-art performance.
