Paper:
Sequence-to-Segments Networks for Segment Detection
Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments. To address this problem, we propose the Sequence-toSegments Network (S 2 N), a novel end-to-end sequential encoder-decoder architecture. S 2 N first encodes the input into a sequence of hidden states that progressively capture both local and holistic information. It then employs a novel decoding architecture, called Segment Detection Unit (SDU), that integrates the decoder state and encoder hidden states to detect segments sequentially. During training, we formulate the assignment of predicted segments to ground truth as the bipartite matching problem and use the Earth Mover's Distance to calculate the localization errors. Experiments on temporal action proposal and video summarization show that S 2 N achieves state-of-the-art performance on both tasks.

Review:
The authors describe an architecture to output a series of temporal regions (segments) of interest in a video. There is a simple RNN encoder and an interesting decoder called a "Segment Detection Unit" which points to multiple segment boundaries.  SDU is applied recurrently to the final state of the encoder, each iteration outputs a segment and a confidence score. (see End-to-End people detection, which is referenced [36]) SDU is similar to Pointer Networks in that it points back to an element of the input sequence as its output rather than outputing a regression value or an index to a dictionary. Also they 'improve' on pointer networks by using the encoded state of each input timestep (along with the current network state) (instead of the input at each timestep ) as input to the function that determines the pointer (function g). The encoder is thus able to learn to encode states with features  to help learn g.  So the model encodes the whole sequence, and the final state is input to a recurrent SDU unit, which has outputs for beginning and end. The SDU unit outputs for each output timestep are calculated for each encoded state of the input sequence, via a function g that uses the encoded states of the input sequence and the current decoder state as input) in this way the authors directly score each of the input frames as beginning and end segment positions and take the max of those scores to output a segment. They also learn to output a confidence score for each segment.  They define a loss function that considers localization loss and confidence of predictions. For localization loss, they use the EarthMover distance. Their loss function depends on an assignment of all output segments to all ground truth segments and use the "Hungarian algorithm" to do just that.  They perform reasonable experiments, compare with other methods, and show good performance.  Concern: There is no terminal output state from the deocder RNN.  In the Implementation section (201) it says: "We set the number of proposals generated from each chunk to be 15 which is the largest possible number of ground truth proposals contained in a chunk during training". Does this mean that 15 outputs are always made, but that the confidence score is used as a cutoff for accepting segments? It is not clear to me that this is explained. I am not saying this is bad, it's just not clear.. if my interpretation is true then what is the threshold used? How is it determined?  This is a clear-headed work, combining ideas from recent advances. The choices made appear creative and solid, and the results are good.  Minor points: This is not true (line 75): "However, the current RNNs are still restricted to output each time a “token” of the input sequence", but the following sentence is fine. Grammar 83: 'a 1D "object" or "objects"