second lstm layer first layer lstm global context memory softmax classifier initialize refine informativeness gate attention .
End-to-End Learning of Action Detection from Frame Glimpses in Videos