Instructions: In this task, you are given a part of an article. Your task is to generate headline (title) for this text. Preferred headlines are under fifteen words.
Input: We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF 1231216. 1 ar X iv :1 61 0. 06 16 0v 1 [ cs .L G ] 1 9 O ct 2 01 6 Approach FF & FC FF & Conv Rec & FC Rec & Conv Online Learning Small Batch All Combined Original Batch Normalization(BN) 3 3 7 7 7 Suboptimal 7 Time-specific BN 3 3 Limited Limited 7 Suboptimal 7 Layer Normalization 3 7* 3 7* 3 3 7* Streaming Normalization 3 3 3 3 3 3 3 Table 1: An overview of normalization techiques for different tasks. 3: works well. 7: does not work well. FF: Feedforward. Rec: Recurrent. FC: Fully-connected. Conv: convolutional. Limited: time-specific BN requires recording normalization statistics for each timestep and thus may not generalize to novel sequence length. *Layer normalization does not fail on these tasks but perform significantly worse than the best approaches.
Output:
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning