Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer