  Neural Arithmetic Logic Modules have become a growing area of interest,
though remain a niche field. These units are small neural networks which aim to
achieve systematic generalisation in learning arithmetic operations such as {+,
-, *, \} while also being interpretive in their weights. This paper is the
first in discussing the current state of progress of this field, explaining key
works, starting with the Neural Arithmetic Logic Unit (NALU). Focusing on the
shortcomings of NALU, we provide an in-depth analysis to reason about design
choices of recent units. A cross-comparison between units is made on experiment
setups and findings, where we highlight inconsistencies in a fundamental
experiment causing the inability to directly compare across papers. We finish
by providing a novel discussion of existing applications for NALU and research
directions requiring further exploration.
