The performance of deep learning in natural language processing has been spectacular, but the reason for this success remains unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a Long Short-Term Memory (LSTM)-based neural language model effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of the reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical law of natural language. This understanding could provide a direction of improvement of architectures of neural networks.
Ref. REF is an empirical study of natural language models and focuses mainly on the "one-point" statistics like Zipf's law.
Do Neural Nets Learn Statistical Laws behind Natural Language?
PloS one
PloS one
Computer Science Medicine