no dropout?
#5
by
Wangchunshu
- opened
I find it weird that in the official internlm repo there is actually dropout in the MHA module, but there is not a single dropout module in the modeling_internlm.py file. Neither in Attention nor in MLP.