A MoE model built on top of 4 identical phi2 models as experts.