Really interesting topic. For about 5 years, I spent much of my hobby time writing a-life sims and messing with them. I wanted an "aquarium" that looked interesting that I could have running 24/7, constantly evolving. I guess I wrote maybe 10 sims from scratch in Unity, Java, Processing, and p5.js.

I have had (limited) success in getting emergent control and coordination behaviours. A couple examples:

I played with a "battleground" simulation, inspired by the "Gladiabots" game, where the entities were NN-based fighting robots organised in teams (4v4, 20v20 etc). Random sensor systems, random fire control systems, signal emitters, etc, hooked up to random neural nets where winning teams procreate and evolve/mutate. Over a LOT of generations (maybe like 20,000+), I'd see teams evolve what looked like coherent tactics, including teamwork, role specialisation, and time-based tactics/formations.

I had a "jellyfish" simulation, similar premise, with small aquatic animals of various "species" who could speciate, etc. The aquarium had qualities like light levels, food production, temperature zones, etc. You always have to be careful to not anthropomorphize too much, but I would clearly see battlegrounds develop and evolve, and successful species rise and fall with behaviours that (again over many thousands of generations) initially look like drunken sailors but evolve into what looks like basic behvaiours, at least at the level you may see in small multicellular organisms.

In my experience, getting emergent behaviours needs a difficult-to-optimize balance between system complexity and system stability. Every neuron, sensor, output, or attribute you add to the sim can cause a super-exponential growth in the "problem space" of your denizens. Behaviours that require any form of coordination take so much longer to evolve than uncoordinated behaviours because you tend to need to "get lucky" where both a signal and response evolve/mutate at the same time.

Another issue is that feed-forward networks are not great at temporal processing. Many basic behaviours require more temporal processing than I expected when I started building these things. I got these to evolve by having features like memory nodes, timer nodes, and recurrent nodes, but if I were to build another one, I'd experiment more with LSTM-like neurons and other recurrent/memory-based architectures.

In my experience (and with my horrendously under-optimised codebases), the amount of evolution you need to simulate is super-exponential as you grow the complexity of the sensors, outputs, and neural nets. In my playing around, doubling any single dimension almost always led to more than a 4x increase in the time you need to see new emergence. In my most complex sim, it got to the point where even tweaking a basic variable like temperature or day/night cycle would need me to run the sim for at least a day to properly observe the impact, and sometimes more like a week or two. I never really knew when the "evolution ceiling" was hit, which was part of the fun for me -- I love seeing a new behavior suddenly appear.

This ALIEN project is really interesting and it has a lot of cool ideas. I haven't played with it yet but I think it's going to be fun! The project I have been keeping an eye on is Neuraquarium, which has similar design goals to what I have built in the past.