"As you will find out if you start experimenting with the code in this article bloat is a big problem, as well as a successful form taking over the entire population."

A very simple way to fight bloat is to penalize an individual's fitness value in proportion to how long it takes to find that individual's fitness value.

Then, if you bias selection such that fitter individuals are more likely to survive, then the individuals that take a shorter time to evaluate should have a slightly higher chance of surviving; and those individuals should be the very individuals that are less bloated.

Of course, the problem is determining just how much to penalize their fitness values. And, as in much else in evolutionary computation, that generally comes down to simply doing a number of runs with different penalties and see how they compare.

One down side to keep in mind when using any kind of penalty for bloat is that some research has shown that unrestricted tree growth is often required for solving larger and more complex problems. So there's a trade-off involved in reducing bloat vs allowing better solutions to evolve.

Now, as to the second problem of maintaining diversity, that is a much tougher nut to crack.

Two of the simplest techniques are using mass extinction events and separating the larger population in to a number of smaller populations (called "demes", "islands", or "subpopulations") and is generally referred to as "the island model".

Mass extinctions are very simple. Just wipe out a large portion of the population in a given number of generations (or when you've determined that the population has become too inbred) and repopulate it with random individuals. This will maintain diversity.

Of course, you usually don't want to wipe out your best individuals, so you'll spare them. But that's where the problem comes in, as the best individuals will likely quickly come to dominate the population again. But hopefully in the mean time they'll breed with some of the new random individuals and the population will get some new blood. And when the population starts to get too inbred again, or a certain number of generations pass, there could be another extinction event, etc..

In the island model, if the islands are kept totally separate and no individuals from any island are allowed to breed with individuals from any other islands, then this is obviously equivalent to simply running a number of separate smaller populations at the same time. This is might be fine for maintaining diversity, but no benefit will accrue from combining the solutions in each of the subpopulations.

So, what's usually done is to let individuals from one subpopulation breed with individuals from another subpopulation at given times during the run. Again, if they are allowed to interbreed at every generation, then this will be equivalent to just having one large population as usual. So you'll want to keep individuals on any given island from breeding with individuals from other islands for a number of generations. The longer you keep the subpopulations from interbreeding, the more diverse they'll be (or so the theory goes). At the same time, the longer you keep them from interbreeding, the slower the convergence rate (the time it takes to reach a global optimum in the search space) will be.

Unfortunately, there have been mixed results reported about the island model. Some research has suggested that it's better to just use one very large population. But on the other hand, there have also been successes reported with the island model, and reports that smaller populations outperform larger ones. So it's a mixed bag, and usually very problem dependent, as much of evolutionary computation tends to be.

Finally, there are also many, many other much more sophisticated and effective techniques for fighting bloat and maintaining diversity. But I'll leave finding those as an exercise to the reader. Just a quick tip: type "genetic programming" and "bloat" or "diversity" in to the form on this page:

http://citeseer.ist.psu.edu/