Repository: godot
Issue: Terrain set gives unexpected results, algorithm creates bit masks impossible to express in 3x3 minimal
Labels: bug,topic:core,topic:2d

### Godot version

4.0.beta8.official (45cac42c0)

### System information

Linux, AMD Radeon Vega 3 Graphics

### Issue description

Terrain sets are an awesome new feature, but the algorithm of the new "connect mode" painting tool shoots itself in the foot at one point, if you work with simple A-in-B and B-in-A terrain sets. There are situations where the algorithm changes neighboring bits in a way that it ultimately needs a tile outside of the typical 3x3 minimal template – and because it doesn't find a matching tile for that bit mask, it chooses the next best tile, which unfortunately causes other tiles to change as well, etc.

The result is both completely deterministic and strange.

I posted this on Reddit before, and it was suggested I open an issue here, so please don't mind me copying some of the descriptions and pictures.

This is the result of painting beach tiles on water tiles and then turn some of the beach tiles back into water again. It works as expected:
![terrains-example-4](https://user-images.githubusercontent.com/98526919/208263823-11b816cf-b5a4-4e99-85ed-4c3a41e4367b.gif)

Now, here is an example of the same procedure with the terrains reversed. This time I paint water tiles on beach tiles and then turn some of them back into beach tiles. Note how the bit mask is different from the one above:
![terrains-example-2](https://user-images.githubusercontent.com/98526919/208263878-82be7a0d-8d9b-4a7e-9095-8e24a2c155b4.gif)

The neighboring edge and corner don't revert back back to beach, but stay water, making the bottom-right tile impossible to be expressed in 3x3 minimal:
![wrong-tile](https://user-images.githubusercontent.com/98526919/208263945-530d2dfc-c864-4b07-813e-d1aa81d20e17.png)

That is a problem, because I'd need to create a lot more tiles to cover those cases! And to make matters worse, wrong tiles affect their immediate neighbors as well, as you can see with the two wrong patches of grass below the rectangle, which exist solely because of the wrong water/sand tile above them.

I doubt it has anything to do with peering bits, as the water-in-grass and water-in-sand tile sets seem identical to me:
![terrain-bits-2](https://user-images.githubusercontent.com/98526919/208263960-75a410b9-95e9-4ffb-b3a6-60bbec363f72.png)
![terrain-bits-3](https://user-images.githubusercontent.com/98526919/208263963-b5080e45-99e3-4ad6-834f-4696d47d534c.png)

It's not only this water-in-beach having this issue, but also grass-in-water, while water-in-grass looks fine (note how it's the other way around this time):
![terrains-10](https://user-images.githubusercontent.com/98526919/208264055-e62f56b5-660f-49ff-b4c6-9547d16cfb85.png)
![terrains-4](https://user-images.githubusercontent.com/98526919/208264057-723c1ccb-865f-46c1-8863-06964ffb4fb3.png)

I'll spare you more tests, it's always the same: A in B works, B in A behaves weird.

The questionable choice the algorithm makes and which causes all the other ones is probably at this point:
![terrains-example-3](https://user-images.githubusercontent.com/98526919/208264089-da9933fc-2147-4c36-a4e1-926929423233.gif)

However, my preferred outcome would have been:
![terrains-12](https://user-images.githubusercontent.com/98526919/208264100-82bb49ac-a79b-4bee-82fe-78019d1c9016.png)

I call the choice of the tile questionable, but I actually understand where it comes from: It changes the central bit from water to beach and connects the tile to its right neighbor of the same terrain, while leaving the water bits connected to the water at its top, bottom and left sides. But that causes the tiles below to adapt and also keep the water bits until it reaches the last tile of the paint stroke, where it can't find a matching tile (see picture further above). And the second-best tile it uses instead causes the two tiles below to also change and put the patches of grass where they don't belong, only because that's the best it can do.

Like I said above, it's deterministic, once you understand it, but at the same time unintuitive and it looks weird.

I'm not completely sure how to solve it. Maybe create a third painting mode called "overlap" alongside "connect" and "path" which not only changes the center bits of selected tiles to the selected terrain, but all bits within the selection plus all directly neighboring bits. It would effectively always place transitions outside the selection, provided the matching tiles exist in the tile sets, of course.

### Steps to reproduce

Load my example project below, which already has a TileMap with lots of terrain rectangles set up to test, and use the "connect mode" to first paint the vertical lines and then revert to the original terrain, until you see the result described above.

Alternatively:
1. Set up two 3x3 minimal tile sets (one for terrain A in B, the other for B in A).
2. Add them to the same terrain set with "Match sides and corners" and paint the peering bits accordingly.
3. In "connect mode", paint two rectangles of terrain A and terrain B, respectively.
4. Still in "connect mode", paint two intersecting lines of the opposite terrain in each rectangle.
5. Broaden the vertical lines.
6. Repeat the same stroke, but now with the same terrain as the rectangular, effectively undoing the change from step 5.
7. Observe that it works in only one of the rectangles.

### Minimal reproduction project

[test.zip](https://github.com/godotengine/godot/files/10252719/test.zip)