Im only using three terrain types to demonstrate this, but when I tried this with all terrain types I could not get it to work either.
Since this, I did noticed a place in the terrain editor I forgot to select. This brought the brush back down to a 9x9 mess instead of sprawling across the board.
What it looks like is the center block is the full block, however some of the other 8 “transition” tiles are incorrect.
This happens if the terrain definition is not complete. The tool then tries to change more surrounding tiles trying to find a valid transition.
It is a bit hard to tell from your screenshots only, but it looks to me like you placed the lower grass from the tileset first and then tried to draw dirt on it. However there is no transition defined for that type of grass?
How do I define the transition tiles?
If I use transparent tiles and simply draw on a separate layer it draws like it should.
Well your tileset does not seem to allow the transition from dirt to the greener / lower grass (at least not in the visible area). So you have no way to define such a transition.
How would you do that transition by hand?
Oh, in that image I had not used that one because I wasnt using the stone texture.
Theres only 1 grass. The terrain brush editor highlighted tiles that have been used by other terrains, making them slightly greyer.