Ah, those are some nice looking tiles!
You can easily calculate how many. There are 6 sides and each side is one of 2 terrains. So 2^6 = 64 total combinations.
Yes, at 3^6 you’d be having a whopping 729 combinations, which is already unmanageable. Though possibly not every combination would make sense.
In practice, the idea would be to define the color of the sides of each (rectangular) tile, and an algorithm could then use these colors to find tiles that can fit adjacent to other tiles. This algorithm can then be used to keep the map in a non-broken state while editing. It could also be used similarly to the terrain tool, in that you could allow painting with a color and it would find out which tiles to replace based on the new combinations of colors.
For the top and bottom edges of the tiles in your first screenshot, I can see three colors, which would represent “grass”, “sand” and “sand in the middle with grass on the sides”. For the sides I can similarly see three variations. If each of your 4 edges can have 3 colors, then you would need 3^4=81 tiles, which appears to be worse compared to using hex tiles though (but an orthogonal map will be more efficient to render and the tiles likely easier to edit).