How would I send the player to the 2nd floor? Is there a z-position I could use? For example, if I assign the custom properties which teleports the player to set positions of a stair tile to "warpX 20, warpY: 43, warpZ: 2(represents the 2nd floor)
Well, there are many approaches you could use, and in the end it depends on what you are most comfortable with to implement in your game and to use while editing your maps.
You could use different map files for different floors. The Mana World and Source of Tales are both games that I have contributed to that have used this technique when different heights exist in large areas.
Like you suggested, you could also teleport the player to another area on the same map. Again, The Mana World has used this for the interior maps for houses, which tend to be small and use the same tilesets, so it can be convenient and performs well.
Your second suggestion, using layers and then showing/hiding layers as the player goes up stairs is also a perfectly valid strategy.
What kind of logic or custom properties you use to implement the switching between floors is up to you. Though rather than hardcoding the target coordinates, you’ll probably want to place an object at the target location, give it a name, and then refer to it by that name.
In the future, I’d like to support “object references” as custom properties, which could make the connection between the objects visible, as well as allowing to jump the view to the target object. This is tracked by the following issue on GitHub: