What kind of sensor can I use to determine approximate distance of small animal?

Use more cameras  - on each side of the wall. Use RF to communicate to the cameras.

I think there are still too many variables. What are the dimensions of your area? What is the size of your pet(s)? And what is the purpose of the whole tracking system? Do you really need specific position or would you satisfy with some zones they stay in?

I don't have experience with RF radars, but I guess in a typical home the animal's fur will provide the worst reflection than everything else in the room. Just speculating here...

Most RF, Bluetooth etc. modules provide signal strength information, maybe this could be suitable for you. However, stronger signal doesn't always mean you are closer, it also depends on reflections, antennas orientations, etc. I think you should state some parameters of what is acceptable in your application and what is not.

Have you considered an iBeacon. Most mobile apps can give you a distance measurement once calibrated. It can be done using an intel edison, i think it can be done on the esp32 module as well

i think the technology will depend on additional parameters.. is this strictly indoors or outside as well? What distances are you looking to detect at? Are you looking to track the animal or react when it gets close?

you could look at a variation of geo fencing using rfid tags and antennae in various rooms.. if you are looking at a reactive system, motion sensors and proximity sensors might do the trick

Theoretically, you could use time of arrival.  The idea is that you have three or more beacons in the area and as long as the target is between them you can compare the time of arrival of each beacon to triangulate the approximate position of the pet.  Of course, with RF you have to have a way to keep the timing accurate to less than a nanosecond to get even 1 meter accuracy.  If you used sound, you could do the same thing with less timing accuracy but you'd have to be able to tell the difference between the beacon and other sounds.

You can use BLE 4.2 sensors with a mobile app. Have one proximity tag in the dog's belt and then place BLE beacons in each room. The beacons needs to be under mesh protocol, so all will be connected. Then, simply tie it up with your mobile app to check the pet's position.

PS: I recently worked in one of the Smart Pet Belt app where in if pet goes outside the proximity of 30 meters from the phone, it shoots static current (its cruel) but good to train pets.

DecaWave could probably work - for a price...

A stick.