Since you worked so close to pilots, how do you think they would feel if their complete vision range was covered by some future night vision system? I am under the impression that the pilots looked out of their window still due to the fact that the actual FOV in goggles was so small.
One thing I learned during my days of of advanced avionics is that you do not want to block both eyes of a pilot. Even though your normal vision cannot see in the dark, there are all kinds of things that can blind the IR goggles. Plus, we found that most pilots are a lot more comfortable with night vision if they can still see out of the window. If you want to fly via sensors, you might as well go fly the UAV's.
Most of the search and rescue folks I worked with always had someone looking out of the windows in addition to all of the fancy IR sensors. That way you get all of the information in real time and sometimes your eyes will see things outside of the normal frame of vision that is essential to the decision making process.