Angle detection

I can't think of anything of-the-shelf that could directly replace 5 mercury switches in an existing system. It may be possible to use a different type of sensor provided some circuitry was developed to interface it.

Perhaps a sensor like this:

https://www.newark.com/sensor-solutions-te-connectivity/g-nsdog2-003/mems-inclinometer-mems-30v-module/dp/03AC1403?st=inclimometer

However, the range of that is 180 degrees, not 260 degrees. I don't know what accuracy and reliability you need, but a couple of those sensors might work, positioned so that they cover the full range you require. It's no bad thing to use two sensors, because if one fails, then the other can continue working until the failed one is replaced. You might also need some mechanical dampening for long life of the sensors (although the sensor PDF datasheet ( https://www.farnell.com/datasheets/2267166.pdf ) does suggest use with engines, they may require mounting to dampen vibrations (I'm guessing - I know nothing in this area). The datasheet does not mention boat applications (there could be all sorts of environmental issues using it for this application, like corrosion for instance).

Your post has an angle theme so in the interest of community promotion there are two RoadTest that might give some insight.

/products/roadtest/rt/roadtests/561/mps_magnetic_angle_p

/products/roadtest/rt/roadtests/584/mti-680_rtk_gnssins

If you are looking for a ready made solution then these will disappoint while promoting ideas for a RoadTest. The RoadTest forum benefits from post such as yours because it gives additional exposure 

Are you asking if there is a commercially available alternative for engine positioning system and indication ?  Are you looking for an infinitely adjustable angle position control ? Not sure this is the best place to ask that question. How is the engine tilt activated, is it manually adjusted to the 5 engage the tilt switches as indication ? Do you select one of 5 positions for a controller to move the engine to that position ?  A sensor would be required to show engine angle relative to the boat. A mechanically activated potentiometer or an absolute encoder would serve that purpose, maybe even a simple quad encoder with a home switch. The control system then could do a position indication, or next step up, position controller.     

This seems a very wide range of tilt angle, on my last (long time ago) power tilt outboard I think the range was about 90 degrees.

Can you provide some pictures.

My experience of boats suggests that finding a place to fit a rotary angle sensor is likely to be hard.

You might be able to fit a linear displacement sensor on the actuator hydraulic cylinder (if there is one) but if it has to satnd up to sea spary it will cost a lot of money.

This sort of thing might do (this one is not waterproof)

MK

Does this sensor simply indicate what angle you are manually setting the motor to, or is it used as feedback in a motorized angle setting mechanism?

A simple accelerometer can measure angle with respect to gravity (which is what mercury switches do) - just like the accelerometer in most smart phones. It may need filtering to remove boat accelerations, or you could have one on the boat and one on the motor.

If it is manual, you could just make a bracket to place your phone in while making the adjustment and run a tilt angle app. 

The BBC micro:bit has a built-in accelerometer. I did a project that showed its use in measuring angles. Unfortunately all the links on this site that point to that project are broken, but here is one of the videos from that project:

https://www.youtube.com/watch?v=T1dI5m7DBO4