I've been looking at some magnetic rotary encoders for a small rotary table and one of the recommendations was to average the output. This reduces the ability to record high rotation speeds but for my application this is not an issue as the table will be rotated by hand.
For numbers you can simply sum the values and then divide by the number of values. This does not work for angles because the numbers reset after 360. Luckily some mathematicians have worked it out already and have a solution using trigonometry.
http://rosettacode.org/wiki/Averages/Mean_angle
For this to work with the Arduino you need to use floating point numbers hence the "double" data type and also the mathematics library.
My version of the code is over at Github, all I've done is swap stdio from the C example with serial.print.
https://github.com/Workshopshed/AverageAngles
There are a couple of things still to investigate. Firstly how much memory does this simple function take up (allowing for all the math functions in the background) and how long does it take to run.
I'm planning to use the function with a fixed number of angles to average (probably 2) so will likely "unroll" the loop in the averaging function.