Xsens MTi-680-DK: Experimenting with performance - Part 1

When dealing with high(er) performance pieces of equipment, it's very important to understand what all those numbers in the datasheet mean in the real-world. For example, for a particular use case, you may ask yourself whether you need to use a MTi-680 module or can you just use a cheap MPU-6050 IMU? In this post, I will do some experiments to see how good the module performs in some simple tests.

I will start with a disclaimer that the MTi-680 module has a lot of features and unfortunately, it's not possible to test everything. So, my testing will be limited by the availability and performance of test equipment I have and the limitations of test setups that I can come up with, and within the limits of my own resources. 

Roll, Pitch Accuracy tests:

The MTi-680 module comes factory calibrated. The first thing I want to test is test how accurate the factory calibration is. For this test, I am using a Mitutoyo Pro-3600 Inclinometer as reference, which boasts an accuracy of ~0.1 degrees. Before performing the tests, I recalibrated the Mitutoyo inclinometer by following the procedure prescribed by the manufacturer in the manual. I have used this instrument in the past to test and compare some accelerometers for commercial work and know that it works well. However, this setup can only be used for static measurements and so I will not attempt to measure dynamic accuracy.

The setup was quite simple. The Mitutoyo inclinometer has a flat machined top surface that is a reference surface. The MTi-680 has a flat top surface. I used these two surfaces lightly pressed against each other to align the MT-680 module with the inclinometer. I also aligned one side-surface of the MTi-680 module with the side surface of inclinometer top using a machined flat piece of aluminum. This was to make sure the module axes (tilt/roll) were aligned with the inclinometer measurement axis while taking measurements.The MTi-680's specifications say it should be accurate in roll, pitch to 0.2 degrees (RMS).

For doing these tests, I had to take the MTi-680 off the development kit and used it standalone using the USB-UART cable provided with the kit (I am very glad it came included!). I also had to change the mounting orientation of the module since I am using the top flat surface as the bottom surface. This was done in MT Manager's "Device Settings" window. It took me some time to figure out how to do this. The way I eventually did it can be found in my forum post.

/products/manufacturers/xsens/f/forum/51841/specifying-mti-680-orientation-in-mt-manager

My final setup looked like this:

The results I obtained:

I did notice that when moving to a new roll angle, while the MTi reading was immediately accurate to < ~0.4 degrees, I had to wait ~10s for it to settle down to its final value which is what I recorded in the table above (and which was closer to the value on the Mitutoyo inclinometer). I think this has to do with the parameters of the filters inside the MTi-680 (I was using default). Another filter could have shown slightly different behavior.

Another thing I noticed that, since for this experiment I was pressing the module to the Inclinometer's surface with my fingers, it seemed to matter where I had placed my fingers. After some trying, I concluded that pressing at the 3 mounting locations seemed to give the best results. I did find myself wishing the module body was made of machined metal, though.

I also realized while taking the actual readings, that while on the MTi-680 I can see both roll and pitch angles at the same time, the Mitutoyo only displays the angle in one dimension, so there can be some inaccuracies due to the other axis of Mitutoyo not being completely horizontal. Interestingly, I worked around this by looking at the reading of the other angle (the one I was not measuring) from MTi-680 and making sure it was very close to zero. This way I knew the Mitutoyo was also inclined on only the axis I was measuring, since the two devices were held pressed together.

Nevertheless, as far as the final values are concerned, they all look fairly accurate, which seems to confirm that the module was indeed well calibrated before sending it out. 

Compass Heading repeatability test:

Since  it is hard to test the absolute accuracy of compass heading readings due to lack of surveyed reference points or other calibrated reference magnetometer, so I will settle with testing the repeatability instead. This will be done by setting a reference position away from any ferrous/magnetic body and noting the heading after aligning the MTi-680 with this reference surface. Then, move/rotate the device randomly and bring it back to the exact same position. Note down heading reading, and then repeat to get several readings.

When I started doing this test, I realized that when using the default settings, there was a gradual drift in the heading value and that the "yaw" reading was not north referenced and didn't seem to use the magnetometer. Since in this test I wanted to see the Magnetometer performance, I needed to have the device use the magnetometers. Opening the Device settings, i noticed that the default filter setting says "24.2 General_RTK", while there was another filter available which sounded more promising "26.2 GeneralMag_RTK". Looking it up in the manual, I found the following information:

This meant I needed to use this one for my test. So, I switched to this filter from the Device settings in the MT Manager. I left all other setting at default values.

I established a reference surface and fixed position by using a big plate of Aluminum weighed down enough that it would not move when I gently press the Xsens module against it. The position along that reference surface was fixed using another piece of aluminum. I made sure there were no magnets or ferrous materials within 1ft of the Xsens module. After that, I took a reading with the module at the reference position, moved it away, move it in random orientations, then took the module back to the same location to take next reading. And then repeating the process several times.

My plan was to check standard deviation of the heading values after this experiment. My assumption was that the heading values should stay constant because we are using same reference surface/position for every reading. However, it seems that the heading is initialized using the magnetometer at initial power-up and afterwards the filter relies more on the gyro values. When I left the device in the same place for ~5 mins in the end, the filter didn't seem to go back to the same initial heading value (which is what I would have assumed if the magnetometers were used constantly). Note that I didn't perform any gyro bias estimations during this experiment, which probably explains the drift.

The manual mentions a few things about the heading values:

Under "Initial Heading Input" in the manual: "Out of the filter profiles available for the MTi-670(G)/680(G), only the GeneralMag filter profile will provide the user with a North-referenced heading when the MTi is powered up. For all other filter profiles, the Yaw will initialize at 0 degrees. As soon as a GNSS fix is available and the MTi starts moving at a sufficient velocity, the Yaw will converge to a North-referenced heading."

In this case we are not using GNSS and not moving so the GNSS part is not relevant. So, the initial heading should come from the magnetometer.

Again, the manual states under "Filter profiles for MTi-670(G) and MTi-680(G) (GNSS/INS)", for "GeneralMag /GeneralMag_RTK":  "This filter profile bases its yaw estimate mainly on magnetic heading and GNSS measurements. A homogenous magnetic environment and proper magnetic calibration are essential for good performance. This filter profile produces a north referenced yaw output directly after powering up the MTi."

Dissecting the above, since I am not using GNSS, that part should be irrelevant. The magnetic environment is homogenous (nothing magnetic/ferrous close by within a foot). I didn't perform a calibration but then, since I am measuring in one direction and same location in space, I expect that any soft/hard iron magnetic distortions, if present, should not matter. So, the only plausible explanation for the behavior seen is that either the Magnetometer is not used after initial start-up (i.e, filter is using integrated gyro values only), or the weightage given to the magnetometer readings is very small and so the filter will very slowly converge back to the magnetometer readings, given enough time. I will investigate this further in a bit. 

One thing is clear, that the device is definitely taking the magnetometer readings for initialization. So, I have a new experiment to test the heading values from magnetometer. I can leave the device in one orientation, power cycle the device and take the readings right after. I will standardize to a gap of 10s between power up and noting down the reading to give time to connect to device and for the internal filters to initialize and stabilize. Here are the results I obtained:

Average: 96.47 deg, Std. Dev: 0.15 deg. The results look quite consistent on the magnetometer readings alone.

Now, I want to understand better the behavior of the filter with regards to how it is calculating the heading readings. For this, I will do a simple experiment: Set the device in a fixed position. Set the device to log Euler angles at 1s interval (lowest possible rate) and log the data right after power up until the angle seems to be stable. Then, move the device to a random tilted orientation (which caused issues previously), and then return the device back to exactly the same position and see if the angle converges back to previous or different final value. This will give us indication on how the internal filter is working.

When performing this test, the data was logged using the red record button in MT Manager which started logging to a file. Then I opened this file in MT Manager and used the export feature (File -> Export) which gave me option to export the orientation angles as comma separated values to a text file (among other options). This new file was super easy to process in Excel. This whole process was very easy, and I realized that I didn't even need to log my data as Euler angles in the first place. Even if I had logged to quaternion angles (default) I could still export as Euler angles. There are plenty of options to customize how and what data is exported. VERY NEAT!!

And without further ado, here are the results graphed using excel:

X axis is sample number (1 per second), and Y axis is the Heading angle reported in degrees. The graph shows some interesting results. First zone is data logged after power up. We can see that it settles to a value (95.85 degrees North) in less than a minute after power up (first 15 seconds are missing). Then the 2nd zone is when I moved the device around in random orientations for about a minute (not too violently). I have removed that data so the graph scale can be better show the data we're interested in. The third zone is after returning the device back to its Initial spot. The device was not disturbed after this and left to log data for a few more minutes. From the graph, I can say the following things:

1. The heading value DOES settle back to the same heading that it settled to initially. Meaning the filter does keep using the magnetometer data even after initial power up. However, as I suspected earlier, the weight given to the magnetometer data is much less than the data from gyros, which explains why it took ~8 minutes to converge back to the real heading from magnetometer. 

2. After power up, the device takes around a minute to converge to its real heading value. Good to know and consider this if heading data is critical to your use case.

3. There is a very interesting overshoot that appears on the graph between samples 300-350. I am very surprised to see this considering the response of the filter is quite slow in terms of converging back to the magnetometer value and the device was stationary in the duration. 

Another thing I have learned here is that instead of "eye-balling" readings and drawing conclusions, the better way is to log data over as suitable interval and then get the bigger picture . 

Btw, I also found some articles on Base related to these:

Estimating Yaw in magnetically disturbed environments (xsens.com)

Reasons why the heading/yaw is not stable or incorrect (xsens.com)

Heading Accuracy/drift tests:

Heading data is one of the trickiest outputs for these types of devices. This is because roll and pitch gyros can have their biases removed using the accelerometers. On the yaw axis, though, there is no gravity component, so the accelerometers can't be used as a reference to remove bias for the heading/Z axis gyro. Btw, these gyro bias estimates are done by the Kalman filter running in the system and is a continuous process because the biases change over time. If you search around for very high performance (and super expensive) IMU/INS systems, they will usually use a high accuracy FOG (Fiber Optic Gyro) at-least on the Z axis. For space applications, where there is no gravity component on either axis, they will use FOGs or RLGs (Ring Laser Gyro) on all three axes. 

In case of MTi-680, all three axes have MEMS gyros, albeit of decent quality and factory calibrated. It also has 3 axis magnetometers inside, which can feed into the EKF (if opted) to provide a heading reference for gyro bias estimation. However, magnetometers are generally very problematic in actual use due to interference from magnetic or ferromagnetic materials around. So, in short, you can't entirely rely on gyro data and you also can't entirely rely on magnetometer data. An EKF would try to play on the strengths of both.

Seeing the heading tracking of the MTi device will give us a good idea of how much accuracy/error to expect and thus what applications the device can be used in. For this test I again started with the factory default settings (by clicking "Revert" button inside the "Device settings" window). Then I changed the "RotSensor" matrix to use the module with connector-up orientation, as described for a previous test. Next, I changed the Output data settings such that I was only getting Orientation data at 1 Hz (one sample per second), as well as the packet counter. This was done to keep the amount of data manageable for longer runs. With these as the base settings for data logging, I logged data for several situations for 30mins each. The module was power cycled before each test. 

The MTi-680 device was kept in the same place & orientation in all the following tests and was NOT moved at all. 

1. Yaw data without magnetometer sensors

This test was done with all default settings to set a baseline. Note that the default settings use the "24.2 General_RTK" filter. This filter DOES NOT use the magnetometers.

On X-axis is sample number from the MTi-680, which in this case mean time in seconds since startup (since we're logging data at 1Hz). In half hour, the heading drifted by about 120 degrees. This should all be coming from the z gyro bias. Since gyro outputs a rate of turn (degrees per second) which is integrated over time to produce the angle, the bias error is also integrated and continuously drifts the yaw angle.

2. Yaw data with "manual gyro bias estimation"

In this test, the same default filter was used ("24.2 General_RTK"), hence no magnetometer input, but the "Manual Gyro Bias Estimation" feature of the MTi-680 was used at the very beginning, with the default 30 seconds time interval. This feature can be used to estimate and compensate for gyro drift when we know that the device is completely stationary. You can read more details about this feature on the base article. Note that the estimation was done only once at the beginning of the test before letting it run for 30 mins.

The yaw angle drifted by 20 degrees during the test. This was much better than without using the Manual Gyro Bias Estimation in the first test.

3. Yaw data with "Manual gyro bias estimation (30 seconds interval) at T=0, 10min, 20mins

For this test, everything was same as previous test, except I performed Manual gyro bias estimation every 10 minutes, starting at the very beginning of the test. The default "24.2 General_RTK" filter was used. Default 30s window was used for the gyro bias estimation.

It's promising to see that the yaw angle drift further improved to <8 degrees over the span of 30 mins. This was expected as now we're performing the Manual gyro bias estimation more often.

4. Yaw data with "Continuous Zero Rotation Update (ZRU)" enabled

This test also used the "24.2 General_RTK" filter (hence no magnetometer input) but instead of performing the Manual gyro bias estimation, we use another feature in the settings, which estimates the gyro bias automatically whenever it detects the device to be motionless for a particular interval of time. More details can be read in the documentation here. 

Interestingly, and somewhat unexpectedly, the total drift over 30 mins increased to ~20 degrees. However, if we exclude the first minute or so, the drift is less than ~7-8 degrees and seems quite consistent. The drift in the first minute might be due to the filter converging(?). The best part is that we didn't have to manually initiate the bias estimation in this case, which can be very handy. There are cases when this feature might not work too well, for example if the device is actually rotating but very slowly because then the MTi-680 will not be able to differentiate between actual motion and gyro bias.

5. Yaw data with magnetometer sensor data incorporated

For this test, I wanted to see the yaw performance WITH the magnetometers, by using "26.2 GeneralMag_RTK" filter. Note the use of "Mag" in this filter name, which indicates the usage of magnetometers. No other feature of the MTi-680 (e.g., ZRU, Manual Gyro Bias Estimation) was used.

The result is very interesting. The Yaw reading varied by no more than 0.6 degrees during the whole 30 minutes. And if we exclude the initial minute or so, then the variation was less than 0.3 degrees over the interval. Also interesting is the fact that since this is using the magnetometers, the heading does not keep drifting due to accumulated bias errors, as we saw in previous cases. This means that even if we had kept logging data for longer periods of time, the heading would have stayed quite accurate, and without a continuous drift. We also saw previously that even though the device is using magnetometers, the weightage given to the magnetometer values is much lower than the gyro values, so magnetic interference would not affect the output in this case as it does for pure magnetometers without sensor fusion.

In the end, we can compare all the Yaw graphs by plotting them together on the same graph. Note that the "GeneralMag" filter produced North referenced reading, so for ease of comparison, I offset all readings to bring the starting point at 180 degrees to be consistent with the other tests (which did not produce North referenced heading values). 

Since the much larger drift on the default settings produced much larger drift and dominates the Y axis auto-scaling, here is one with Y axis scale manually limited to enable better comparison of the other 4 experiments:

From this comparison, my conclusions are:

1. If Yaw/heading tracking is very important, then it seems that "GeneralMag" filter is the best (only?) option. 

2. If you can't use the magnetometers for any reason, then best results can be obtained either with ZRU option, or with performing Manual Gyro Bias estimation frequently, assuming the application allows. Note that the heading is not North-referenced in these cases.

3. In most cases the data gets better after the first minute. This is probably from the filter converging. So if the data is being used in the first minute, perhaps use it "with a grain of salt".

Lastly, I want to point out that gyro drifts change with time (hence we can't just cancel them out once and for all), which means that slightly different results will be obtained each time if the same experiment were performed repeatedly. However, the general trend should follow as seen above.

Before I conclude this, I want to add a graphic I found in a brochure from Tamagawa Seiki Co. Ltd. This gives a good idea about how errors/drift in heading per hour can translate to errors in final position (assuming no GPS)

Looking at the above, you can go back to the graphs shared above for each use case and get an idea of what position error you can expect to have after driving for certain amount of time, using the heading values from MTi-680 as reference. As an example, with the GeneralMag filter, and assuming Compass calibration was done properly, this should give you error of around 10m after driving for 5 kilometers!! That is VERY impressive, I think!

I want to mention again that I was able to perform all these tests without writing a single line of code. Just the MT Manager software and Microsoft Excel. This is because of all the flexibility available in the MTi device settings as well as the excellent way in which the data can be logged to files, replayed, reprocessed and exported. I have come to REALLY love this part of the MTi ecosystem. I have used tools from some other vendors of similar modules in the same price range at work, and this is without doubt my best user-experience so far by a decent margin.