Introduction and Disclosure

Part 2 was supposed to be a post about my notes while I was investigating what the PAN1780 can do better than other development boards. However, this section turned out to be longer than expected, so I decided to spin it off into a new installment The original draft would be continually updated since found that I am adding more and more to the notes as I go through the projects I'm doing with the PAN1780.

I've been really curious about the nRF52840 and acquired a few development boards to experiment with. This was also partly the reason why I applied for this roadtest; I wanted to know the state of the art regarding manufacturers' approach with development boards based on the chip.

Since I already have other boards with different module manufacturers, I thought it might be a good idea to compare them with each other. Here are the contenders:

From upper left, clockwise: Two PAN1780 Evaluation KitsPAN1780 Evaluation Kits, Waveshare nRF52840 Eval board, nRF52DKnRF52DK, nRF52840DKnRF52840DK

I wanted to also include the nRF52840 Dongle and the Adafruit Clue (which has the Raytac chip) for comparison, but I was still figuring out the Board Support Packages in Segger Embedded Studio as of this writing, and I thought I should just publish the results now and I can update the tables later once I get the data.

Setup

I live in a small apartment, and unfortunately the city I'm staying in has extended the lockdowns so I am unable to do testing outdoors.

Considering this, I thought of five scenarios to test that would make the most of the current situation:

• Adjacent
• 1 meter apart
• 5 meters apart
• 1 meter apart but with the board under test inside a plastic box
• 1 meter apart but with the board under test on top of my wifi router

Adjacent (Horizontal, Vertical, and Side orientations)

This scenario consisted of the boards being at most a couple of centimeters apart. It should theoretically be the "best case" scenario and would ideally show the maximum throughput. However, antennas have an "affinity" to orientation, so I also measured the throughput with the board oriented in Horizontal (as above), Vertical, and Side positions.

Here's an example of a board under test in the Vertical orientation.

Here is an example of a board under test in the Side orientation.

1 Meter Apart

I took a tape measure and measured 1 meter between the edges of the boards. I chose 1 meter because that is also the distance that many beacon protocols (eddystone and ibeacon) use for calibrating their RSSI values for distance estimates based on signal strength.

The board under test is on top of the hamster cage, one meter away from the test controller. Pay no attention to the sleeping hamster.

5 meters apart

I didn't have a long enough tape measure to accurately measure 5 meters. What I do have is a roll of power extension cables that is purportedly 5 meters.

So I unrolled the whole length of cable and used that as the marker for 5 meters.

It so happens that the measured 5 meters was the same distance between my computer desk and the dining table, so that was fortunate.

1 Meter apart but with the board under test inside a plastic box

I wanted to simulate some obstacles that may affect BLE throughput, so I used the same 1 meter distance, but placed the board under test inside a plastic box.

The box is just your everyday, food safe and microwave safe plastic box. There are no markings to determine what kind of plastic it is though.

1 meter apart but with the board under test on top of my wifi router

Since BLE operates in the 2.4ghz band, I also wanted to test how much would throughput be affected if there are a strong 2.4ghz emissions in the area.

While doing the throughput test, I connected a laptop to the 2.4ghz SSID and streamed youtube (the lofi hiphop channel).

Methodology

I wrote about Nordic's ATT_MTU throughput example before, and I used the same project to flash each of the boards to be tested with it. Briefly, the example requires two boards: the tester, and the "dummy" (as it is called in the source files) which is determined by which button is pressed (BTN 3 for the tester mode, BTN 4 for the dummy mode).

Since the non-nordic boards (the PAN1780 and the Waveshare eval board) both implement the gpio assignments similar to how the nRF52840DK does it, I was able to just flash the PCA10056 included hex directly without modification. The only change was with the Waveshare eval board, where the jumper system allowed me to assign the single button to the correct gpio assignment as the code was expecting (this is the reason behind the white connector wire).

Each of the scenarios were tested three times, and the results averaged to get the final throughput.

Each of the boards under test were powered via a USB powerbank to make sure that the module is adequately powered. The control board is powered via a USB cable connected to my computer.

There were two sets of tests: the first one with default settings:

And the second set with the optimal settings for best throughput, as mentioned in the novelbits article that explored the theory behind it.

The throughput results are monitored via UART with TeraTerm being the client.

I do not have professional test equipment nor am an expert with regard to test setups, so my results may be skewed. Nevertheless, I took effort to make sure that each of my scenarios are repeatable, so that any future critique may be addressed and then the experiments reperformed.

Results

If you would like to see the results together with the data of each run, you can take a look at this google sheet (warning: spoilers).

Orientation Check

Since orientation is important when dealing with antennas, I wanted to know which orientation would yield the best throughput. All of the boards were tested adjacent to the control board, with their orientation being the variable. The control board's orientation was matched to the board under test accordingly.

All of the boards seem to perform the best in the Horizontal orientation, at the max of nearly 341Kbps. There is very little deviation between the boards, and we can probably consider these differences as statistically insignifcant.

Now that I know that the Horizontal orientation gives the best throughput, all the succeeding tests were then made with this orientation.

Set 1

1 Meter Apart

The experiment took off with the PAN1780 in the lead, but not by much. The nRF52840DK trails closely behind, with the Waveshare being left way at the back.

5 Meters Apart

The story changes when we move the boards 5 meters apart. This time, the two official Nordic development kits took the top places, with the PAN1780 snugly in the middle and the Waveshare still lagging far behind the rest of the pack.

1 meter apart but with the board under test inside a plastic box

This time the nRF52DK (which has the older nRF52832 chip) takes the lead, with the Waveshare surprisingly closing in. The PAN1780 takes last place this time around.

1 meter apart but with the board under test on top of my wifi router

The nRF52840DK again leads the pack, with the PAN1780 towards the rear together with the Waveshare.

Set 2

Adjacent

These settings really improved the throughput, with three boards (including the PAN1780) at 1.3Mbps and the Waveshare trailing far behind at 1Mbps. I didn't test the other orientations at this point, as I didn't feel it was necessary to find out which orientation is the best performing. I considered the default settings orientation test to be representative of the two sets.

1 Meter Apart

It was surprising (for me at least) that with the high throughput settings, the signal is severely attenuated with just a distance of 1 meter. The best performing was the nRF52840DK while the PAN1780 and the nRF52DK are comfortably in the middle of the pack. I'm not sure what happened with the Waveshare, which barely improved its throughput scores from the default settings test.

5 Meters Apart

The signal was attenuated even further at a distance of 5 meters; however we have a very strange outlier in the nRF52DK which saw throughput scores even higher than it was when it was just 1 meter away from the control board.

1 meter apart but with the board under test inside a plastic box

Again, the nRF52DK performed well ahead of its peers, improving yet again its throughput from the 1 meter distance test without the plastic box. What's interesting as well, is that the Waveshare also improved its score compared to when it wasn't inside the box; could the reason be that the signals were being reflected inside the plastic box and therefore the board has additional chances to receive them?

1 meter apart but with the board under test on top of my wifi router

At first I hypothesized that the PCB antenna would cause throughput scores to fall as they are known to be more susceptible to RF interference. However, the PAN1780 strangely performs even worse (albeit not that worse) than the Nordic Devkits even if the PAN1780 uses a chip antenna. The Waveshare uses a chip antenna, and its performance seems to be consistent with its throughput at a distance of 1 meter.

Conclusion

The PAN1780 holds it own with the Nordic development kits at the default MTU settings, but have performed worse with the "best throughput" settings.

While there may be numerous factors contributing to this (e.g. unknown RF traffic, small deviations in distance, board design, etc.) they are uncontrolled, and thus it is very difficult to pinpoint the exact reasons why the PAN1780 behaved in such a manner.

This might be an area where possible future research might explore

If anyone has any comment or feedback or other ideas regarding this test, please feel free to mention that. I'd gladly redo or setup the tests with your input.

Here are graphs, if you're in those sort of things

With the default settings

With the "best throughput" settings

What is interesting is that with the "best throughput" settings, the boards (except for the Waveshare) performed considerably worse than the with the default settings in an environment with lots of RF noise (on the router).