SBC Network Throughput

I'll try this out on some of the Arm boards I have over the weekend.

On your x86 machines low figures..  Gigabit wire speed is a much harder thing to accomplish, you'd need to investigate ethernet packet sizes, switch capability, and things like whether your ethernet adapter is PCI, PCIe and whether it's a PCIe x1, x4 etc.   Also, for example, I've seen various problems with onboard realtek 8169 series gigabit chips in the past, so you would have to look at the chip, the kernel and driver versions too.

Typical cheap 'desktop' gigabit adapters will often struggle, server adapters are better but come with much increased cost and usually require a type of slot that's not available in a desktop.

As you can imagine this becomes even more of a problem when you have a 10G card in your machine.

Anyway, as a first step, here's some results from a 32bit x86 kernel 3.7.0 machine to a 64 bit x86 machine kernel 3.8.0

1128.4432 MB /  10.01 sec =  945.8611 Mbps 3 %TX 11 %RX 0 retrans 0.26 msRTT

1129.1255 MB /  10.01 sec =  946.3785 Mbps 3 %TX 15 %RX 0 retrans 0.25 msRTT

client machine has the following NIC

Asus P5Q with 4GB ram &

Server

HP ML110 G6

There's a Cisco SG300-52 switch sitting between them.

The OS isn't particularly meaningful. once upon a time the client was Slackware but virtually everything on it has been replaced with self compiled stuff.  The server is entirely self compiled.

Both TX and RX results drop to ~932Mbps when using IPv6

Awesome info, thanks selsinork!

The big problem is trying to summarize the info that most affects the results, so I've added a column "Limits" for information on anything that may be a significant limit or constraint on the throughput measured.  For example the bus through which the NIC operates can be a major limiting factor, and in my case is a known constraint on gig-to-gig transfers since my server's D-Link DGE-528T card is plugged into a lowly PCI slot running at 33MHz --- adequate at 100Mbps but certainly not at gigabit speeds.

I've added your Asus machine as "client" in the table, and have made the assumption that your two nuttcp output lines correspond to client Rx and Tx thoughput in that order.  (Not that there is any huge difference of course.)

I have machines with on-motherboard gigabit Ethernet  too (typically the NIC is on a PCIe channel of one or more lanes), so I'll try to find a box that's more suitable for network throughput tests and rerun everything.

This is an interesting area of testing, since not only will we gain better understanding of our ARM boards but also improve our home networks as our bottlenecks show up in the numbers.  I intend to update any entry that is affected by improvements at the server end or in network infrastructure, since each entry is intended to quantify the client's Rx/Tx throughput only (the server and network are assumed faster).  This will be an asymptotic process geting progressively closer to valid measurements for the SBCs.

As the engineering mantra says, you don't really know something until you measure it.

PS.  Disappointingly, upgrading the Pi kernel and  firmware with a whole year's improvements (now on 3.6.11+ #545) has only increased Pi's internally-constrained Tx throughput by 2.3%.  No other variables had been modified prior to running the new tests.  New table entry added.

Morgaine Dinova wrote:

 

The big problem is trying to summarize the info that most affects the results,

Yeah, and trying is probably a pointless task. Think of it more as a set of starting points to look at when you have a result that seems out of place.

I've added your Asus machine as "client" in the table, and have made the assumption that your two nuttcp output lines correspond to client Rx and Tx thoughput in that order.  (Not that there is any huge difference of course.)

Yes, receive than transmit, would probably be nice if the output reflected which direction.  The other bit of info provided in the output that's likely to be interesting/relevant is the transmitter and receiver CPU utilisation when running these tests. Being able to do the full 100Mbps in a synthetic test is one thing, but if it takes 100% CPU to do it then it's not realistic to expect to be able to do that while the system is doing other things like reading the file it's transferring from disk.

I have machines with on-motherboard gigabit Ethernet  too (typically the NIC is on a PCIe channel of one or more lanes), so I'll try to find a box that's more suitable for network throughput tests and rerun everything.

In my experience it's unusual for onboard gigabit NIC's on desktop class boards to use anything other than x1 and usually the cheapest chip they can find, even on high end servers it's often only x2. Multi-port add-in cards and 10G cards can be different of course.

Morgaine Dinova wrote:

For example the bus through which the NIC operates can be a major limiting factor, and in my case is a known constraint on gig-to-gig transfers since my server's D-Link DGE-528T card is plugged into a lowly PCI slot running at 33MHz --- adequate at 100Mbps but certainly not at gigabit speeds.

I was curious about this, 33MHz PCI has a theoretical max bandwidth of 132MB/s, which ought to be enough for a gigabit network in one direction at least.

So, with a 33MHz PCI adapter using an Intel 82540EM which is a desktop adapter I get the following:

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -t 1.2.3.4

1092.4375 MB /  10.01 sec =  915.5943 Mbps 7 %TX 15 %RX 0 retrans 0.23 msRTT

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -r 1.2.3.4

1088.5742 MB /  10.00 sec =  912.7412 Mbps 4 %TX 28 %RX 0 retrans 0.19 msRTT

trying to run a transmit and receive instance simultaneously gives

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -t 1.2.3.4

  832.2246 MB /  10.01 sec =  697.3698 Mbps 7 %TX 12 %RX 0 retrans 0.27 msRTT

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -r 1.2.3.4

  278.1735 MB /  10.00 sec =  233.2731 Mbps 1 %TX 15 %RX 0 retrans 1.94 msRTT

so overall throughput roughly the same and obviously limited by PCI bandwidth. 

The system I have this card in is an Atom 230 @ 1.6Ghz which is significantly slower than your E6550 based system, but does at least prove that card/driver could be a factor as the bus itself is theoretically capable of more.

Morgaine Dinova wrote:

For example the bus through which the NIC operates can be a major limiting factor, and in my case is a known constraint on gig-to-gig transfers since my server's D-Link DGE-528T card is plugged into a lowly PCI slot running at 33MHz --- adequate at 100Mbps but certainly not at gigabit speeds.

I was curious about this, 33MHz PCI has a theoretical max bandwidth of 132MB/s, which ought to be enough for a gigabit network in one direction at least.

So, with a 33MHz PCI adapter using an Intel 82540EM which is a desktop adapter I get the following:

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -t 1.2.3.4

1092.4375 MB /  10.01 sec =  915.5943 Mbps 7 %TX 15 %RX 0 retrans 0.23 msRTT

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -r 1.2.3.4

1088.5742 MB /  10.00 sec =  912.7412 Mbps 4 %TX 28 %RX 0 retrans 0.19 msRTT

trying to run a transmit and receive instance simultaneously gives

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -t 1.2.3.4

  832.2246 MB /  10.01 sec =  697.3698 Mbps 7 %TX 12 %RX 0 retrans 0.27 msRTT

root@solos2:~/nuttcp-6.1.2# ./nuttcp-6.1.2 -r 1.2.3.4

  278.1735 MB /  10.00 sec =  233.2731 Mbps 1 %TX 15 %RX 0 retrans 1.94 msRTT

so overall throughput roughly the same and obviously limited by PCI bandwidth. 

The system I have this card in is an Atom 230 @ 1.6Ghz which is significantly slower than your E6550 based system, but does at least prove that card/driver could be a factor as the bus itself is theoretically capable of more.

Very good points here.  It seems we need an nutpci utility to determine the real throughout of PCI.

Next time I'm opening machines I'll do some additional testing using the on-board PCIe NICs for comparison.

Further to your various comments about ARM SoCs with gigabit MACs widely having trouble obtaining good throughput over this fast medium, Olimex's latest blog post on "A20-SOM Update"  confirms this for A20 as well:

OLIMEX Ltd wrote:

 

What is a bit dissapointing is that Gigabit Ethernet can't achieve more than 330Mbit/s transfer with the current state of the drivers, there were few comments that this may improve a bit later, but ARMs seems not quite capable with Gigabit Interfaces if you read for other popular ARMs like imx6 also can't have full throughput of Gigabit Ethernet.

So,  it seems to be a fairly common issue.  I wonder if there's any ARM SoC at all with gigabit throughput in the 900's of Mbps?  Maybe we should ask ARM whether there is any relevant limit in the bus fabric or CPU of modern ARMs at this speed, or whether the measured limitations are entirely due to peripheral issues in the licensees' SoCs.

(I wouldn't expect an answer though.  ARM aren't known for their openness or willingness to engage the community.)

Morgaine Dinova wrote:

 

So,  it seems to be a fairly common issue.  I wonder if there's any ARM SoC at all with gigabit throughput in the 900's of Mbps? 

I think it could be interesting to get a PCIe network card with known functional drivers connected to some Arm board that has PCIe.. That might give some idea if it's just a driver issue or something more fundamental  The errata for the i.MX6 suggested that it was a limitation on an internal bus somewhere, but didn't give much more detail than that.

I may have to try and see if I have a PCIe NIC that'll work without the differential clock available. That was my original problem with the Sabre Lite, the PCIe header doesn't include the clock so you'd have to add an external chip to provide it.. I could find buffers for an existing clock, but not a clock generator.

I've described our data gathering over in the Parallella forum under Technical Q & A: "Ethernet throughput measurements".  It's possible that some of those who have already received their Parallella boards will want to measure their Ethernet throughput and contribute their results, which could be very interesting.

Since the Zynq is not a low-end mass market SoC and has some very fast internal buses, I think there's a very good chance that this might be the first ARM device encountered that approaches the maximum theoretical throughput for gigabit Ethernet.

Anyway, we'll see.  

We now have our first data pair for the Parallella's Zynq over gigabit Ethernet, courtesy of notzed.

There were some severe networking problems encountered during notzed's tests though, and I expect that these figures will rise to over 400Mbps once those issues are resolved.  Andreas posted in that thread a link to Xilinx Application Note XAPP1082, in which Ethernet throughput is specified over the full range of frame payload sizes, and it tops out at just over 400Mbps without jumbo frames --- a throughput also seen by notzed in one test.  A different test utility is being used though.

At this point I haven't yet understood the test conditions in that document properly, so I'm not sure if we're comparing like with like yet.  Anyway, this is a start, and any measurements are better than none.

Morgaine Dinova wrote:

A different test utility is being used though.

netperf is probably the better tool to use, it's more widely known and therefore easier to compare results.  It's also a lot more complex.

nuttcp on the other hand is simple to build, doesn't appear to have any dependencies, is simple to use etc. Probably the biggest drawback is that we can't be sure the results compare properly with netperf.

@Morgaine,

I can't quite match your reading of the Xilinx app note.

On page 13, fig 10 the inbound performance is 679MBps for MTU = 1500,

On page 12 fig 9 they show 683 for what I take to be a standard setup in terms of MTU.

I can't find your 400Mbps figure .

MK

Morgaine Dinova wrote:

 

We now have our first data pair for the Parallella's Zynq over gigabit Ethernet, courtesy of notzed.

Just had a quick read at that, but as a suggestion, make sure that any sort of powersave is turned off and that cpu frequency governors at both ends are forced to performance.  I've seen variations in results where one end has a governor slowing down the clock because it sees little demand

Michael Kellett wrote:

 

I can't quite match your reading of the Xilinx app note.

 

On page 13, fig 10 the inbound performance is 679MBps for MTU = 1500,

On page 12 fig 9 they show 683 for what I take to be a standard setup in terms of MTU.

 

I can't find your 400Mbps figure .

It's from the graphs of throughput versus message size on pages 10 and 11, respectively for Processing System (PS) and Programmable Logic (PL) throughput as defined on page 1.  The AXI Ethernet bar graphs on page 13 may provide the more relevant figures (I hope so!), but I'm still figuring it out at this point.  I certainly expected far better figures than we saw for gigabit i.MX6 and AM3359, because Zynq is designed with high speed internal buses and for applications which need to feed the FPGA at high rate.

Note that notzed's figures are nowhere near the 680 Mbps mark, although that could be due to a local fault.

Addendum:  The Xilinx App Note says on page 2:

Figure 1 shows the various Ethernet implementations on the ZC706 board.

Note:  The three Ethernet links cannot be active at the same time because the ZC706 board offers only one SFP cage for the 1000BASE-X PHY. The PS-GEM0 is always tied to the RGMII Marvell PHY. The PS-GEM1 and the PL Ethernet shar e the 1000 BASE-X PHY so only two Et hernet Links can be active at a given time.

Unfortunately I haven't yet been able to figure out from the Gen1 reference and schematic which of the three is actually in use on Parallella.

Thanks. BTW - the page 11 isn't using jumbo frames but just increasing the message size using normal sized frames. The Xilinx note about the dip in performance confirms this. The max performance still isn't that good.

MK