Linux performance and analysis tools

Hi Drew,

I've not used all these, just some, but it looks about right. I've never seen it in a diagram form, but maybe that will help people.

Although the above is perhaps intended for Linux perf, the commands are still useful for user application debugging, when those apps for example eventually call Linux library functions. But generally for app debugging there are other tools too of course, gdb being a prominent one of course. And for app perf, there are tools like google perftools.

Also some other general useful tools on my list would be nm, tshark (prefer it to just using tcpdump), file, ldd.

perf seems to have become the 'swiss army knife' for analysing all sorts of stuff in linux.

The one I'd probably question is netstat. The linux networking stack is actually very much more capable than tools like netstat can describe. You'd want to use things like ctstat, rtstat, ifstat, lnstat, nstat, rtacct, ss from the iproute2 package to get a better view of what's going on.

Similarly, it's missing things like mtr, traceroute, tracepath, tcptraceroute on the network side and I'm sure there are many others.

Possibly not directly applicable to SBC's, but there's things like powertop and tools/x86/turbostat in the kernel source that any Arm equivalents of would be interesting to use to analyse what pointless things the system is wasting power on.

The list will probably get huge - depends on exactly what things you want to examine..

thanks for the tip on tshark!  I use Ethereal/Wireshark GUI but didn't realize there was a terminal program other than tcpdump

Hi Drew!

Yes, it is really great and super-simple to use. I have some personal notes on it, pasted below.

To display packets in real time, the command (as root user) is (you may have a different path to it):

/usr/sbin/tshark -i eth0

Output:

Running as user "root" and group "root". This could be dangerous.
Capturing on eth0
0.000000 Oracle_2b:14:b3 -> Broadcast    ARP Who has 10.10.71.1?  Tell 10.10.71.12
0.459221 Oracle_16:23:b6 -> Broadcast    ARP Who has 10.10.71.65?  Tell 10.10.71.90
0.682997     10.0.0.0 -> 224.0.0.1    IGMP V3 Membership Query, general
0.708566 10.10.67.233 -> 10.10.65.49 SSH Encrypted response packet len=132
0.708680 10.10.67.233 -> 10.10.65.49 SSH Encrypted response packet len=132
0.708787 10.10.67.233 -> 10.10.65.49 SSH Encrypted response packet len=116

A ‘display filter’ is useful:

/usr/sbin/tshark -i eth0 -R "ip.addr==10.10.66.57"

Another example is:

/usr/sbin/tshark -i eth0 -R "udp.port==5000"

This will capture everything:

/usr/sbin/tshark -i eth0 -w test.cap

To capture and display in real time (most useful), the ‘-S’ switch is used:

/usr/sbin/tshark -i eth0 -w test.cap –S -R "ip.addr==10.10.66.57”

Note that the interface ‘eth0’ can be replaced with ‘any’, which will allow traffic across ports on the same machine to be captured. This is very useful to capture traffic between components running on the same Linux platform.

To capture only certain packets, a ‘capture filter’ can be specified. However, please note that the capture filter syntax is unfortunately different from display filter syntax. This example also shows how to write to a file.

/usr/sbin/tshark -i eth0 -f "port 5000" –w test.cap

To view the contents of the captured file, either Wireshark can be used, or a display filter can again be used with tshark:

/usr/sbin/tshark -r test.cap -R "ip.addr==10.10.66.57"

Output:

Running as user "root" and group "root". This could be dangerous.
32 5.681169  10.10.66.57 -> 10.10.67.233 SIP Request: REGISTER sip:abc.test
39 5.684898 10.10.67.233 -> 10.10.66.57 SIP Status: 403 Forbidden - User Unknown    (0 bindings)

Thanks!