Much of network performance analysis will be comparative in nature. Thus, seeing the output of multiple commands side by side can be quite useful. Bash has a useful little utility called pr and we’ll make use of it.

Side-by-side traceroutes

Step 1: create hostlist.txt and populate it with IPs or hostnames of the servers you want to traceroute. Example:

Step 2: declare some functions and variables

Step 3: parse the hostlist.txt and generate traceroute commands for each host:

Step 4: insert the line generated in Step 3 into the script below, as appropriate:

Step 5: runs trace function and you should see something like the example below. Note that the egrep --color line in the trace function will highlight devices showing latency in triple-digits: s+[0-9]{3,}, which you can modify as needed. The traceroute loop will continue until you cancel it.

So far we’ve been testing multiple remote hosts from the same local server. We may also need to test the same remote host from multiple local servers. Here’s a quick example that requires passwordless root SSH, which you can enable temporarily. Also note that I adjusted the egrep --color regex to highlight nodes with latency of 30ms or greater.

And here’s a sample output:

More side-by-side examples

While I am on the subject of pr, here are a few more example of how this command can be useful. In this example we’re comparing primary interface stats for the localhost and a remote server:

And the sample output:

Another option for viewing output of multiple commands side by side is vimdiff (up to four side-by-side windows). Here’s an example comparing kernel routing table on two remote servers:

Network bandwidth tests

So you want to know how fast your gigabit link really is in practical terms? There are a couple of simple tools that can help.

You can use the pv utility if you have SSH connectivity between the nodes. The syntax is very simple:

A more sophisticated test can be performed using iperf. With this tool you can use different ports, run a bi-directional transfer, specify multiple threads, test with UDP, and much more. Here’s a basic example:

The ability of iperf to initiate multiple transfer threads is very useful, as some network performance issues may result in degraded speed on per-connection basis. Running multiple threads will show if the overall network throughput is limited or if it is close to maximum available bandwidth.

A good example here would be the TCP window overflow problem when the receiver for whatever reason runs out of the receive buffer space for individual connections.

You may also want to transfer some real-world data using rsync and use that for your testing. In the following example we generate some sample data and run rsync over SSH:

If you’re looking to just test your Internet connection link, check out this article. And here I have fairly complete listing of various CLI system monitoring tools (including network).

Running tcpdump

While your bandwidth test is running and you’re waiting, it would make sense to capture some packets between source and target servers for further analysis. Ideally, you want to capture traffic from both sides of the data transfer simultaneously and limit captured data to the two IPs and the specific port.

Of course, the tcpdump command has its own timeout feature, I found that doesn’t always work as expected and using the timeout command is just more reliable.

Leave A Reply

Please enter your comment!
Please enter your name here