Application latency due to network saturation

When an application has high latency, look for network saturation if you have already:

Found no OOM killing or CPU throttling
Increased CPU and memory

The limitation of bandwidth for an application can cause dropped packets.

Navigate to the Network tab to view bandwidth and saturation.

Bandwidth limits and saturation

In the following example, these graphs shows an approximate 6-hour time period for a Cluster.

The Network Bandwidth by node graph indicates a harmonic effect resembling a straight line. The peaks do not extend beyond 2 GiB/s, perhaps because the network interfaces of those nodes are at bandwidth capacity.

The Network Saturation by node graph shows many spikes on the receive side (upper half), indicating saturation and dropped packets. This can lead to request queueing, packet retries, and increased response latencies as a result.

Approximate six-hour time period showing network saturation for a Cluster

A normal set of graphs for an approximate six-hour period shows no harmonic effect for bandwidth and little activity for saturation.

Approximate six-hour time period showing no saturation and normal bandwidth for a Cluster

Resolution for network saturation

To increase bandwidth and eliminate saturation, increase the number of Nodes where the application is deployed. You can then recheck the Network tab to see the results.