[Nexus 5 Android Development] rawe_etc:
The short summary:

TCP performance tuning - how to tune linux

The default Linux tcp window sizing parameters before 2.6.17 sucks.

The short fix [wirespeed for gigE within 5 ms RTT and fastE within 50 ms RTT]:

in /etc/sysctl.conf

net/core/rmem_max = 8738000

net/core/wmem_max = 6553600

net/ipv4/tcp_rmem = 8192 873800 8738000

net/ipv4/tcp_wmem = 4096 655360 6553600

It might also be a good idea to increase vm/min_free_kbytes, especially

if you have e1000 with NAPI or similar. A sensible value is 16M or 64M:

vm/min_free_kbytes = 65536

If you run an ancient kernel, increase the txqueuelen to at least 1000:

ifconfig ethN txqueuelen 1000

If you are seeing "TCP: drop open request" for real load (not a DDoS),

you need to increase tcp_max_syn_backlog (8192 worked much better than

1024 on heavy webserver load).

The background:

TCP performance is limited by latency and window size (and overhead, which

reduces the effective window size) by window_size/RTT (this is how much data

that can be "in transit" over the link at any given moment).

To get the actual transfer speeds possible you have to divide the resulting

window by the latency (in seconds):

The overhead is: window/2^tcp_adv_win_scale (tcp_adv_win_scale default is 2)

So for linux default parameters for the recieve window (tcp_rmem):

87380 - (87380 / 2^2) = 65536.

Given a transatlantic link (150 ms RTT), the maximum performance ends up at:

65536/0.150 = 436906 bytes/s or about 400 kbyte/s, which is really slow today.

With the increased default size:

(873800 - 873800/2^2)/0.150 = 4369000 bytes/s, or about 4Mbytes/s, which

is resonable for a modern network. And note that this is the default, if

the sender is configured with a larger window size it will happily scale

up to 10 times this (8738000*0.75/0.150 = ~40Mbytes/s), pretty good for

a modern network.

2.6.17 and later have resonably good defaults values, and actually tune

the window size up to the max allowed, if the other side supports it. So

since then most of this guide is not needed. For good long-haul throughput

the maxiumum value might need to be increased though.

For the txqueuelen, this is mostly relevant for gigE, but should not hurt

anything else. Old kernels have shipped with a default txqueuelen of 100,

which is definately too low and hurts performance.

net/core/[rw]mem_max is in bytes, and the largest possible window size.

net/ipv4/tcp_[rw]mem is in bytes and is "min default max" for the tcp

windows, this is negotiated between both sender and reciever. "r" is for

when this machine is on the recieving end, "w" when the connection is

initiated from this machine.

There are more tuning parameters, for the Linux kernel they are documented

in Documentation/networking/ip-sysctl.txt, but in our experience only the

parameters above need tuning to get good tcp performance..

Know How

list the sysctl’s that usually have impact on network performance:

net.nf_conntrack_max

maximum number of tracked connection in iptables

net.ipv4.ip_local_port_range

range of ports used for outgoing TCP connections (useful to change it if you have a lot of outgoing connections from host)

net.ipv4.tcp_rmem

autotuning tcp receive buffer parameters

net.ipv4.tcp_wmem

autotuning tcp send buffer parameters

net.ipv4.rmem_max

maximum tcp socket receive buffer memory size (in bytes)

net.ipv4.wmem_max

maximum tcp socket send buffer memory size (in bytes)

net.ipv4.tcp_mem

TCP buffer memory usage thresholds for autotuning, in memory pages (1 page = 4kb)

net.core.somaxconn

maximum listen queue size for sockets (useful and often overlooked setting for loadbalancers, webservers and application servers (like unicorn, php-fpm). If all server processes/threads are busy, then incoming client connections are put in “backlog” waiting for being served). Full backlog causes client connections to be immediately rejected, causing client error.

net.core.dev_weight

maximum number of frames that kernel may drain from device queue during one interrupt cycle (setting higher value will allow more packets to be processed during one softirq cycle, but longer gaps between CPU availability for applications)

net.core.netdev_max_backlog

number of slots in the receiver's ring buffer for arriving packets (kernel put packets in this queue if the CPU is not available to process them, for example by application)

net.ipv4.tcp_keepalive_time

interval in seconds between last packet sent and first keepalive probe sent by linux

net.ipv4.tcp_keepalive_probes

number of unacknowledged keepalive probe packets to send before