First step: Prepare your Tor relay to handle a large number of connections.
There are a number of optimisations you can do to improve the resource efficiency of your Tor relay. These and many other methods are used for web servers that have to handle very high loads.
In addition to the basic CPU, memory, and swap optimisation settings, I have listed the Optimised Network Stack settings here.
As always, please note that some of the options may already be in your sysctl.conf. Please edit them without duplicating them.
Here are the settings:
net.ipv4.tcp_fin_timeout = 20
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
The default value for the net.ipv4.tcp_fin_timeout setting on a Debian Linux system is usually 60 seconds. This value specifies how long the system remains in the FIN-WAIT-2 state after closing a TCP connection before the connection is finally closed.
The setting net.ipv4.tcp_fin_timeout on a Debian system specifies the time (in seconds) how long the system remains in the FIN-WAIT-2 state after sending a FIN packet before the connection is finally closed. The FIN-WAIT-2 state occurs when a TCP connection is closed by one side and the other side has not yet responded to the FIN message.
Effects of the setting net.ipv4.tcp_fin_timeout = 20:
Faster release of resources:
Reducing the tcp_fin_timeout value to 20 seconds means that the system resources used by the TCP connection (e.g. memory for connection status information) are released more quickly after the connection is closed.
Reduced number of open connections:
This reduces the number of connections that are in the FIN-WAIT-2 state. This can be particularly helpful on systems with many short TCP connections in order to limit the number of open connections.
Possible connection problems:
If a remote peer takes longer than 20 seconds to respond to the FIN packet, the connection could be closed unexpectedly. This can lead to problems if applications on the remote peer need this extra time to close properly.
Security aspects:
A shorter tcp_fin_timeout value can reduce the likelihood of Denial of Service (DoS) attacks, where a large number of half-closed connections are deliberately kept open to exhaust the server's resources.
The default value for the net.ipv4.tcp_keepalive_time setting on a Debian Linux system is usually 7200 seconds, which corresponds to 2 hours. This value specifies how long a system waits in an inactive TCP connection before it sends the first keepalive packet to check whether the connection is still active.
The setting net.ipv4.tcp_keepalive_time = 1200 on a Debian system specifies that the system will send a keepalive packet after 1200 seconds (20 minutes) of inactivity to check if the TCP connection is still active. Here are the effects of this change in detail:
Effects of setting net.ipv4.tcp_keepalive_time = 1200:
Faster detection of inactive connections:
With a keepalive interval of 20 minutes instead of the default value of 2 hours (7200 seconds), the system will detect inactive connections faster. This can be useful to terminate connections that are no longer actively used.
Better resource management:
Sending keepalive packets faster helps save resources by detecting and closing inactive connections sooner. This is particularly beneficial in environments with many connections where it is important to quickly identify and remove unneeded connections.
Faster response to network problems:
If a connection has been interrupted due to a network problem, the faster keepalive interval can help to recognise this interruption earlier and take appropriate action, such as starting a new connection attempt or reporting the connection.
Increased network traffic:
A disadvantage of reducing the tcp_keepalive_time value is the increased network traffic due to the more frequent keepalive packets. In environments with many connections, this can lead to a slight increase in the network load.
The default value for the net.ipv4.tcp_syncookies setting on a Debian Linux system is normally 1. This setting enables or disables TCP syncookies, which are a security measure against SYN flood attacks.
Meaning of net.ipv4.tcp_syncookies:
Value 1:
TCP syncookies are activated. This means that in the event of a SYN flood attack (a type of denial of service attack in which a large number of SYN requests are sent to exhaust the system's resources), the system uses syncookies to accept legitimate connections without consuming memory for half-opened connections.
Value 0:
TCP syncookies are disabled. This is not recommended as the system becomes more vulnerable to SYN flood attacks.
The setting net.ipv4.tcp_syncookies = 1 on a Debian system enables the use of TCP syncookies. This is a security measure that helps to protect the system against SYN flood attacks. Here is a detailed
explanation of the effects of this setting:
Effects of net.ipv4.tcp_syncookies = 1:
Protection against SYN flood attacks:
A SYN flood attack is a form of denial of service attack in which an attacker sends a large number of SYN packets (connection requests) to a server without completing the handshake procedure. This can lead to the server's connection queue overflowing and legitimate connections being rejected.
With tcp_syncookies enabled, the system can continue to accept new connections even when the connection queue is full by using so-called syncookies.
How syncookies work:
When the connection queue for half-opened connections is full, the server sends back a special SYN-ACK packet with an encrypted cookie in the Initial Sequence Number (ISN). This cookie contains information about the connection without the server having to reserve memory for the half-open connection.
When the client responds with an ACK packet, the server can check the cookie and establish the connection if the cookie is valid.
Efficiency and resource utilisation:
Using syncookies does not require additional resources on the server to keep track of half-opened connections, keeping the system stable even under load conditions (such as a SYN flood attack).
Compatibility:
Syncookies are an effective method of defence against SYN flood attacks, but they are only an emergency measure. They ensure that the system can continue to accept connections without the connection queue overflowing.
In some rare cases, problems may occur with certain TCP options that cannot be integrated into the cookie. However, this should not be a significant problem in most modern networks.
Summary:
The setting net.ipv4.tcp_syncookies = 1 ensures that the system:
Protected against SYN flood attacks by continuing to accept new connections even when the connection queue is full.
Uses resources efficiently by not reserving memory for half-opened connections.
Uses a special cookie procedure to check the validity of incoming connections.
The default value for the setting net.ipv4.tcp_tw_reuse on a Debian Linux system is usually 0. This setting specifies whether the system allows the reuse of TCP connections in the TIME_WAIT state.
Meaning of net.ipv4.tcp_tw_reuse:
Value 0:
The reuse of TCP connections in the TIME_WAIT state is deactivated. This means that the system waits for the normal timeout for these connections before releasing the resources. This is the safe and default mode to avoid possible problems caused by the reuse of connections.
Value 1:
The reuse of TCP connections in the TIME_WAIT state is enabled. This can be useful in certain high-load scenarios to reduce the number of connections in the TIME_WAIT state and release resources more quickly. However, this can also lead to problems if connections are reused prematurely.
The setting net.ipv4.tcp_tw_reuse = 1 on a Debian system allows the reuse of TCP connections in the TIME_WAIT state. This configuration can improve resource utilisation and performance in networks with many ephemeral connections. Here are the specific implications and considerations in detail:
Impact of net.ipv4.tcp_tw_reuse = 1:
Faster reuse of connections:
Connections in the TIME_WAIT state can be reused instead of waiting for the TIME_WAIT timeout to expire. This reduces the number of connections in the TIME_WAIT state and speeds up the release of system resources.
Improved resource utilisation:
In systems with a high number of connections that are opened and closed quickly (e.g. web servers with many clients), enabling this setting can improve performance as less memory and other resources are required for connections in the TIME_WAIT state.
Possible security risks:
Reusing connections in the TIME_WAIT state can lead to security issues, especially if a new connection overlaps an old connection and data conflicts occur. This can potentially lead to data integrity issues and security vulnerabilities.
Network stability:
In networks with properly configured clients and servers, the reuse of TIME_WAIT connections should be stable. However, in less controlled environments this can lead to unexpected behaviour.
The default value for net.ipv4.ip_local_port_range on a Debian Linux system is in the range of 32768 to 60999.The default value for net.ipv4.ip_local_port_range on a Debian Linux system is in the range of 32768 to 60999.
The setting net.ipv4.ip_local_port_range = 10000 65000 on a Debian Linux system causes the range of dynamic or temporary ports used by the system for outgoing connections to be changed. Specifically, this means the following:
Dynamic port range: When an application initiates an outgoing connection and does not specify a specific source port, the system automatically selects a free port from this defined range. The change to 10000 65000 increases the available pool of ports.
Increasing the available ports: The standard range of 32768-60999 offers approximately 28231 ports. By expanding to 10000-65000, 55001 ports are now available. This can be particularly useful when many outgoing connections are required on the system, for example for servers with high network traffic or for certain applications that establish many temporary connections.
Reduction of port collisions: A larger range reduces the risk of port collisions where two outgoing connections want to use the same source port. This is particularly relevant in high-traffic network environments.
The default value for net.ipv4.tcp_max_syn_backlog on a Debian Linux system varies depending on the version of the kernel and the specific system configuration. However, a common default value for many systems is 128.
This parameter defines the maximum number of half-open connections (connections in SYN_RECV state) that the system can hold in the queue. This is important for establishing connections via the TCP protocol, especially under high load.
The setting net.ipv4.tcp_max_syn_backlog = 8192 on a Debian Linux system has the following detailed effects:
Increasing the queue for incoming connections: The value 8192 specifies the maximum number of half-open TCP connections that the system can hold in the queue. Half-open connections are in the SYN_RECV state, which means that the server has received the SYN packet from the client and sent back the SYN-ACK packet, but has not yet received an ACK packet from the client.
Improved handling of high connection rates: A higher value for tcp_max_syn_backlog means that the system can handle more incoming connections simultaneously before it starts rejecting new connections. This is particularly useful in environments with high network traffic, such as web servers that receive many simultaneous connection requests.
Reducing the risk of SYN flood attacks: In a SYN flood attack, an attacker sends many SYN packets without completing the connection to exhaust the queue resource. A higher value for tcp_max_syn_backlog can help mitigate such attacks by allowing the system to process a larger number of half-open connections. However, it is also important to use other security mechanisms such as SYN cookies.
Increasing resource consumption: Holding a larger number of half-open connections requires more memory and other system resources. Therefore, the value should be chosen carefully to ensure that the system has enough resources to handle the increased queue without affecting performance.
The default value for net.ipv4.tcp_max_tw_buckets on a Debian Linux system is usually 180000.
This parameter determines the maximum number of TCP time-wait (TIME-WAIT) states that the system can hold simultaneously. The TIME-WAIT state occurs when a TCP connection has been closed and the system ensures that all packets from the old connection have expired before the same source and destination addresses and port numbers can be reused.
The setting net.ipv4.tcp_max_tw_buckets = 5000 on a Debian Linux system has the following detailed effects:
Limitation of TIME-WAIT states: This setting specifies that a maximum of 5000 TCP connections may be in the TIME-WAIT state at the same time. The TIME-WAIT state occurs after a TCP connection has been closed to ensure that all remaining packets from the old connection have disappeared from the network before the same combination of source and destination addresses and port numbers can be reused.
Resource management: As every TCP connection in the TIME-WAIT state consumes memory resources (especially kernel memory), this setting limits the number of TIME-WAIT entries in order to control memory consumption. When the limit of 5000 TIME-WAIT connections is reached, the system will start to remove older entries.
Increased risk of port reuse: By limiting to 5000 TIME-WAIT states, ports may be reused earlier more frequently, increasing the risk of "port reuse" issues. This can lead to potential conflicts if old connection data packets are still travelling on the network and are mistakenly interpreted as part of a new connection.
Performance in high load environments: In environments with high network traffic, such as busy web servers, setting tcp_max_tw_buckets too low can lead to performance issues as the system often has to clean up TIME-WAIT entries to make room for new connections. This can impair the efficiency of connection processing.
Warning messages in the system log: When the system reaches the maximum number of TIME-WAIT states and begins to remove older entries, warning messages may appear in the system log (/var/log/syslog or /var/log/messages). These messages serve as an indicator that the system is reaching its limit for TIME-WAIT states.