TCP deadlock problems

When a TCP connection is made, the TCP mechanism creates two buffers at each end of the connection, an input buffer and an output buffer, for a total of four buffers.

If the client sends a request to the server, the request is copied into the client's output buffer. From there the request moves through the TCP connection to the server's input buffer. Note that this clears the client's output buffer. When the server reads the request from its input buffer, that buffer is also cleared. The server then reacts to the request and sends a response, which is written to the server's output buffer. The response then moves through the TCP connection to the client's input buffer, clearing the server's output buffer. The client then reads the response, clearing its input buffer. The cycle is completed and all buffers are clear.

Under certain circumstances, however, a TCP connection can end up in a "deadlock", where neither the client nor the server is able to write data out or read data in. This is caused by two factors. First, a client or server cannot perform two transactions at once; a read cannot be performed if a write transaction is in progress, and vice versa. Second, the buffers that exist at either end of the TCP connection are of limited size. The deadlock occurs when both the client and server are trying to send an amount of data that is larger than the combined input and output buffer size.

For example, let's assume that the buffers on both ends of a TCP connection have a maximum size of 100KB. The client sends a 220KB request to the server which fills both the client's output buffer and the server's input buffer. When the server reads the first 10KB of that request, it begins to send a 210KB response. After the first 200KB of the response is sent, the client's input buffer and the server's output buffer are both full, and the server cannot finish sending the response. The client hasn't finished sending the request, however, because the server's input buffer is also full. Since the server cannot read and write at the same time, it is stuck. Similarly, the client cannot read and write at the same time, so it is also stuck. A deadlock has formed, and there is no further communication across the TCP connection.

Note that this deadlock is a common problem encountered when using TCP connections in general, and is not specific to TCP connections created using OmniMark. In the technical literature, this is known as the Bounded Buffer Deadlock Problem.

If you use the OmniMark tcp.set-buffering function and specify unbuffered input and output, you will need to programmatically guard against this problem. The usual method for this is either to fully input the data prior to doing any processing or to fully buffer the output data prior to transmitting it.

If you use the TCP library with buffering specified on HP-UX or Windows, the library implements an unbounded output buffer solution for you, and this deadlocking problem will not occur. Note, however, that on other platforms you will have to provide a programmatic solution as outlined above.