tcp-gateway, how to end tcp-communication

[posta07]

I am trying to use the tcp-inbound-gateway implementation to implement a request/response with an external, TCP-based client, and our spring integration based messaging system.

The requirements call for the client to make a request, the messaging system to handle the request, and for the response to be sent back to the client on the same TCP socket.

I have scoured through the documentation, and the source code for many hours to try and figure out how best to solve the issue I'm seeing, but I'm reaching out to those that developed it to help me understand something I may be missing.

Everything happens correctly in that the tcp-inbound-gateway properly receives the message on the declared port, processes it, and sends back a response, however it seems the port does not get closed properly after the response is sent; it eventually times out, but the client receives a SocketException as a result of the timeout instead of the response.

I've stepped through the sample tcp client/server project, but that seems to implement telnet where the connection properly does stay open while the client and server communicate back and forth, but I have not seen any examples of where the socket gets closed after a response, nor can i discern which is the appropriate way to go given my analysis of the source.

Can someone help here?
I can provide more details (config, etc) as necessary, but I am specifically using the regular socket implementation (not NIO) and setting single-use= true.

Thanks in advance.

[Gary Russell]

Sorry for the delay, I was on a cross-country flight.

So here's the deal...

...it seems the port does not get closed properly after the response is sent...

The server can't close the socket immediately after sending the response, because the FIN packet (close) might get propagated back to the client before the data packet(s).

In this type of environment, it is normal for the client to close the socket after it receives the response. The server expects this, and will not complain when it detects the socket close.

If, however, the client does not close the socket, we do what you have observed, when we get a read timeout, we close the socket. This is to protect the server from mis-behaved clients that might leave "single-use" sockets open.

For single-use sockets, if no timeout is provided, we default to 10 seconds; this should be plenty to ensure the FIN doesn't beat the response to the client and, again, we're protecting the server because the normal default timeout is infinity.


I am a little confused by your comments...

...the tcp-inbound-gateway properly receives the message on the declared port, processes it, and sends back a response...

...but the client receives a SocketException as a result of the timeout instead of the response....

It seems you are saying we send the response OK, but you are implying the client doesn't get it; instead he later detects the socket closure after the timeout.

That doesn't make sense to me, unless there is a problem with the protocol you are using for the response - and the client has read the response but is still expecting more data for some reason (because there is a mismatch in the wire protocol for the response compared to what he is expecting).

You can verify what going on by using wireshark or similar to get a tcp trace (you will need to run the client on a different machine, though; localhost won't work).

What format is the client expecting the response to have?

[Gary Russell]

BTW note that, in the sample, single-use=true is set on the client side and each test is run over a separate socket...

...
DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Closing single use socket after inbound message localhost:11111:1653578290
...
DEBUG: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Closing single use socket after inbound message localhost:11111:803660549
...

These logs are coming from the client side, of course.

In this sample, we don't set single-use=true on the server side because the other part of the sample demonstrates a telnet server where we want the connection to remain open.

single-use=true doesn't do much on the server side (for a gateway); it just suppresses the error when the socket times out, and logs it as debug instead.

For an inbound channel adapter, where no reply is sent, single-use=true causes the server to close the socket after a message is received.

I hope that further clarifies things.

[posta07]

Hi Gary,

Thanks for your detailed responses.

It seems you are saying we send the response OK, but you are implying the client doesn't get it; instead he later detects the socket closure after the timeout.

That doesn't make sense to me, unless there is a problem with the protocol you are using for the response - and the client has read the response but is still expecting more data for some reason (because there is a mismatch in the wire protocol for the response compared to what he is expecting).

Yes, the implication you picked up on is correct; I should have been more clear. The TCP gateway (and associated connection classes, etc) does log that the response was sent, but the client doesn't see a response. Your suggestion of using wireshark to review what actually gets passed is a good idea.

The type of data sent through the TCP port is a COBOL copybook string. We have created appropriate serializers/deserializers to handle this format. In our legacy java code that does this TCP handling, I noticed it closes the socket after the response is sent. I'm not sure if this is appropriate behavior as you've pointed out the FIN packet could arrive before the data packets.

We've always used jmeter, or some home-grown socket testing tool to test the TCP handler we have, and I was hoping to use that to also test this implementation with SI and the TCP gateway. Effectively, the client apps we use to test (jmeter, etc) view the data being passed to and from the TCP handler as 'strings' with no particular protocol or format.

[Gary Russell]

I noticed it closes the socket after the response is sent. I'm not sure if this is appropriate behavior as you've pointed out the FIN packet could arrive before the data packets.

Yes; for the most part, it's never going to be an issue but once in a great while it *will* happen and you'll spend a long time figuring out what happened. Typically, you only have to be burned by this once in your lifetime and you'll never make the mistake again :-)

...view the data being passed to and from the TCP handler as 'strings' with no particular protocol or format.

Tcp data is a stream so you *have* to have some mechanism that will allow the recipient to know when it has received a message, whether it's a simple fixed COBOL-like format (all message are, say, 100 bytes long), or whether you add something to the data, such as [data]<CRLF>, <STX>[data]<ETX>, <length>[data] etc, etc

If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.

[posta07]

If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.

This makes perfect sense.

I'll take a second look at the legacy code we are using here to see how it accomplished notifying the client that the stream of data can be considered 'ended.'

BTW... thanks much for your quick and detailed responses.

[posta07]

If you are seeing the data go out, I *have* to believe the client is having a problem interpreting the data, and is expecting more to arrive when the socket is closed.

Yep. That's exactly what's happening.
Using wireshark, I'm able to see that the TcpInboundGateway classes (connections, connection factories, etc) indeed send back the response, but the client (jmeter for these tests) does not know where the end of the stream is... it then continues to wait as you suggested and then the server times out. With our legacy code, jmeter (and other clients we use for testing) will correctly realize the end of the response by the FIN message sent when the server closes the socket.

The legacy code (which we have to keep the same at this point... we're lucky enough that we can do any refactoring, but we won't be able to change the behavior at this point) also does not delimit or demarcate the end of the TCP stream; it uses the fact that closing the socket will send the FIN message to the client as the delimiter. To integrate with our SI messaging, I may have to look closer at the TCP channel adapters or custom channel adapters instead of the TcpInboundGateway, which I was really hoping to use.

I can try to make the case for forcing the client to know how to demarcate the stream (as I would tend to agree with you), but I cannot find much documentation detailing the possibility that the FIN message might be propagated to the client before the data. Can you help me understand better why that would happen (I realize the FIN message can get there earlier, but does the TCP transport not realize that there may be more data and to continue trying to gather all the relevant packets?)? I can't seem to find any blogs or forum threads that detail this possible issue. Maybe I'm not using the correct search terms, so any guidance in finding documentation would be helpful in presenting my findings to those that can make a decision about changing the behavior of our legacy code.
Thanks Gary!

[Gary Russell]

Interesting - I just re-read the TCP RFC (it's been a few years) and it does, indeed, say that it should be ok to send a FIN after data and the remote TCP should reassemble things and take the FIN after the data, even if the FIN arrives first.

All I can say is that I have certainly had it happen in my past; perhaps there was a bug in the TCP stack I was using; perhaps there was a buggy router - I did find some references on the 'net where routers dropped data packets after seeing a FIN.

Let me look into it some more.

In the meantime, I am a little confused as to why the default behavior does not help you out, though. When we get the read error (no more inbound messages), we go ahead and close the socket anyway, which will send the FIN, albeit 10 seconds (by default) after your data. Maybe JMeter is timing out before the 10 seconds?

Perhaps you could try, say, setting so-timeout="100"; this will cause us to close the socket 100 milliseconds after the write.

You need to be careful not to set the timeout so low that we might timeout during reads on a busy network (assuming the inbound message is large enough to get segmented). The same timeout is used for both situations.

Before giving up on the gateway, you might also consider adding an interceptor to explicitly close the socket after the write. I realize this is an advanced feature and, although documented, there are no samples. There is, however, a HelloWorld handshaking interceptor in the test cases for the ip module.

I can help you with the interceptor but, in essence, you would extend AbstractTcpConnectionInterceptor, overriding the send() method something like as follows...

	@Override
	public void send(Message<?> message) throws Exception {
		super.send(message);
                this.close();
	}

I have not tested this, so no warranties :-)

If my investigations point me to it being OK to do the close, I will, of course, make the change - I am not sure when it will be available in a release, though.

[Gary Russell]

Here is the code for the interceptor approach - I added it to the tcp-client-server-sample and it worked fine. The only problem with it is it causes the gateway to emit an ERROR log when it detects the socket closed unexpectedly.

ERROR: org.springframework.integration.ip.tcp.connection.TcpNetConnection - Read exception localhost:46020:1579948023 SocketException:null:Socket is closed

package org.springframework.integration.samples.tcpclientserver;

import org.springframework.integration.Message;
import org.springframework.integration.ip.tcp.connection.AbstractTcpConnectionInterceptor;
import org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptor;
import org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptorFactory;

/**
 * @author Gary Russell
 *
 */
public class SocketClosingInterceptorFactory implements
		TcpConnectionInterceptorFactory {

	public TcpConnectionInterceptor getInterceptor() {
		
		return new AbstractTcpConnectionInterceptor() {
			@Override
			public void send(Message<?> message) throws Exception {
				super.send(message);
				this.close();
			}			
		};
	}

}

	<ip:tcp-connection-factory id="crLfServer"
		type="server"
                single-use="true"
		interceptor-factory-chain="socketClosingFactoryChain"
		port="11111"/>
			
	<beans:bean id="socketClosingFactoryChain" class="org.springframework.integration.ip.tcp.connection.TcpConnectionInterceptorFactoryChain">
		<beans:property name="interceptors">
			<beans:array>
				<beans:bean class="org.springframework.integration.samples.tcpclientserver.SocketClosingInterceptorFactory"/>
			</beans:array>
		</beans:property>
	</beans:bean>

[posta07]

Hi Gary,

So here's what I have to report back

I tried the interceptor approach you recommended. I still wasn't getting the expected behavior (the FIN message) when sending the response back to the client. Instead I was getting the RST message upon closing the socket. Watching all of this over wireshark and jmeter, I could see that the packets were sent, but the appropriate TCP end-communication steps weren't taken, ie, the RST message was sent and jmeter would consider it a "connection reset" and drop all of the data (which is why it wasn't showing up in the response console).

I tried all kinds of things (editing some of the files in the Tcp module of SI and observing the behavior) to get the expected behavior, to no avail. Both trying things with the interceptor and without the interceptor. I always seemed to get the RST message and never a FIN message.

I found in this document (http://download.oracle.com/javase/1....n_release.html) that the 'abortive procedure' (as opposed to the orderly connection release) will always be used when Socket.setLinger(true,0) is called. In other words, a RST message will always be sent when a socket is closed.

So, how do Java applications perform orderly and abortive releases? Let's consider abortive releases first. A convention that has existed since the days of the original BSD sockets is that the "linger" socket option can be used to force an abortive connection release. Either application can call Socket.setLinger (true, 0) to tell the TCP stack that when this socket is closed, the abortive (RST) procedure is to be used

When I changed the so-linger option to a non-negative, non-zero number (5000 for my case) in the config files, in conjunction with the interceptor-close method you showed in the previous post, I noticed that the FIN message was indeed sent properly and everything worked as expected.

Further digging in the source, I see that if the so-linger option is omitted from the xml config, the Socket.setLinger(true,0) is still called:

from org.Springframework.integration.ip.tcp.connection. AbstractConnectionFactory.setSocketAttributes(Sock et socket)....

...
		if (this.soLinger >= 0) {
			socket.setSoLinger(true, this.soLinger);
		}
...

Is it a good idea to use to the abortive connection release (sending the RST message) by default? Or would it be better to only set the so-linger field on the socket object only when one is specified in the config files (ie, this.soLinger > 0 instead of greater than or equal)?

As you mentioned, the exceptions do show up in the log when using the interceptor to close the socket, but the behavior is as I expected, ie, the request/response streams are properly terminated with the FIN messages and the client app (jmeter in this case) receives the response.