Weird error 500 messages on ALTERNATE calls to a remote service

[Paul Newport]

We are using Spring remoting to call service methods on one machine from another. An example of a service might be, check that a person with user id = "fred" is on system (e.g. a simple search that takes a string and returns a boolean).

Lately we have been getting odd error 500 messages coming back from these services. I have written a test case to do remote calls to the server, and these are failing on every other call i.e. if I call it 6 times it works on calls 1,3 and 5.

In each case I get an error 500 back but there are no messages on the server log at all about errors. The client stack traces look like that pasted below.

Any ideas ?

java.io.IOException: Did not receive successful HTTP response: status code = 500, status message = [Internal Server Error]
Caused by: java.io.IOException: Did not receive successful HTTP response: status code = 500, status message = [Internal Server Error]
	at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:139)
	at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.doExecuteRequest(SimpleHttpInvokerRequestExecutor.java:62)
	at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:134)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:162)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:139)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)

[Paul Newport]

This was caused by having a cluster with two nodes in, and one node was not working properly (incorrectly configured Websphere cluster in other words)