RestTemplate I/O Errors and HttpClient

I working on a large middleware application that users RestTemplate to handle RESTful calls to about 40 different services.

We are using HttpClient 3.1 as the ClientHttpRequestFactory and MultiThreadedHttpConnectionManager as HTTPClient's connection manage.

We get intermittent I/O Errors as we scale to a multi-user / multi-thread environment that we do not see when running single-user tests in the development environment.

Has anyone else experienced a similar problem? Can anyone suggest a solution or approach we might try? Should we replace HttpClient 3.1?

The network people at the company where I'm working say that they see "out of state packets" causing their firewall to drop our connections.

Below is a typical stack trace from an I/O Exception.

Caused by: org.springframework.web.client.ResourceAccessExcep tion: I/O error: Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
at org.springframework.web.client.RestTemplate.doExec ute(RestTemplate.java:453)
at org.springframework.web.client.RestTemplate.execut e(RestTemplate.java:409)
at org.springframework.web.client.RestTemplate.getFor Object(RestTemplate.java:206)
at com.nytimes.cil.service.regi.RegiServiceImpl.searc hRegiAccount(RegiServiceImpl.java:115)
... 91 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully (Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unkn own Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRec ord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDat aRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(U nknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.commons.httpclient.HttpParser.readRawLi ne(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine( HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readL ine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpCon nectionManager$HttpConnectionAdapter.readLine(Mult iThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readS tatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readR esponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execu te(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:323)
at org.springframework.http.client.CommonsClientHttpR equest.executeInternal(CommonsClientHttpRequest.ja va:83)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:52)
at org.springframework.web.client.RestTemplate.doExec ute(RestTemplate.java:438)
... 94 more