How do I configure the socket read timeout in spring social

[Rob Blair]

I'm using Spring social 1.0.1.RELEASE and framework 3.1.0.RELEASE to make various Facebook calls, and a 1.1.0 facebook variant.

I'm having a very hard time figuring out how to set the read timeout on the low level calls. The level of indirection is maddening, and ends deep in private static classes I have little hope of sub-classing.

The system ends up delegating to org.springframework.social.support.HttpComponentsC lientHttpRequestFactory which sets a read timeout of 60,000 ms. I wish to set the timeout lower, since some fraction of my Facebook calls timeout, and I wish to have them error out faster!

Oddly, these errors happen more when several threads are making simultaneous requests.

n.b. I have made modifications to my spring social libraries for other requested features, so am not running pure. If this has been fixed in more up to date versions, that would make me happy, and have to merge.

Thanks,
rbb
--
exception info:
org.springframework.web.client.ResourceAccessExcep tion: I/O error: Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
at org.springframework.web.client.RestTemplate.doExec ute(RestTemplate.java:453)
at org.springframework.web.client.RestTemplate.execut e(RestTemplate.java:415)
at org.springframework.web.client.RestTemplate.getFor Object(RestTemplate.java:213)
at org.springframework.social.facebook.api.impl.Faceb ookTemplate.fetchObject(FacebookTemplate.java:205)
at com.inqmobile.inqcloud.service.FacebookQueryServic e.multiQuery(FacebookQueryService.java:75)
at com.inqmobile.inqcloud.service.FacebookNewsHarvest Service.doHarvestDetails(FacebookNewsHarvestServic e.java:187)
at com.inqmobile.inqcloud.service.FacebookNewsHarvest Service.doHarvestByKeys(FacebookNewsHarvestService .java:133)
at com.inqmobile.inqcloud.service.FacebookNewsHarvest Service.getHarvestItemsBySocialKeys(FacebookNewsHa rvestService.java:108)
at com.inqmobile.inqcloud.service.HarvestFeedService. harvestSocialKey(HarvestFeedService.java:73)
at com.inqmobile.inqcloud.service.HarvestFeedService. harvestFeed(HarvestFeedService.java:64)
at com.inqmobile.inqcloud.task.HarvestFeedAsyncTask.r un(HarvestFeedAsyncTask.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream. java:146)
at sun.security.ssl.InputRecord.readFully(InputRecord .java:312)
at sun.security.ssl.InputRecord.read(InputRecord.java :350)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocke tImpl.java:850)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLS ocketImpl.java:807)
at sun.security.ssl.AppInputStream.read(AppInputStrea m.java:94)
at org.apache.http.impl.io.AbstractSessionInputBuffer .fillBuffer(AbstractSessionInputBuffer.java:149)
at org.apache.http.impl.io.SocketInputBuffer.fillBuff er(SocketInputBuffer.java:110)
at org.apache.http.impl.io.AbstractSessionInputBuffer .readLine(AbstractSessionInputBuffer.java:264)
at org.apache.http.impl.conn.DefaultResponseParser.pa rseHead(DefaultResponseParser.java:98)
at org.apache.http.impl.io.AbstractMessageParser.pars e(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection. receiveResponseHeader(AbstractHttpClientConnection .java:281)
at org.apache.http.impl.conn.DefaultClientConnection. receiveResponseHeader(DefaultClientConnection.java :247)
at org.apache.http.impl.conn.AbstractClientConnAdapte r.receiveResponseHeader(AbstractClientConnAdapter. java:216)
at org.apache.http.protocol.HttpRequestExecutor.doRec eiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execu te(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector .tryExecute(DefaultRequestDirector.java:647)
at org.apache.http.impl.client.DefaultRequestDirector .execute(DefaultRequestDirector.java:464)
at org.apache.http.impl.client.AbstractHttpClient.exe cute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.exe cute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.exe cute(AbstractHttpClient.java:732)
at org.springframework.social.support.HttpComponentsC lientHttpRequest.executeInternal(HttpComponentsCli entHttpRequest.java:81)
at org.springframework.social.support.AbstractBufferi ngClientHttpRequest.executeInternal(AbstractBuffer ingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:91)
at org.springframework.social.oauth2.OAuth2RequestInt erceptor.intercept(OAuth2RequestInterceptor.java:4 5)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:81)
at org.springframework.http.client.InterceptingClient HttpRequest.executeInternal(InterceptingClientHttp Request.java:67)
at org.springframework.http.client.AbstractBufferingC lientHttpRequest.executeInternal(AbstractBuffering ClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.social.support.BufferingClient HttpRequest.executeInternal(BufferingClientHttpReq uest.java:57)
at org.springframework.social.support.AbstractBufferi ngClientHttpRequest.executeInternal(AbstractBuffer ingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:91)
at org.springframework.social.oauth2.OAuth2RequestInt erceptor.intercept(OAuth2RequestInterceptor.java:4 5)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:81)
at org.springframework.http.client.InterceptingClient HttpRequest.executeInternal(InterceptingClientHttp Request.java:67)
at org.springframework.http.client.AbstractBufferingC lientHttpRequest.executeInternal(AbstractBuffering ClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.social.support.BufferingClient HttpRequest.executeInternal(BufferingClientHttpReq uest.java:57)
at org.springframework.social.support.AbstractBufferi ngClientHttpRequest.executeInternal(AbstractBuffer ingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:91)
at org.springframework.social.oauth2.OAuth2RequestInt erceptor.intercept(OAuth2RequestInterceptor.java:4 5)
at org.springframework.http.client.InterceptingClient HttpRequest$RequestExecution.execute(InterceptingC lientHttpRequest.java:81)
at org.springframework.http.client.InterceptingClient HttpRequest.executeInternal(InterceptingClientHttp Request.java:67)
at org.springframework.http.client.AbstractBufferingC lientHttpRequest.executeInternal(AbstractBuffering ClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttp Request.execute(AbstractClientHttpRequest.java:49)
at org.springframework.web.client.RestTemplate.doExec ute(RestTemplate.java:438)
... 13 more

[habuma]

Ultimately, configuring things like this are a matter of configuring the underlying request factory used by the RestTemplate inside of Spring Social. Since each request factory is different (some offer configuration features that others don't), it doesn't make much sense to expose such configuration directly on Spring Social classes just so that they can pass that info down when creating the request factory. (Never mind that it's just poor design to expose request factory-level configuration at the Spring Social level.)

That said, you're right that it's inconvenient to do such configuration on the request factory. OAuth1Template and OAuth2Template both have a setRequestFactory() method, but unless you're working at Spring Social's low-level OAuth API or you've written your own service provider implementation, there's no way to use these methods with the connection framework. Likewise, AbstractOAuth1ApiBinding and AbstractOAuth2ApiBinding also have setRequestFactory() methods (and therefore, FacebookTemplate, TwitterTemplate, etc) have those methods; but when using the connection framework, you're given the API bindings by their interface which doesn't expose that method.

I've created https://jira.springsource.org/browse/SOCIAL-321 to track this and have scheduled it for 1.1.0.M1. I'm thinking it might be good to introduce a request factory provider strategy. You'd write an implementation of this strategy that produces a ClientHttpRequestFactory that is configured however you want it to be and declare it as a bean in Spring. If the bean exists, then it could be used instead of the current strategy.

[Rob Blair]

The hard part is that because so many fields are final, or don't have accessors, I can't even kludge it. I just want the hooks to do it in code, worst case scenario.

I'm not even sure if it's a spring social bug, or a threading issue in HttpClient / ConnectionManager:
http://hc.apache.org/httpcomponents-...nnManager.html
in HttpClient 4.2, ThreadSafeClientConnManager is deprecated. Spring social is set to use 4.1 or 4.0 currently.

[habuma]

Well, in the case of Facebook, you *could*...

1. Create a custom implementation of FacebookServiceProvider to create a FacebookTemplate and then set a request factory of your choosing on it via setRequestFactory()...and set timeouts, proxy settings, etc, etc on that request factory.

public class MyFacebookServiceProvider extends AbstractOAuth2ServiceProvider<Facebook> {
	public MyFacebookServiceProvider(String clientId, String clientSecret) {
		super(new FacebookOAuth2Template(clientId, clientSecret));
	}

	public Facebook getApi(String accessToken) {
		FacebookTemplate facebookTemplate = new FacebookTemplate(accessToken);
		facebookTemplate.setRequestFactory(getRequestFactory());
		return facebookTemplate;
	}
	
	private ClientHttpRequestFactory getRequestFactory() {
		// create a request factory and set connection details on it (such as read timeout)
		// for example, using SimpleClientHttpRequestFactory...
		SimpleClientHttpRequestFactory rf = new SimpleClientHttpRequestFactory();
		rf.setReadTimeout(20000);
		return rf;
	}
}

2. Create a custom implementation of FacebookConnectionFactory that uses the custom FacebookServiceProvider

public class MyFacebookConnectionFactory extends OAuth2ConnectionFactory<Facebook> {

	public MyFacebookConnectionFactory(String clientId, String clientSecret) {
		super("facebook", new MyFacebookServiceProvider(clientId, clientSecret), new FacebookAdapter());
	}

}

3. When registering the connection factory locator in Spring, use your custom FacebookConnectionFactory instead of the one that comes with Spring Social Facebook.

@Bean
@Scope(value="singleton", proxyMode=ScopedProxyMode.INTERFACES)	
public ConnectionFactoryLocator connectionFactoryLocator() {
	ConnectionFactoryRegistry registry = new ConnectionFactoryRegistry();
	registry.addConnectionFactory(new MyFacebookConnectionFactory(environment.getProperty("facebook.appId"), environment.getProperty("facebook.appSecret")));
	return registry;
}

No final methods, classes, or properties are in play in that arrangement and you can certainly customize the request factory to your heart's content. It's a bit more involved than I care for it to be, but it is do-able.

I'm hesitant to directly expose *too* much via setters, constructor args, or hooks given the place where the request factory fits relative to Spring Social's components (it's really a RestTemplate detail, not a Spring Social detail) and the fact that every request factory is different (some connection factories support settings that others do not).

[Rob Blair]

Yes, I could call setRequstFactory(). I figured that there was some reason the guts picked HttpComponentsClientHttpRequestFactory over SimpleClientHttpRequestFactory if HttpClient was present. Also the default behavior deals with proxy settings (not important to me), and the template applies some interceptors, which I thought were important.
It is honestly confusing to examine.


I'll try the setRequestFactory approach with SimpleClientHttpRequestFactory, on the theory that my 60 second timeout may be due to some threading errors in the HttpClient version.

The calling code is creating and using several FacebookTemplate's in threads that are kicked off simultaneously.
When I run one thread, I never see problems.
When I have two threads (different accessTokens) I get 3/480 error occurrences. Four threads, 12 / 960.

Thanks.
I'll let you know if it works, or not.

[habuma]

Note that I only used SimpleClientHttpRequestFactory in the example, because it's really simple to use. But I favor (and Spring Social favors) HttpComponentsClientHttpRequestFactory because it's far more powerful. But with that power comes some complexity and configuring HttpComponentsClientHttpRequestFactory involves configuring its HttpClient...and, in my opinion, the HttpClient API is not intuitive. (And thus the reason I didn't bother with it for the sample above...I left the configuration of HttpClient as an "exercise for the reader.)

I don't suspect that your problems are in Spring Social itself, as Spring Social's API bindings do very little other than call RestTemplate with an appropriate set of parameters. The problem *might* be within RestTemplate, but even then I have my doubts given that RestTemplate doesn't do much more than delegate to the underlying request factory. Given the nature of the error ("Read timed out"), this sounds like either a network issue or an issue on the provider's server. I'd assume that Facebook's servers are beefy enough to handle whatever you're doing, but you never know when they might be having trouble.

But my guess is that you're taxing the network between your app and Facebook. Again, it's just a guess, but a guess weighed by the "Read timed out" error (which according to the stack trace happens much lower in the stack than any of Spring's code). That said, you may be right and there may be some tweaks to be made to the underlying connection factory (or to the HttpClient) that will resolve the issue. And your guess of setting the timeout lower is a reasonable one...you'll still get timeouts, but possibly fewer of them.

[Rob Blair]

I've extensively profiled the call times, and can vouch that it's not taxing the network. The data access rate is far below Facebook limits. When performing normally, calls never are more than 5 seconds. When the timeout occurs, it's waiting 60 seconds+.

I'll work on tweaking the connections, trying it on different server setups.
p.s. do you ever sleep?

...
But my guess is that you're taxing the network between your app and Facebook. Again, it's just a guess, but a guess weighed by the "Read timed out" error (which according to the stack trace happens much lower in the stack than any of Spring's code). That said, you may be right and there may be some tweaks to be made to the underlying connection factory (or to the HttpClient) that will resolve the issue. And your guess of setting the timeout lower is a reasonable one...you'll still get timeouts, but possibly fewer of them.

[habuma]

I've extensively profiled the call times, and can vouch that it's not taxing the network. The data access rate is far below Facebook limits. When performing normally, calls never are more than 5 seconds. When the timeout occurs, it's waiting 60 seconds+.

Well, I'm not saying that it *is* the network...just saying that seems like the most likely culprit given the nature of the error you receive. I'd be eager to hear what comes out of your tweaking.

p.s. do you ever sleep?

Nasty habit. Trying to break it. :-)

Actually, I'm unsure why I was up so late...just felt compelled to do a bit of Spring Social stuff (I've been otherwise engaged on another project and haven't had much time on Spring Social lately...hoping to get back to it in the very very very near future.)

[Rob Blair]

So I've not been able to reproduce my problems locally, but did get it to repro on the remote server, even after substituting the SimpleClientHttpRequestFactory in to the FacebookTemplate.

The hosting servers are my next spot to investigate. Grr.
Thanks for the assistance, and remember sleep is not your enemy.

[habuma]

So I've not been able to reproduce my problems locally, but did get it to repro on the remote server, even after substituting the SimpleClientHttpRequestFactory in to the FacebookTemplate.

The hosting servers are my next spot to investigate. Grr.

Interesting. Let me know how that turns out. I'll stay up late waiting for your results. :-)

and remember sleep is not your enemy.

Or maybe I should just wait to hear from you tomorrow.