'Impossible' ClassCastException in ProtectedResourceProcessingFilter?

[sethcall]

This is a real head-scratcher. At least, for me.

I'm using OAuth 1.0. If I hit a URL that I have protected one-at-a-time, there is no issue. But if make a bunch of concurrent HTTP requests to that URL (~50), then I intermittently see this error:

java.lang.ClassCastException: com.me.CustomAuthentication cannot be cast to org.springframework.security.oauth.provider.ConsumerAuthentication 
at org.springframework.security.oauth.provider.ProtectedResourceProcessingFilter.onValidSignature(ProtectedResourceProcessingFilter.java:58) [spring-security-oauth-3.17.SS3.jar:na]
at org.springframework.security.oauth.provider.OAuthProviderProcessingFilter.doFilter(OAuthProviderProcessingFilter.java:158) [spring-security-oauth-3.17.SS3.jar:na]

So the only background I think I need to explain is the com.me.CustomAuthentication -- it's a custom Authentication implementation I use in different parts of the code, but as I think you'll see, it's irrelevant; here's why.

If you look at the stack trace, we start in OAuthProviderProcessFilter, a base class. Here's what leads up to the exception:

//create an authentication request.
ConsumerAuthentication authentication = 
    new ConsumerAuthentication(consumerDetails, credentials, oauthParams);
authentication.setDetails(createDetails(request, consumerDetails));

Authentication previousAuthentication = SecurityContextHolder.getContext().getAuthentication();
try {
              ### ADDED BY sethcall:   TAKE NOTE--HERE AUTH IS BEING SET INTO SECURITY CONTEXT HOLDER
              //set the authentication request (unauthenticated) into the context.
              SecurityContextHolder.getContext().setAuthentication(authentication);

              //validate the signature.
              validateSignature(authentication);

              //mark the authentication request as validated.
              authentication.setSignatureValidated(true);

              //mark that processing has been handled.
              request.setAttribute(OAUTH_PROCESSING_HANDLED, Boolean.TRUE);

              if (log.isDebugEnabled()) {
                log.debug("Signature validated.");
              }

              //go.
              onValidSignature(request, response, chain);
 }
finally {
              //clear out the consumer authentication to make sure it doesn't get cached.
              resetPreviousAuthentication(previousAuthentication);
}

From the stack trace, you can see that onValidSignature() is executed. Here's the code for that (only the 1st line because that's where the ClassCastException occurs)

ConsumerAuthentication authentication = 
    (ConsumerAuthentication) SecurityContextHolder.getContext().getAuthentication();

So let's recap:

In the first block of code, SecurityContextHolder.getContext().setAuthenticati on(<consumer auth>) is called, and then in onValidSignature, SecurityContextHolder.getContext().getAuthenticati on() is called -- but now it's not of type ConsumerAuthentication ??

This seems outright impossible. If you look at SecurityContextHolder, you'll notice that it has various strategies; by default, it uses a ThreadLocal strategy (and I've verified multiple times--I'm using the default). So it's as if the ThreadLocal data is getting mixed up for various threads coming into the system. It's extremely odd.

Does anyone have any ideas? I'm just now beginning debugging, but I thought perhaps someone has seen something similiar. I haven't opened up a bug because I feel it's somehow something I've done or otherwise not really the fault of the OAuth library...

Anyway, thanks for your help...

Regards,
Seth

[stoicflame]

Wow. That's weird.

So somehow your CustomAuthentication is getting into the SecurityContextHolder. Do you ever do SecurityContextHolder.getContext().setAuthenticati on(customAuth) where customAuth is an instance of com.me.CustomAuthentication? It might be a case where it's not getting cleaned up properly--you've got to call SecurityContextHolder.getContext().setAuthenticati on(previousAuth) on your way out of the logic where you set your customAuth.

[sethcall]

Hey stoicflame,

I do call it elsewhere in the code, but the issue is, in this flow, there should be no opportunity for any other code to do 'get in' The path from the OAuth library calling setAuthentication, to when it then calls getAuthentication, is a 'straight shot' (i.e., the path of execution is straight through, with no other 'external code' having a chance to mess with the context).

I've been debugging this nonstop. At this point, I am just simply confused. It looks like a traditional race condition... here is an example of what I've seen (I changed the oauth code a little locally so that I can figure out what's going on, by pulling out the Authentication object before the cast is tried).

protected void onValidSignature(HttpServletRequest request, HttpServletResponse response, FilterChain chain) throws IOException, ServletException {
        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        // DO A CHECK TO SEE IF THE PROBLEM WILL OCCUR
        if (!(auth instanceof ConsumerAuthentication)) {
            LOG.error("here's the problem path")
        }
        // THIS NEXT LINE DOESNT FAIL SOMETIMES EVEN THOUGH THE ABOVE ERROR CHECK IS TRUE!
        ConsumerAuthentication authentication = (ConsumerAuthentication) SecurityContextHolder.getContext().getAuthentication();

This should not be possible...

[stoicflame]

I admit, I'm having a hard time coming up with an example of how this is occurring. Of course (just a sanity check) I assume you've verified that the strategy for the SecurityContextHolder is the ThreadLocal one?

All I can share is my own experience with problems with thread local variables, which problems have usually been caused by failing to "clean up" the thread local variable on my way "out" of the logic (like in a finally block). The problem occurs when two different requests are being serviced by the same thread. Tomcat (or whatever app server you're using) has a pool of threads that it draws from in order to service it's requests, and if one request sets the variable without cleaning it up, then the next request (serviced by the same thread) looks at that varable, it finds the one that was set by the previous request.

[sethcall]

Definitely using ThreadLocal.

I'm logging Thread.currentThread().getId() in all logging, to make sure I know what's going on--definitely the same thread is being re-used. So that's a good point. I've been considering how that might be part of the issue.

While you're pointing out a valid potential problem, in this flow, the Oauth library writes over whatever is in the security context (with ConsumerAuthentication), and then a few lines of code down from that, tries to read back out from the same security context (where I get the class cast exception), on the same thread--and in the same try block (which has a finally that reverts the authentication). So this library is doing the right thing...

Still, maybe this is somehow the issue... I don't see how though.

[sethcall]

I am logging every occurrence of SecurityContext.setAuthentication() with the thread ID.

From observing it, it's really as if there is only one SecurityContext, instead of one-per-thread. For instance, when I get the cast exception, I can see that a *different thread than the one with the exception* calls setAuthentication just before my thread calls getAuthentication() -- but what I get out of the security context is the one from the *other thread*.

At the moment, it really does seem that there is only one SecurityContext, no matter how many threads are going (again, you think--it must be the strategy is wrong, but I assure you all it's using ThreadLocal).

Meh.

[sethcall]

I am close now. SecurityContextPersistenceFilter is calling:

SecurityContextHolder.setContext(contextBeforeChai nExecution);

This is, over time and in with exceptions occuring in request threads, causing every SecurityContext in every thread to be the same instance, causing this issue.

I'll have to investigate tomorrow. I'm thinking I must be doing something wrong or incorrect to upset this 'standard' spring filter.

[sethcall]

I think this bug is my root issue.

https://jira.springframework.org/browse/SEC-356

Wouldn't any OAuth-based server have this concern? Not sure yet.