Spring Data Redis-Closed Connection after first access and any subsequent operations

[bdangubic]

All,

I am investigating usage of Redis for some simple caching of data. The problem I am running into is that after the first time the application attempts to access Redis any subsequent operation results in an exception:

org.jredis.connector.NotConnectedException: Not connected!
	at org.jredis.ri.alphazero.connection.SynchConnection.serviceRequest(SynchConnection.java:124)
	at org.jredis.ri.alphazero.JRedisService.serviceRequest(JRedisService.java:180)
	at org.jredis.ri.alphazero.JRedisSupport.quit(JRedisSupport.java:1465)
	at org.springframework.data.keyvalue.redis.connection.jredis.JredisConnection.close(JredisConnection.java:63)
	at org.springframework.data.keyvalue.redis.core.RedisConnectionUtils.releaseConnection(RedisConnectionUtils.java:96)
	at org.springframework.data.keyvalue.redis.core.RedisTemplate.execute(RedisTemplate.java:141)
	at org.springframework.data.keyvalue.redis.core.RedisTemplate.execute(RedisTemplate.java:114)
	at org.springframework.data.keyvalue.redis.core.RedisTemplate.execute(RedisTemplate.java:102)

Here's the configuration:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   	   xmlns:aop="http://www.springframework.org/schema/aop"
   	   xmlns:p="http://www.springframework.org/schema/p"
	   xsi:schemaLocation=
	    "http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
	     http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd">

<bean id="jredisConnectionFactory" class="org.springframework.data.keyvalue.redis.connection.jredis.JredisConnectionFactory" p:hostName="localhost" p:port="6379" />
	<bean id="serializer" class="org.springframework.data.keyvalue.redis.serializer.JdkSerializationRedisSerializer" />
	<bean id="optionsDataCache" class="com.xxx.OptionsDataCache" p:cacheProvider-ref="redisDataCacheProvider" />
	<bean id="optionsFacadeAdvice" class="com.xxx.OptionsFacadeDataCacheAdvice" p:optionsDataCache-ref="optionsDataCache" />
	<bean id="redisDataCacheProvider" class="com.xxx.RedisDataCacheProvider" p:serializer-ref="serializer" p:connectionFactory-ref="jredisConnectionFactory"/>

<aop:config>
		<aop:pointcut id="optionsFacadePointcut" expression="execution(* com.xxx.OptionsFacade.getOptions(..))"/>
		
		<aop:aspect id="optionsFacadeAspect" ref="optionsFacadeAdvice" order="0">
		    <aop:around	pointcut-ref="optionsFacadePointcut" method="doAroundCache"/>
		</aop:aspect>
	</aop:config>
</beans>

Code in the around advice:

public Object doAroundCache(ProceedingJoinPoint call) throws Throwable {
        Object[] args = call.getArgs();
        String cacheKey = generateCacheKeyFromArguments(args);
        Object dataFromCache = optionsDataCache.getFromCache((String) args[1], cacheKey); // this call will succeed the first time it is accessed, fails on any subsequent invocation
        if (null == dataFromCache) {
            log.debug("data not in the cache, invoking options facade to get the data with key > " + cacheKey);
            dataFromCache = call.proceed();
            optionsDataCache.storeDataIntoCache((String) args[1], cacheKey, (IDTO) dataFromCache); // this call never executes, not even on the first invocation, getting Not Connected exception
        }
        return dataFromCache;
    }

in the optionsDataCache I have the following:

...
@Override
    public Object getFromCache(final String region, final String cacheKey) {
        final byte[] cacheKeyHash = serializer.serialize(cacheKey);
        return getTemplate().execute(new RedisCallback<Object>() {
            @Override
            public Object doInRedis(RedisConnection paramRedisConnection) throws DataAccessException {
                Object cachedObject = null;
                byte[] data = paramRedisConnection.get(cacheKeyHash);
                if (null != data) {
                    cachedObject = serializer.deserialize(data);
                    log.debug("found data in the cache of class " + cachedObject.getClass().getSimpleName());
                    return cachedObject;
                }
                return cachedObject;
            }

        });
    }

...

 private RedisTemplate getTemplate() {
        RedisTemplate redisTemplate = new RedisTemplate(connectionFactory);
        redisTemplate.setExposeConnection(true);
        return redisTemplate;
    }

I also tried to define redisTemplate in the configuration and inject into optionsDataCache and I also tried with and without expose-connection set to true and as well as pooled and non-pooled connection factory.

Any help appreciated!

[bdangubic]

All,

I created an even simpler test, it appears as if once the first connection retrieved from the connection factory is closed, any subsequent connection that is retrieved from the factory is unusable. Any input would be greatly appreciated. Here's simple configuration:

<bean id="jredisConnectionFactory" class="org.springframework.data.keyvalue.redis.connection.jredis.JredisConnectionFactory" p:hostName="localhost" p:port="6379" />

Test code:

JredisConnectionFactory connectionFactory = (JredisConnectionFactory) appCtx
                    .getBean("jredisConnectionFactory");
RedisConnection paramRedisConnection = connectionFactory.getConnection();
System.out.println(paramRedisConnection);
paramRedisConnection.close();

paramRedisConnection = connectionFactory.getConnection();
System.out.println(paramRedisConnection);
byte[] data = paramRedisConnection.get(key);

Produces output:

org.springframework.data.keyvalue.redis.connection.jredis.JredisConnection@12922f6
org.springframework.data.keyvalue.redis.connection.jredis.JredisConnection@1c486f2
2011-06-14 15:58:23,912 ERROR [main] (redis.TestRedisConnectionFactoryAfterClosedConnection:63) - Error in TestRedisConnectionFactoryAfterClosedConnection: 
org.jredis.connector.NotConnectedException: Not connected!
	at org.jredis.ri.alphazero.connection.SynchConnection.serviceRequest(SynchConnection.java:124)
	at org.jredis.ri.alphazero.JRedisService.serviceRequest(JRedisService.java:180)
	at org.jredis.ri.alphazero.JRedisSupport.get(JRedisSupport.java:950)
	at org.springframework.data.keyvalue.redis.connection.jredis.JredisConnection.get(JredisConnection.java:258)
...
Exception in thread "main" org.jredis.connector.NotConnectedException: Not connected!
	at org.jredis.ri.alphazero.connection.SynchConnection.serviceRequest(SynchConnection.java:124)
	at org.jredis.ri.alphazero.JRedisService.serviceRequest(JRedisService.java:180)
	at org.jredis.ri.alphazero.JRedisSupport.get(JRedisSupport.java:950)
	at org.springframework.data.keyvalue.redis.connection.jredis.JredisConnection.get(JredisConnection.java:258)
...

[Costin Leau]

Try switching the "driver" since it manages its connections natively and there's not much you can do from outside. The latest jedis I believe has some support for doing connection reconnecting and validation (onBorrow).

[bdangubic]

Thanks Costin! I'll give it a try. I got things "working" by creating a single connection in init-method method of the cache provider and closing it out on destroy-method (using JRedis). The connection re-connecting seems to be working ok, running some load tests now to see if there are any problems with this approach under high load.