Mongo application fails to stop thread MongoCleaner with redeploy or restart Tomcat

[Bert Haartjes]

Hi,

I have an issue with my application based on spring-data-mongodb (101 release) and I use the mongo-java-driver 2.8.0.

When restarting Tomcat I get this message:
SEVERE: The web application [/sky-rest-provider] appears to have started a thread named [MongoCleaner1055191651] but has failed to stop it. This is very likely to create a memory leak.

When redeploying i get this message:
SEVERE: The web application [/sky-rest-provider] created a ThreadLocal with key of type [com.mongodb.DBTCPConnector$1] (value [com.mongodb.DBTCPConnector$1@346f76f9]) and a value of type [com.mongodb.DBTCPConnector.MyPort] (value [com.mongodb.DBTCPConnector$MyPort@543efe7e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

and indeed after a few redeploys I run into OutOfMemory:Permgen space error. I thought this was fixed in version 2.6, see link: https://jira.mongodb.org/browse/JAVA-306.

Is there something else I have missed? Couldn't find anything usefully with an internetsearch... Time to ask the experts.

This is my spring config for mongo

<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
		<constructor-arg name="mongo" ref="mongo" />
		<constructor-arg name="databaseName" value="CLOUD" />
	</bean>
	<!-- Factory bean that creates the Mongo instance -->
	<bean id="mongo" class="org.springframework.data.mongodb.core.MongoFactoryBean">
		<property name="host" value="localhost" />
	</bean>

	<mongo:repositories base-package="com.highway.sky.repositories"
		mongo-template-ref="mongoTemplate" />

[harshal.veera]

Hi,

Even I am facing the same problem. Can SomeOne Help?

Thank you

[robert.wlaschin]

Harshal and Bert,

It looks like mongo has delayed processing for it's threaded connection manager which is causing your problem. I'm sorry that it is difficult to find this information by searching.

This is what you need to do.

Create a class that implements the ServletContextListener interface (eg. com.service.ServiceContextListener).

Add the following code to the contextDestroyed method

// public void contextDestroyed(ServletContextEvent context)
ServletContext sc = context.getServletContext();
WebApplicationContext springContext = WebApplicationContextUtils.getWebApplicationContex t(sc);
if( springContext.containsBean("mongoTemplate") ) {
MongoTemplate mt = (MongoTemplate) springContext.getBean("mongoTemplate");
if( mt != null ) {
DB db = mt.getDb();
if( db != null ) {
Mongo mongo = db.getMongo();
if( mongo != null ) {
mongo.close();
}
}
}
}

System.gc();

do {
try { // https://jira.mongodb.org/browse/JAVA-400
Thread.sleep(5000);
break;
} catch (InterruptedException e) {
// e.printStackTrace();
}
}while(true);

java.beans.Introspector.flushCaches();

/// end snippet

Get the mongoTemplate singleton from the springContext. With that you can get the DB, and intern get the Mongo instance. The mongo instance controls the cleanup which is triggered by calling mongo.close(). Unfortunately, you are not done yet Call the garbage collector to 'suggest' that mongo should clean up all of the open threads. Then you need to wait for ReplicaSetStatus.Update thread to wake up and clean it's self. This is on a 5 second timer, hence the 5 second wait. Then just for good measure flushCaches.

In web.xml you'll need to define a listener. Please note that you'll need to define your listener after the 'org.springframework.web.context.ContextLoaderList ener' otherwise your function parameters will be null.

e.g.

<listener>
<listener-class>com.service.ServiceContextListener</listener-class>
</listener>

This should allow you to close tomcat with the shutdown command. There were a lot of bugs that are still being resolved and perhaps some day this will get fixed but until then I hope this helps you out.

Enjoy!

[dacedos]

Hello robert.wlaschin,

I followed what you point but now my app do not start.

This is my java class:

package com.test.services;

import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.web.context.WebApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import com.mongodb.DB;
import com.mongodb.Mongo;

public class ServiceContextListener implements ServletContextListener {

	@Override
	public void contextDestroyed(ServletContextEvent context) {
		ServletContext sc = context.getServletContext();
		WebApplicationContext springContext = WebApplicationContextUtils.getWebApplicationContext(sc);
		if (springContext.containsBean("mongoTemplate")) {
			MongoTemplate mt = (MongoTemplate) springContext.getBean("mongoTemplate");
			if (mt != null) {
				DB db = mt.getDb();
				if (db != null) {
					Mongo mongo = db.getMongo();
					if (mongo != null) {
						mongo.close();
					}
				}
			}
		}

		System.gc();

		do {
			try { // https://jira.mongodb.org/browse/JAVA-400
				Thread.sleep(5000);
				break;
			} catch (InterruptedException e) {
				// e.printStackTrace();
			}
		} while (true);

		java.beans.Introspector.flushCaches();
	}

	@Override
	public void contextInitialized(ServletContextEvent arg0) {

	}

}

and the lister just after loading the ContextLoaderListener in web.xml

	<listener>
		<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
	</listener>
	<listener>
		<listener-class>com.test.ServiceContextListener</listener-class>
	</listener>

I do not get any error in console but I cannot see the welcome page!

I only see 404 not found error

I use Apache Tomcat/7.0.26 with Spring and Spring-data-mongo 3.1.0.RELEASE.

Is there something wrong??

It is critic because my tomcat dies in pre-production environment. How could I tune my server to avoid it?


Thanks at all!