Error on undeploy/deploy

[amsmota]

I'm having this strange error on try to undeploy/deploy a jar in JBoss, with this strange message:


Illegal access: this web application instance has been stopped already. Could not load org.springframework.integration.message.ErrorMessa ge. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.

Note that JBoss enter a infinite loop, outputting in sucession the following trace:


11:04:40,318 INFO [WebappClassLoader] Illegal access: this web application instance has been stopped already. Could not load org.springframework.integration.message.ErrorMessa ge. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1244)
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1204)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)
at org.springframework.integration.scheduling.Message PublishingErrorHandler.handle(MessagePublishingErr orHandler.java:56)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
11:04:40,318 WARN [MessagePublishingErrorHandler] failure occurred in messaging task
java.lang.NullPointerException
at org.apache.commons.logging.LogFactory.getCachedFac tory(LogFactory.java:908)
at org.apache.commons.logging.LogFactory.getFactory(L ogFactory.java:395)
at org.apache.commons.logging.LogFactory.getLog(LogFa ctory.java:645)
at org.springframework.integration.dispatcher.Default MessageDistributor.<init>(DefaultMessageDistributor.java:43)
at org.springframework.integration.dispatcher.Default MessageDispatcher.getDistributor(DefaultMessageDis patcher.java:186)
at org.springframework.integration.dispatcher.Default MessageDispatcher.access$1(DefaultMessageDispatche r.java:182)
at org.springframework.integration.dispatcher.Default MessageDispatcher$DispatcherTask.run(DefaultMessag eDispatcher.java:208)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:136)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

I'll try to investigate a little more.

Regards.

[amsmota]

I get rid of the infinite loop (that forced me to stop/restart JBoss that is annoying) but not of the first error. And I don't know if my change will break something else...

In SimpleMessagingTaskScheduler I added executor.shutdown();


public void run() {
try {
this.task.run();
}
catch (Throwable t) {
if (errorHandler != null) {
errorHandler.handle(t);
}
executor.shutdown();
}
if (this.shouldRepeat) {
MessagingTaskRunner runner = new MessagingTaskRunner(this.task);
runner.setShouldRepeat(true);
executor.execute(runner);
}
}

[amsmota]

This problem still happens with M2.

[amsmota]

No one else has this problem? I'm mavenizing my project and now I'm using the m3 version of the repository that doesn't have the "hack" I made, so now I have to shutdown/restart JBoss every time I want to redeploy my war.

[Mark Fisher]

Can you post some information about how you're deploying the components. For example, are you using ContextLoaderListener and pointing to the configuration that includes all of the Spring Integration components?

Thanks,
Mark

[amsmota]

Hmmm, I think I don't understand your question, sorry. What I'm doing is just drop a war file in the server deploy dir, rewriting the old one.

What I think it's happening is this:

- The webapp is running a SimpleMessagingTaskScheduler::MessagingTaskRunner: :run() continuosly, from what I understand to see if there are some futures to run.

- I drop the war on the deploy dir

- JBoss undeploys the war

- The webapp tries to run the MessagingTaskRunner again, it can't do it, so goes to the MessagePublishingErrorHandler::handle()

- In here, it tries to send a ErrorMessage to the error queue

- The error occurs when JBoss tries to WebappClassLoader::loadClass the ErrorMessage class, that is not available because the webapp was undeployed

- it enter a loop because this error isn't catched


public void run() {
try {
this.task.run();
}
catch (Throwable t) {
if (errorHandler != null) {
ERROR HERE, NO CATCH --> errorHandler.handle(t);
}
}
if (this.shouldRepeat) {
MessagingTaskRunner runner = new MessagingTaskRunner(this.task);
runner.setShouldRepeat(true);
executor.execute(runner);
}
}


I hope I made myself clear...

Just a correction, I'm now using m2 from the repository, not m3.

[amsmota]

Like Caroline said, that first error is a NPE, not the classLoader I was telling. With my "hack" (executor.shutdown() in catch) the stack trace is this:


java.lang.NullPointerException
at org.apache.commons.logging.LogFactory.getCachedFac tory(LogFactory.java:979)
at org.apache.commons.logging.LogFactory.getFactory(L ogFactory.java:435)
at org.apache.commons.logging.LogFactory.getLog(LogFa ctory.java:685)
at org.springframework.integration.dispatcher.Default MessageDistributor.<init>(DefaultMessageDistributor.java:43)
at org.springframework.integration.dispatcher.Default MessageDispatcher.getDistributor(DefaultMessageDis patcher.java:186)
at org.springframework.integration.dispatcher.Default MessageDispatcher.access$1(DefaultMessageDispatche r.java:182)
at org.springframework.integration.dispatcher.Default MessageDispatcher$DispatcherTask.run(DefaultMessag eDispatcher.java:208)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:136)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
14:25:43,744 INFO [WebappClassLoader] Illegal access: this web application instance has been stopped already. Could not load org.springframework.integration.message.ErrorMessa ge. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1244)
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1204)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)
at org.springframework.integration.scheduling.Message PublishingErrorHandler.handle(MessagePublishingErr orHandler.java:59)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:141)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

[Mark Fisher]

Are you using log4j? If not, can you try with log4j?

Thanks
-Mark

[amsmota]

I suppose so, since I'm using JBoss and it seems it iuses it by default. But I'm not a expert on this, specially not on a Friday afternoon.

What do you want to know?

[Mark Fisher]

I have committed a change to avoid the loop between a MessagePublishingErrorHandler and a listener on the "errorChannel" that throws an Exception itself. It is available on the HEAD version from SVN and will be in tonight's snapshot. If you get a chance to test it please let me know.

Also, there is a short-term workaround that you could try as described here: http://forum.springframework.org/showthread.php?p=174364

Thanks,
Mark

[amsmota]

I just tried with 1.0.0.M3 and it gets rid of the loop and the NPE that was giving, it still gives the WebappClassLoader exception:


09:13:12,474 INFO [WebappClassLoader] Illegal access: this web application instance has been stopped already. Could not load org.apache.log4j.spi.VectorWriter. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1244)
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1204)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)
at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
at org.apache.log4j.Category.forcedLog(Category.java: 388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.warn(L og4JLogger.java:234)
at org.springframework.integration.scheduling.Message PublishingErrorHandler.handle(MessagePublishingErr orHandler.java:55)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:140)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

[amsmota]

And now I'm getting *another* endless loop...


15:37:22,224 INFO [STDOUT] - DefaultErrorChannel has reached capacity. Are any handlers subscribed?
15:37:22,224 INFO [STDOUT] - DefaultErrorChannel has reached capacity. Are any handlers subscribed?
15:37:22,224 INFO [STDOUT] - failure occurred in messaging task
java.lang.NullPointerException
at org.apache.commons.logging.LogFactory.getCachedFac tory(LogFactory.java:979)
at org.apache.commons.logging.LogFactory.getFactory(L ogFactory.java:435)
at org.apache.commons.logging.LogFactory.getLog(LogFa ctory.java:685)
at org.springframework.integration.dispatcher.Default MessageDistributor.<init>(DefaultMessageDistributor.java:43)
at org.springframework.integration.dispatcher.Default MessageDispatcher.getDistributor(DefaultMessageDis patcher.java:195)
at org.springframework.integration.dispatcher.Default MessageDispatcher.access$1(DefaultMessageDispatche r.java:191)
at org.springframework.integration.dispatcher.Default MessageDispatcher$DispatcherTask.run(DefaultMessag eDispatcher.java:217)
at org.springframework.integration.scheduling.SimpleM essagingTaskScheduler$MessagingTaskRunner.run(Simp leMessagingTaskScheduler.java:136)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:142)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:166)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

[Mark Fisher]

Are you using 1.0.0.M3 or the latest snapshot?

-Mark

[amsmota]

I do have the latest snapshot on my workspace, but since I'm using maven to build and the top pom has in the dependencyManagement

<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-core</artifactId>
<version>1.0.0.M3</version>
</dependency>

I assume that on runtime is the 1.0.0.M3 that's being used.

[amsmota]

This probably has to do with the implementation I'm trying to do of a synchronous-that-falls-down-to-asynchronous-with-correlationId-when-timedout messages, because the same error (or at least a similar one) is occurring elsewhere besides hot-redeploying...

It's very difficult to debug with time outs and futures!

BTW, I also using modified versions of RequestReplyTemplate, MessageStore, RetrievalBlockingMessageStore ans SimpleMessage, but except for the first all I did was adding a iterator.