dm Server undeploys my application

[Iassan]

Hi everyone,

I have a problem with dm Server which undeploys my application by itself. This application consists of one .properties file, several .jar files and one final .war file. After successfully deploying application I open my browser and try to login into my application. Sometimes it works, I can go past logging in and can work, but much more often dm Server undeploys my application, everything in pickup and several other bundles. My logs show no errors, all I see is:

HD0001I Hot deployer processing 'DELETED' event for file 'MRPWeb.war'.

I don't see any reason for dm Server to do that. I do not touch this file while dm Server is working, I have no antivirus software installed or see any strange behaviour in my system. So I really can say that this file wasn't deleted by anyone, it was dm Server who did this.

Has anyone experienced similar behaviour? Does anyone have a clue or a hint where to look, or what to look for? I'm really stuck and out of ideas. Maybe you need additional information? I will gladly provide whatever you need to resolve this.

Best regards

Jacek Bilski

[Glyn Normington]

I presume this is on dm Server 2.0.x. Please confirm.

I am chasing a somewhat similar bug with a commercial customer but in they undeploy a previously deployed application and then many apparently unrelated bundles are impacted (they are stopped and then restarted).

Your problem does seem different in one respect: the deletion does not appear to be your doing.

Please would you take a look in the diagnostic log serviceability/logs/log.log and see if there is any evidence of some activity preceding the HD0001I message which may be initiating a deletion.

Also, it would be interesting to know if this happens after a clean start (add the "-clean" switch to the startup script invocation).

[Iassan]

Hi Glyn,

I confirm, it's dm Server 2.0.x. We tested both 2.0.0 and 2.0.1, same results. And it seems that doing clean start or not does not have any influence, it still can undeploy our application.

Our logs show absolutely nothing out of ordinary, it looks perfectly OK until "processing 'DELETED' event".

We will be investigating memory parameters and aspects. We feel that those areas could give us a clue as to what's happening. I'll post here any results we'll get.

Best regards

Jacek Bilski

[Glyn Normington]

Hi Jacek

Hmmm. Please confirm that you looked in the log.log and not simply in the eventlog.log, which only contains the console messages.

If so, it seems that the file system watcher thread is somehow getting confused and considering the file to have been deleted. Debugging through that code sequence would be the best way to determine what's going on. I would set a breakpoint on HotDeploymentFileSystemListener.onChange and try to determine the state of the file.

Unfortunately, if some other part of dm Server is doing the deletion, this won't tell you much. One trick there would be to set the permissions of the file which is deleted to read only and hopefully see file I/O exceptions when the deletion occurs.

Glyn

[Iassan]

Hi Glyn,

We tried to make our WAR read-only and, so far, it works, dm Server doesn't (can't) undeploy it. We'll try to find out what process deletes our WAR, and if it is dm Server itself, we'll let you know and maybe try to investigate it.

Best regards

Jacek Bilski

[Iassan]

Hi,

We did find out two things:

1. The process which deletes files in pickup is java.exe itself. So dm Server is to blame (I assume), no other Java processes were running at that time.

2. I looked more closely into my log files. Strange thing, but it seems that uninstalling bundles is caused by exception thrown in my application, but those exceptions are logged much later than undeploying. I have two examples, different exceptions, different applications (yet very similar, their architecture and connections are almost the same), same effect. One time exception (java.lang.IllegalStateException: No WebApplicationContext found: no ContextLoaderListener registered?) is logged 14 seconds after first undeployment, another time exception (my custom exception) is logged 7 seconds after undeployment.

I would have to probably read a bit more about handling exceptions in OSGi services, but it definetly looks wrong for dm Server to undeploy bundles. Second example, with my custom exception thrown in business tier, was properly handled in web tier.

Maybe that would give you some clue as to what happens inside dm Server. Or maybe you already can tell me where to look further?

Best regards

Jacek Bilski

[Glyn Normington]

Isn't this a juicy problem? :-)

The most likely reason dm Server would delete an artifact out of pickup is if the the artifact was undeployed for some reason.

Do you see any evidence in serviceability/logs/log.log that the application fails to start?

You could put a breakpoint on PipelinedApplicationDeployed.deploy(URI location, DeploymentOptions deploymentOptions) and see if the catch block is driven. This class is in the com.springsource.kernel.deployer.core.internal in the kernel deployer bundle.

[Iassan]

Hi Glyn,

Just in case. I grabbed sm Server sources from git, but it seems to me that I'm missing something. How can I build dm Server myself? Where from I can get spring-build. I got that from some other Spring sources, but still I am unable to build dm Server. Where can I get help?

I suppose that, when we find this bug, we'll try to prepare a patch, but to do that we would need to build dm Server.

Best regards

Jacek Bilski

[Glyn Normington]

Get spring build by issuing git submodule update --init from the root directory of the cloned git repo. cd into the build-xxx directory and issue "ant clean clean-integration test" typically. I can provide more details next week, including how to "ripple" changes through the repos into a dm Server packaging build.

[Glyn Normington]

Actually, the simplest way to patch dm Server once you have rebuilt one of its bundles is to copy the updated bundle into all the places in the dm Server directory structure where the original bundle occurred, keeping the same file name as the original. Please come back if this is unclear.

(I'll document rippling in due course, but it currently depends on having write access to certain Amazon S3 storage and this is something we'll be addressing in Virgo. So rippling is not something you could do right now. There is a manual version of rippling involving running the update dependencies script manually, but it's pretty laborious.)