dm Server undeploys my application

[Iassan]

Hi Glyn,

We got that far:

In class com.springsource.util.io.FileSystemChecker there are two interesting methods: listCurrentDirFiles and check. Second one scans every second pickup directory to check if there are any changes. This method uses listCurrentDirFiles to get all files in pickup directory. This is done by calling standard Java method java.io.File.listFiles(FilenameFilter). What we found out is that once in a while this method returns null, even though there are files in pickup. Java documentation for listFiles method says:

Returns null if this abstract pathname does not denote a directory, or if an I/O error occurs.

We digged a bit in Java sources, but cannot see anything interesting. Searching the web reveals that some users had similar problems with listing files in directories. One thread is especially interesting: http://72.5.124.102/thread.jspa?messageID=2455691. We'll try to go that way and increase open file descriptors available.

If you have any other ideas, we're open.

Best regards

Jacek Bilski

[Glyn Normington]

Hi Jacek

That's a great piece of detective work. Well done!

I suppose it's possible that dm Server is leaking some type of file reader by failing to close them. You could wait until the problem manifests itself and then trigger a heap dump and use a heap analysis tool to see if there are a large number of lives objects corresponding to readers, input streams, etc.

I guess the rate of such a leak would be affected by the number of files that dm Server processes. Do you have a particularly large number of artifacts either in pickup or in repository/usr (which is a watched repository and therefore scanned periodically)? Bit of a long shot, but worth asking.

Please come back when you need more ideas. If this could be captured in a testcase, then we could look at it.

Regards,
Glyn

[iluu]

Hello Glyn,

I'm trying to resolve the mysterious Null problem together with Jacek. And unfortunately I have to report we are running out of ideas...

We have tried to increase number of file handlers but it didn't help.
We've also searched the code for any input/output streams left open - non found.
We've put some double check on null - but the problem still appears.

One thing keeps bothering me, I'm testing our application on DM Server on Ms Virtual PC, and the error occurs about 1 time for every 20 tries, but Jacek has Sun VirtualBox and the undeployment happens in 19/20 times.

The number of artifacts you asked about, it's not that big I think (9 in pickup and 45 in repository/usr).

I wish we could capture this in a testcase, but as the reason remains unknown it is nearly impossible.

Best regards
Karolina Rusin

[Glyn Normington]

I had a careful trawl through the kernel artifact handling logic, which is the most intensive area of file usage in dm Server, and found that the ServiceScoper class contains two loops which get input streams but do not close them. This is clearly a bug which needs to be fixed, so I have raised DMS-2470 and will fix this in the 2.0.2 release.

The number of iterations of these loops depends on the number of XML files present in META-INF/spring directories of bundles of scoped plans and PARs. I wonder if you have a larger than typical number of these XML files?

Or are you deploying some scoped plans or PARs multiple times before the problem occurs?

Anyway, would you be willing to try out a fix in the form of a patched bundle, as I could produce that sooner than the 2.0.2 release? It's somewhat of a shot in the dark, but it certainly closes a possible source of the problem you are seeing.

[iluu]

Hi Glyn,

Of course we are willing to try.

Best regards
Karolina Rusin

[Glyn Normington]

Ok. I'll leave the fix building over lunch and then be in touch later today.

[Glyn Normington]

Please down the patched deployer bundle from http://dl.dropbox.com/u/2487064/com....l.deployer.jar and copy it to both the following files in dm Server 2.0.1.

./lib/kernel/com.springsource.kernel.deployer-2.0.1.RELEASE.jar
./repository/ext/com.springsource.kernel.deployer-2.0.1.RELEASE.jar

It is important to preserve the above filenames.

Does this make any difference to the problem you are seeing?

[Iassan]

Hi Glyn,

Unfortunately this patch of yours didn't help us. Maybe it solves some problems anyway, but we had to create a workaround. I attach patch we created, it's agains util project. It just ignores null result from File.listFiles(). If you find it helpful, you can apply it on dm Server.

And thanks for the help, it was much easier to do it with you that struggle alone.

Best regards

Jacek Bilski

[Glyn Normington]

Sorry to hear that. We'll consider adding your patch, or equivalent, to 2.0.2.

[Steve Powell]

Dear Jacek,

If you have been following the dm Server jira issues, you may have noticed DMS-2489 recently.

In it we have altered the implementation of FileSystemChecker, but instead of making a point fix, we have introduced a collection of replacements for File.list*(*) methods with the corresponding signatures: FileSystemUtils.list*(File,*). These wrap the original calls, guarantee never to return null, retry if the underlying calls return null, and throw a RunTimeException (unchecked) if retry doesn't help.

The changes have meant that usage in other parts of dm Server is updated to use FileSystemUtils.list*(File,*) in a few places.

I am keen to see if these changes fix your problem. Could you try them out for us? The latest build of dm Server 2.0.x is here, with SHA1.

Respond here (or let me know directly) if these links don't work for you.

Thank you

Steve Powell