Deployer performance issue: plan file deployment takes several minutes

[peter.gardfjell]

Hi,

I'm experimenting with dm-kernel (not the full dm-server at the moment) as a runtime for my applications. As part of this, I have a set of api/library bundles that I like to think of as a platform for the applications. Anyhow, I have a plan outlining the platform bundles. It looks as follows

<?xml version="1.0" encoding="UTF-8"?>
<plan name="platform.plan" version="2.0" scoped="false" atomic="true"
                xmlns="http://www.springsource.org/schema/dm-server/plan"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:schemaLocation="
                        http://www.springsource.org/schema/dm-server/plan
                        http://www.springsource.org/schema/dm-server/plan/springsource-dm-server-plan.xsd">

  <artifact type="bundle" name="com.springsource.com.mysql.jdbc" version="5.1.6"/>
  <artifact type="bundle" name="com.springsource.org.apache.derby" version="10.5.1000001.764942"/>

  <artifact type="bundle" name="com.springsource.org.eclipse.persistence" version="1.1.0"/>
  <artifact type="bundle" name="com.springsource.org.eclipse.persistence.antlr" version="1.1.0"/>
  <artifact type="bundle" name="com.springsource.org.eclipse.persistence.asm" version="1.1.0"/>
  <artifact type="bundle" name="com.springsource.org.eclipse.persistence.jpa" version="1.1.0"/>

  <artifact type="bundle" name="org.ops4j.pax.web.extender.war" version="0.5.1"/>
  <artifact type="bundle" name="org.ops4j.pax.web.extender.whiteboard" version="0.5.1"/>
  <artifact type="bundle" name="org.ops4j.pax.web.pax-web-service" version="0.6.0"/>
  <artifact type="bundle" name="org.apache.felix.webconsole" version="1.2.8"/>

  <artifact type="bundle" name="org.apache.cxf.bundle-minimal" version="2.2.1"/>
  <artifact type="bundle" name="org.apache.servicemix.cxf.transport.osgi" version="4.0.0"/>

  <artifact type="bundle" name="org.apache.wicket.wicket" version="1.3.6"/>
  <artifact type="bundle" name="org.apache.wicket.wicket-datetime" version="1.3.6"/>
  <artifact type="bundle" name="org.apache.wicket.wicket-extensions" version="1.3.6"/>
  <artifact type="bundle" name="org.apache.wicket.wicket-ioc" version="1.3.6"/>
  <artifact type="bundle" name="org.apache.wicket.wicket-spring" version="1.3.6"/>
  <artifact type="bundle" name="org.apache.wicket.wicket-spring-annot" version="1.3.6"/>
</plan>

and, with dependencies, it pulls in around 40 bundles. What I noticed, and what concerns me, is that starting this plan (by placing the plan file in the pickup directory) takes over two minutes.

[2010-03-21 09:01:58.952] fs-watcher                   <DE0000I> Installing plan 'platform.plan' version '2.0.0'.
[2010-03-21 09:01:59.204] fs-watcher                   <DE0000I> Installing bundle 'com.springsource.com.mysql.jdbc' version '5.1.6'.
...
[2010-03-21 09:04:25.624] start-signalling-3           <DE0005I> Started bundle 'org.apache.servicemix.cxf.transport.osgi' version '4.0.0'.
[2010-03-21 09:04:25.628] start-signalling-3           <DE0005I> Started plan 'platform.plan' version '2.0.0'.

As a comparison, installing the transitive closure of the plan bundles with FileInstall (simply copying all the bundles into FileInstall's load directory) finishes within 20 seconds.

I know that the deployment pipeline of the dm server is a lot more sophisticated but taking several minutes to install a bunch of bundles seems somewhat excessive to me.

Are the startup times I'm seeing to be expected?
I would be happy to see someone comment on my observations.

BTW, I'm looking forward to the Eclipse Virgo move!

best regards, Peter

/PS If anyone would like to look closer into this particular case (launching dm kernel with the same bundle repo and plan file), I'd be happy provide a Maven pom file that gathers the bundles I have in my bundle repo.

[Glyn Normington]

I suspect if you put all 40 bundles in the plan, it would install pretty fast. What is probably happening is that, for some reason, the resolutions that the deployer performs in a side state in order to determine missing dependencies are taking a while and that multiple passes are required because of the shape of the dependency tree.

I can't think of any obvious way to speed up the deployer algorithm, so it may be worth you experimenting with putting some or all of the missing transitive dependencies into the plan, or possibly into a plan referred to by the top-level plan if that's more structured.

[peter.gardfjell]

Thanks Glyn,

I followed your advice and placed all the bundles in the plan file. I even tried to order them so they are processed in the correct dependency order. Unfortunately, this did not improve matters at all. Deployment times are still around three minutes (I should probably mention that I'm running on a pretty good machine so the problem is to be found elsewhere).

[2010-03-28 15:59:37.685] system-artifacts             <DE0000I> Installing plan 'platform.plan' version '2.0.0'.
[2010-03-28 15:59:37.784] system-artifacts             <DE0000I> Installing bundle 'com.springsource.org.aspectj.runtime
' version '1.6.2.RELEASE'.
...
[2010-03-28 16:02:29.045] start-signalling-2           <DE0005I> Started bundle 'org.apache.servicemix.cxf.transport.osg
i' version '4.0.0'.
[2010-03-28 16:02:29.049] start-signalling-2           <DE0005I> Started plan 'platform.plan' version '2.0.0'.
[2010-03-28 16:02:29.059] Thread-2                     <UR0001I> User region ready.

Is there anything else I can try to improve the situation?
Is there e.g. any way of by-passing the (complex) resolution pipeline and just have the deployer make an attempt to install, resolve and start the plan bundles (much like the fileinstaller does)? If not, would it make sense to add such a mechanism: marking a plan as "self-sufficient" or something similar?

In this case I'm paying a high price (time-wise) for resolution functionality that I don't need. Taking over three minutes just to get the server up on its feet (I haven't even deployed the actual application) just isn't good enough, so I would really love to see a solution to this situation. Any advice would be much appreciated!

best regards, Peter

[Glyn Normington]

There is a way to bypass the deployment pipeline which you may like to experiment with. It's not recommended for general purpose use as it modifies the contents of the kernel region and therefore reduces the isolation of the kernel. But it would certainly be interesting to compare the performance and it may even be satisfactory for your "platform".

Place the bundles (including their transitive dependencies) in lib/kernel, or in any other suitable directory, and then edit lib/com.springsource.kernel.launch.properties and add your bundles in dependency order to the list in the launcher.bundles property.

It should then be possible to start the kernel and measure the start-up time. The bundles will be installed and started (provided you add @start after each one in the above property) before the user region is created. If you want to inspect the contents of the kernel region after start-up, add the property:

osgi.console=2402

(with a different port number if 2402 is already taken) to the com.springsource.kernel.launch.properties and then you can telnet to this port and run the Equinox console inside the kernel region.

If this works, it would then be possible to modify config/com.springsource.kernel.userregion.properties to import the relevant packages and services (that is, the packages and services exported by your bundles that need to be available in the user region) from the kernel region and export any relevant services (that is, any services which need to be published in the user region but found by your bundles in the kernel region) from the user region.

Please note that with the current nested framework implementation in Equinox, your bundles will not be visible as bundles in the user region. The packages and services you import from then kernel region will appear to be exported by the "surrogate" bundle "com.springsource.region.user". The surrogate bundle will also appear to consume any services you export from the user region.

Longer term the idea of marking a plan as "already transitively complete" looks plausible. This is one of the ideas being considered in the OSGi Enterprise Expert Group in the work to standardise an application model (based on input from SpringSource, IBM, and others).

[peter.gardfjell]

Thanks Glyn,

I followed your instructions and it really improved the situation. The startup time was drastically reduced. Now it takes about ten seconds to start the server (including all platform bundles).

$ ./bin/startup.bat -clean
The system cannot find the file specified.
Listening on port 2402 ...
[2010-04-02 10:59:17.742] startup-tracker              <KE0001I> Kernel starting.
[2010-04-02 10:59:22.557] kernel-dm-13                 <SH0001I> dm Kernel ssh shell available on port 2401.
[2010-04-02 10:59:22.684] startup-tracker              <KE0002I> Kernel started.
...
[2010-04-02 10:59:27.052] start-signalling-1           <DE0005I> Started plan 'com.springsource.kernel.userregion.spring
dm' version '2.0.0'.
[2010-04-02 10:59:27.063] Thread-2                     <UR0001I> User region ready.

I checked the osgi console and all bundles were successfully started (they are in the ACTIVE state).

In the long run I don't see this as a good solution, however. In fact, I am trying to get away from maintaining this type of fine-grained platform configuration. I really find the concept of being able to deploy the platform as a single compound artifact appealing.

I don't see my platform as overly complex either. Quite on the opposite, I see it as a bare minimum for writing enterprise apps (a web framework, JAX-WS, JPA, database connectors, etc). I'm imagining that this is quite a common (if not _the_) use case for the spring dm server. Have other developers, with similar "platforms", reported similar startup times? What about the sample applications you guys have written? It doesn't add up for me. Perhaps my platform is a particulary unfortunate mixture of bundles.

I guess I have to wait until the OSGi EEG will agree on a "transitively complete" application/subsystem spec, or that a similar feature makes its way into Spring dm/Eclipse Virgo.

best regards, Peter

[Glyn Normington]

It's strange because a transitively complete plan should only need resolving once in the side state and once in the OSGi framework and so should be quite fast. I wonder if you've hit an edge condition where the Equinox resolver takes a long time to perform resolution? Running with a profiler, such as YourKit, is the next logical step.