Example of having two PropertyPlaceholderConfigurer in same application context

[staleks]

Hi all,

I've started with investigation of Spring-Batch for my next project. Beside spring-batch core I would also like to add web-console of spring-batch-admin in order to get more info/control over jobs.

Therefore I need two DataSources: one that spring-batch uses and other one is DataSource that I am using to connect to my DB - MySQL.

For spring-batch internal datasource that is used to store/monitor/control jobs external property file works fine.

But once I want to have same feature, external property file to hold database specific stuff, like URL, username and password, new property placeholder is never used and I can't have parametrized datasource bean.


Does anyone encounter same problem ?

[Gary Russell]

First, you should really have the Spring Batch tables in the same database as your application. Otherwise you'll need to use an XA transaction manager with the overhead of 2 phase commit.

That said, you can add as many property placeholders as you want; you just have to define it as a bean and set placeholderPrefix and placeHolderSuffix properties (you can't do this through the context namespace). For example...

	<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
		<property name="location" value="my.properties"/>
		<property name="placeholderPrefix" value="!abc{"/>
		<property name="placeholderSuffix" value="}"/>
	</bean>

Then you can have placeholders such as

value="!abc{my.property}".

You can have as many as you want, just give them unique patterns to match.

If you are using Spring 3.0, another alternative is to use a util properties bean, together with SpEL expressions:

	<util:properties id="props" location="my.properties"/>

You can then refer to individual properties from the file with

value="#{props['some.property']}"

[staleks]

Why do I have to have all in one DB ?

I am really happy with solution to have JOBS described in in-memory DB, while my JOBS should do their work in regular DB.

Why I choose not to have XADataSources? Well I don't have any transaction critical operations.

Short description: I have live DB where some users are stored, and I have background job that should be triggered every evening to check how many users are not logged in last 3 months. So this is just simple SELECT COUNT(*) on users table.

And to involve XA in this story, well no thank you. I prefer to keep it simple.


In my web module (presentation layer - another Maven2 WAR module beside this spring-batch) I have easily added two PropertiesPlaceholders, one jdbc.properties for DataSource, and other mail.properties for SMTP specific stuff.

I really don't see any reason why I can't have same feature here in spring-batch ?

Any good explanation? Lead ? Anything that would be helpfull?

Thanks again for your time.

[Gary Russell]

If you don't need persistent job meta data for restart and recovery, two datasources are fine.

If you need to be able to restart a job after a failure (where the job stores meta data about the job's progress so batch can figure out where to restart the job) then you need a persistent repository and, preferably, the tables are in the same database as the application data, to avoid XA.

Regarding the properties file; there is some special handling of application context loading (each job gets its own application context to avoid bean name collisions). Refer to the batch admin documentation (override scenarios) for techniques to modify the property placeholder handling.

[joshlong]

quick aside at a simpler level, if you're getting an error, check that the first propertyplaceholderconfigurer isnt trying, and failing to resolve properties that the second propertyplaceholderconfigurer can resolve. Use the ignoreUnresolvable attribute or property to make the first one give up quietly so that the second one can attempt to resolve the property for you. Otherwise, you'll get an exception which will proclude you from starting up.

<beans:beans
        xmlns:beans="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:context="http://www.springframework.org/schema/context" xmlns:ws="http://www.springframework.org/schema/integration/ws"
        xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
         ">


    <context:property-placeholder ignore-unresolvable="true" location="a.properties"/>
    <context:property-placeholder ignore-unresolvable="true" location="b.properties"/>

    <beans:bean class="my.Cat">
        <beans:property name="name" value="${a}"/>
    </beans:bean>
    <beans:bean class="my.Cat">
        <beans:property name="name" value="${b}"/>
    </beans:bean>


</beans:beans>

[staleks]

Hi Josh,

first of all I wish to say that I was pretty impressed @Devoxx2010 firstly on Spring BOF and later that week with your presentation about new stuff comming from Spring.
Great work.

While there are many peoples still using Spring 2.5, me and my team in my company are trying to keep track with you guys and we started new project in October 2010 using Spring 3. On Spring BOF someone of you guys from Spring asked about some Spring products and mentioned Spring Batch. I was interested about this project and waited for chance to introduce it. And finally I managed to "sell this feature" to PM's

Well to get to the point, I made it work:

1. I have a web front end application that will deployed to multiple servers, in order to increase availability and to support load balancing.

2. Beside that I have a few, let's say background jobs, something like "services". I didn't want to "polute" Web Front End application with them using quartz, since they don't have to be deployed to that many servers and to use hardware resources while there is no need to do that.

3. So my decision was to make it separate WAR that will be deployed to one(or maybe more) servers, totally independent of server where web front end application is running. But again this WAR (background job) should target live DB in order to do some calculation and to mail those results to recipients.

So this is just short description of architecture of system.

Now I came to the idea that it would be nice to use spring-batch as shell for those background jobs and to have one more feature extra to have web console to "monitor and control" jobs.

Just to make myself clear about this, I made it work, but I really don't want to have those parameters (DataSource URL, DataSource username, etc.) in my application context file. I would like to have them in properties file as they should be.

Now about your latest post,

I am using slf4j as facade to logback. I've enabled DEBUG log level on root logger. So I am sure that spring-batch loads own properties file

10:50:36.458 [main] INFO  o.s.b.f.c.PropertyPlaceholderConfigurer:177 - Loading properties file from class path resource [org/springframework/batch/admin/bootstrap/batch.properties]
10:50:36.463 [main] INFO  o.s.b.f.c.PropertyPlaceholderConfigurer:177 - Loading properties file from class path resource [batch-default.properties]
10:50:36.464 [main] INFO  o.s.b.f.c.PropertyPlaceholderConfigurer:177 - Loading properties file from class path resource [batch-hsql.properties]

And again there is a bean definition of DataSource in following file:
spring-batch-admin-manager-1.0.0.RELEASE\META-INF\spring\batch\bootstrap\manager\env-context.xml

	<!--  Use this to set additional properties on beans at run time -->
	<bean id="placeholderProperties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
		<property name="locations">
			<list>
				<value>classpath:/org/springframework/batch/admin/bootstrap/batch.properties</value>
				<value>classpath:batch-default.properties</value>
				<value>classpath:batch-${ENVIRONMENT:hsql}.properties</value>
			</list>
		</property>
		<property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
		<property name="ignoreResourceNotFound" value="true" />
		<property name="ignoreUnresolvablePlaceholders" value="false" />
		<property name="order" value="1" />
	</bean>

So if I am following your logic right and understand you correctly, you were wondering that spring-batch PropertyPlaceholder bean property "ignoreUnresolvablePlaceholders=false" is making me headache?

In my own "job description application context file", I have something like this:

	<util:list id="propertiesLocations">
		<beans:value>classpath:com/telegraaf/relatieplanet/fakeprofile/cfg/jdbc.properties</beans:value>
		<beans:value>classpath:com/telegraaf/relatieplanet/fakeprofile/cfg/suspicious-profiles.properties</beans:value>
	</util:list>
	
	<beans:bean id="appDataPlaceholderConfig" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
		p:ignoreUnresolvablePlaceholders="false"
		p:ignoreResourceNotFound="true"		
		p:locations-ref="propertiesLocations" />

	<beans:bean id="fakeProfileDS" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
		p:driverClassName="${jdbc.relatieplanet.driver}"
		p:url="${jdbc.relatieplanet.url}"
		p:username="${jdbc.relatieplanet.username}"
		p:password="${jdbc.relatieplanet.password}"
		p:validationQuery="${jdbc.relatieplanet.commons_dbcp.validationQuery}" />

And this doesn't work. But again if I replace property placeholders with real data like

	<beans:bean id="fakeProfileDS" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"
		p:driverClassName="com.mysql.jdbc.Driver"
		p:url="jdbc:mysql://localhost:3306/rpl_test"
		p:username="root"
		p:password="root"
		p:validationQuery="SELECT 1" />

This works like a charm


And error that I am getting is that there are some unresolvable properties:
${jdbc.relatieplanet.driver}


Quite big explanation, sorry for such big post.

[joshlong]

Thanks for the kind words.

I'm not clear what the exact issue you are facing is. Could you provide the exception or stack trace?

If you have two PropertyPlaceholderConfigurers in the same application context, you'll get an exception. If you do, then set ignoreUnresolvableProperties to be "true" for both PropertyPlaceholderConfigurers so that you won't get an exception. Without the error or exception output, I'm afraid I'm only guessing... I could be completely wrong :-)