ThreadPoolTaskExecutor never completed

[andibaso]

Hi All,
I have problem handling partitioning with ThreadPoolTaskExecutor. This configuration is working fine with 27 records processed in one seconds, but my goal is processing 120 records per second. When i enable task-executor handler, batch job execution never completed. Do you have any suggestion how to run several threads to process my partitioning data?.
FYI, my company code record count is greater than 50 records.


My batch configuration is :

  
<beans:bean id="lendingMasterItemReader"
                class="org.springframework.batch.item.database.HibernateCursorItemReader"
                scope="step">
        <beans:property name="sessionFactory" ref="sessionFactory"/>
        <!--
        <beans:property name="queryString"
                        value="from LendingMaster lm where  lm.status = 'NEW'"/>
       -->

       <beans:property name="queryString"
                       value="from LendingMaster lm where  lm.status = 'NEW' and lm.companyCode = :companyCode"/>
       <beans:property name="parameterValues">
           <beans:map>
               <beans:entry key="companyCode" value="#{stepExecutionContext['companyCode']}" />
           </beans:map>
       </beans:property>

        <beans:property name="fetchSize" value="10"/>

    </beans:bean>


    <beans:bean id="lendingMasterItemProcessor"
                class="com.ati.if.service.BatchLendingItemProcessor">
    </beans:bean>

    <beans:bean id="lendingMasterItemWriter"
                class="org.springframework.batch.item.database.HibernateItemWriter">
        <beans:property name="sessionFactory" ref="sessionFactory"/>
    </beans:bean>

    <step id="processNewLendingRecord">
        <tasklet>
            <chunk reader="lendingMasterItemReader" processor="lendingMasterItemProcessor"
                   writer="lendingMasterItemWriter"
                   commit-interval="10"/>
        </tasklet>
    </step>

    <beans:bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
        <beans:property name="corePoolSize" value="5"/>
        <beans:property name="maxPoolSize" value="5" />
        <beans:property name="queueCapacity" value="100"/>
    </beans:bean>

    <beans:bean id="partition" class="com.ati.if.service.BatchLendingItemPartition">
    </beans:bean>


    <job id="lendingMasterJob">
        <step id="partitioningStep">
            <partition step="processNewLendingRecord" partitioner="partition">
               <!-- <handler task-executor="taskExecutor" grid-size="10"/> -->
            </partition>
        </step>
    </job>

 <beans:bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <beans:property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver"/>
        <beans:property name="url" value="jdbc:jtds:sqlserver://localhost:1433/ati_if"/>
        <beans:property name="username" value="sa"/>
        <beans:property name="password" value="123456"/>
        <beans:property name="maxActive" value="1000"/>
        <beans:property name="maxWait" value="1000"/>
        <beans:property name="poolPreparedStatements" value="true"/>
        <beans:property name="defaultAutoCommit" value="true"/>
        <beans:property name="initialSize" value="20"/>
    </beans:bean>

Thanks,
Andi

[sambaran]

When you say it is not COMPLETED...how do you know it is not completed..did you look at the Spring Batch tables ? What is the exit code ? Do you see any exceptions ?

[andibaso]

I saw it in the BATCH_JOB_EXECUTION table, the last 5 records with END_TIME = null, STATUS = STARTED, EXIT_CODE = UNKNOWN. And my application is never exited so i must use CTRL+C to exit my application.

Thanks

[sambaran]

That is very interesting. Are you overriding any of the EXIT_STATUS'es using any listeners? May be the spring batch thinks that are are actually processing your business logic but in reality your are done with the job.

[andibaso]

No, I don't use any listeners. This is happen only if i activate ThreadPool. Another situation, when my applications is "hang", then when i run simple query like counting updated record, it keep running. I suspect, my threads lock my table.

[sambaran]

Can you try this and let me know :

<bean id="jobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean" >
		<property name="databaseType" value="oracle" />// your DB
		<property name="dataSource" ref="dataSource" />// your DS
		<property name="transactionManager" ref="transactionManager" /> //Your TM
		<property name="isolationLevelForCreate" value="ISOLATION_DEFAULT"/> // default isolation from DB vendor
	</bean>

[andibaso]

Hi, Thanks for your reply. I change my partition class to use id range. But i can not reach 100 records processed for seconds because sql server is very busy handling read and write process. Do you have reference (maybe a simple formula)how to configure connection pooling, thread pool size, commit interval to gain maximum performance?

This is my latest config in case another person having some problem :

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/batch"
             xmlns:beans="http://www.springframework.org/schema/beans"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://www.springframework.org/schema/beans
                    http://www.springframework.org/schema/beans/spring-beans-3.0.xsd 
					http://www.springframework.org/schema/batch 
	                http://www.springframework.org/schema/batch/spring-batch-2.1.xsd">

    <beans:import resource="applicationContext.xml"/>
    <beans:import resource="applicationContext-dao.xml"/>
    <beans:import resource="applicationContext-service.xml"/>
    <beans:import resource="applicationContext-launch.xml"/>
    <beans:bean id="lendingMasterItemReader"
                class="org.springframework.batch.item.database.HibernateCursorItemReader"
                scope="step">
        <beans:property name="sessionFactory" ref="sessionFactory"/>
        <beans:property name="queryProvider">
            <beans:bean class="org.springframework.batch.item.database.orm.HibernateNativeQueryProvider">
                <beans:property name="entityClass" value="com.ati.if.model.LendingMaster"/>
                <beans:property name="sqlQuery"
                                value="select * from lending_master where id &gt;= :minId and id &lt;= :maxId and status='NEW'"/>

            </beans:bean>
        </beans:property>
        <beans:property name="parameterValues">
            <beans:map>
                <beans:entry key="minId" value="#{stepExecutionContext[minValue]}"/>
                <beans:entry key="maxId" value="#{stepExecutionContext[maxValue]}"/>
            </beans:map>
        </beans:property>
        <beans:property name="useStatelessSession" value="true"/>
        <beans:property name="fetchSize" value="80"/>
    </beans:bean>


    <beans:bean id="lendingMasterItemProcessor"
                class="com.ati.if.service.BatchLendingItemProcessor" scope="step" >
    </beans:bean>

    <beans:bean id="lendingMasterItemWriter"
                class="org.springframework.batch.item.database.HibernateItemWriter">
        <beans:property name="sessionFactory" ref="sessionFactory"/>

    </beans:bean>

    <step id="processNewLendingRecord">
        <tasklet>
            <chunk reader="lendingMasterItemReader" processor="lendingMasterItemProcessor"
                   writer="lendingMasterItemWriter"
                   commit-interval="24" retry-limit="100">
                <retryable-exception-classes>
                    <include class="org.springframework.dao.DeadlockLoserDataAccessException"/>
                </retryable-exception-classes>
            </chunk>
        </tasklet>
    </step>

    <beans:bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
        <beans:property name="corePoolSize" value="8"/>
        <beans:property name="maxPoolSize" value="8"/>
    </beans:bean>

    <beans:bean id="partitioner" class="com.ati.if.service.BatchColumnRangePartitioner">
        <beans:property name="dataSource" ref="dataSource"/>
        <beans:property name="table" value="LENDING_MASTER"/>
        <beans:property name="column" value="ID"/>
    </beans:bean>


    <job id="lendingMasterJob">
        <step id="partitioningStep">
            <partition step="processNewLendingRecord" partitioner="partitioner">
                <handler task-executor="taskExecutor" grid-size="8"/>
            </partition>
        </step>
    </job>
   
    <beans:bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <beans:property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver"/>
        <beans:property name="url" value="jdbc:jtds:sqlserver://localhost:1433/ati"/>
        <beans:property name="username" value="sa"/>
        <beans:property name="password" value="111222333"/>
        <beans:property name="maxActive" value="1000"/>
        <beans:property name="maxWait" value="1000"/>
        <beans:property name="poolPreparedStatements" value="true"/>
        <beans:property name="defaultAutoCommit" value="false"/>
        <beans:property name="initialSize" value="20"/>

    </beans:bean>

</beans:beans>

[sambaran]

I do not think there is any magic formula.
The general advice is to took at your queries first and see if that can be tweaked.
If thats fine then use task executors for stateless processing.
Try to look at some batch insert/updates may be thats going to decrease the write time a lot. I have used the Spring JDBC batch updates and it is very fast
Also try tweak your commit intervals, fetch sizes , throttle limits. You can also look at other implementations of the task executors (I do not know how much of a difference it will make)

Look at:
http://static.springsource.org/sprin...tithreadedStep