Spring batch - Commit interval overrides pageSize

[kanmisc]

Hi

I am implementing the batch in our project to process huge records from database and I have the below configuration. In this commit-interval is 8, and page size to fetch 10; but regardless of page size it always fetches 8 records at a time. Is it the behaviour or am i missing something in configuration?

<batch:job id="simpleJob" job-repository="jobRepository">
	<batch:step id="step1">
		<batch:tasklet>
			<batch:chunk reader="dataItemReader" processor="dataProcessor" writer="flatFileWriter"
				commit-interval="8" />
		</batch:tasklet>
	</batch:step>
</batch:job>

<bean id="dataItemReader" class="org.springframework.batch.item.database.JdbcPagingItemReader">
	<property name="dataSource" ref="dataSource"/>
	<property name="queryProvider">
		<bean class="org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean">
			<property name="dataSource" ref="dataSource"/>
			<property name="selectClause" value="select name,place,addr,phone"/>
			<property name="fromClause" value="from test"/>
			<!-- <property name="whereClause" value="where status=:status"/> -->
			<property name="sortKey" value="name"/> 
		</bean>
	</property>
	<property name="pageSize" value="10"/>
	<property name="rowMapper" ref="rowMapper"/>
</bean>

Thanks
Kannan

[SaifuddinMerchant]

How did you determine that it fetches only 8 records at a time? Did you see the value in some database level logs?

[kanmisc]

Sorry my mistake. Not sure how many it fetches from database. But after every 8 records it goes to processor and writer. Does it imply that it fetches 10 records at a time and commits 8 at a time in chunk?

[SaifuddinMerchant]

Yes the commit interval determines how many record would be processed in a Chunk. In you case after 8 record are read and processed, the 8 record would go to the writer for committing.

The database page size determines how many record would be fetched from the database in one go. It's more of an optimization setting between how big the buffer would you like to have v/s the number of trips the driver would how to make to fetch the data from the database.

In your case it should fetch 10 records at a time and commits 8 at a time in chunk.
(However I think that the page-size is driver dependent setting and no all drivers honer it!)

[paragflume]

Hi Friends,

I like to know, in the meantime while the "dataItemReader" is reading lines and before it reach the commit-interval=8 and processed by "dataProcessor" and write by "flatFileWriter" where the data get stored. Is those data get stored in memory..