How to handle partitions ?

[arun4]

Hi all,
How is the partioning taking place(in Spring Batch 2.1) using the grid size ?

For example:

I have 10,000 records and If the grid size is 10 , so each threads take 1000 records and executes them in parallell.

1. How the records are splitted across the 10 different threads ? . On what basis the splitting of records takes place ?

2. If each thread reads records from the same database, will there be any object level locking taking place while partitioning ?

3. If one thread fails, how to assign the un-processed records of the failed thread to the remaining threads ?

<code>

<job id="Prices_LoadJob" restartable="true"
xmlns="http://www.springframework.org/schema/batch">
<step id="Prices_Load_ManagerStep">
<partition step="Prices_LoadStep" partitioner="partitioner">
<handler grid-size="10" task-executor="taskExecutor" />
</partition>
</step>
</job>
<bean id="partitioner"
class="org.springframework.batch.core.partition.su pport.SimplePartitioner" />

<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTa skExecutor" />



<step id="Prices_LoadStep" xmlns="http://www.springframework.org/schema/batch">
<tasklet transaction-manager="transactionManager">
<chunk reader="pricesLoadReader" processor="pricesLoadProcessor"
writer="pricesLoadWriter" commit-interval="8">
</chunk>
<listeners>
<listener ref="partitiontepListener" />
</listeners>
</tasklet>

</step>

<bean id="partitiontepListener"
class="batch.core.partition.PartitionStepListener" ></bean>

</code>

Kindly reply at the earliest .

[Dave Syer]

1. How the records are splitted across the 10 different threads ? . On what basis the splitting of records takes place ?

Using a Partitioner. The SimplePartitioner will not work for your use case.

2. If each thread reads records from the same database, will there be any object level locking taking place while partitioning ?

Not unless you do the locking yourself.

3. If one thread fails, how to assign the un-processed records of the failed thread to the remaining threads ?

Restart the job.

[marcuri.mon]

Using a Partitioner. The SimplePartitioner will not work for your use case.

Dave can you suggest how to write a custom paritioner for this purpose. or any other approach to achieve this functionality.

[Dave Syer]

Did you look at the sample (https://fisheye.springsource.org/vie...b.xml?r=3831)?

[arun4]

Hi All,
Thanks for your reply dave.
I have to process 1,00,000 records, my grid size is 5,
ColumnRangePartitioner will Split the records like
1 - 20,000 --partition1
20,000 - 40,000 --partition2
40,000 - 60,000 -- partition3
60,000 -80,000 -- partition4
80,000 -1,00,000 -- partition5

If the number of records need to process starts from the range 80,000 to 1,00,000 means, only partition5 will initiate the process and remaining all the partition will not be initiated.
Then this approach is not so efficient for my use case.

Dave can you suggest any other approach. ( i am using spring batch 2.1)

Thanks for your reply in advance..