skipping a block of code in the Partitioner code

[gerson721]

The framework could skip that block if allowStartIfComplete=false and we are in a restart. Wouldn't that be better?

Hi Dave,

Reading the API, it sounded like setting allowStartIfComplete to false would only make my application skip the steps that have successfully finished and only rerun the failed ones. The API also states that the default value of allowStartIfComplete is false so I didn't set it in my code anymore. I had a go at it again and explicitly set

allow-start-if-complete="false"

in my tasklet but the block of code that does the partitioning still runs everytime the job is restarted.

[snicoll]

I assume you meant true. That's exactly my problem. IMO, the partitioner should not run again but just reuse the stepexecution that were created previously.

[Dave Syer]

the block of code that does the partitioning still runs everytime the job is restarted.

I know. I understood the last post already. I offered a couple of workarounds and asked a question about whether a potential solution would work in your case. I'll ask that one again: if the framework would skip that block only if allowStartIfComplete=false and the partition step is restarting would that help? (No promises that will be the actual implementation or in what future version, but open a JIRA, and it might make it to 2.1.1.)

[snicoll]

As I understood it, that is allowStartIfComplete=false on the step defining the partition and the underlying step is restarting then no partition will be computed again, yes it'll help us.

The current behavior is broken right now anyway since if the partitioner decides to create optimized partitions for the restart, spring batch overrides the created execution context with the previous one if a given partition has failed (and does not if the partition has succeeded!)

[gerson721]

I assume you meant true. That's exactly my problem. IMO, the partitioner should not run again but just reuse the stepexecution that were created previously.

Actually I meant false. I think the default behavior for the partitioned steps (the slave steps) is to only rerun the failed steps if allowStartIfComplete is set to false. The PartitionStep (the master step) however still runs the block of code that does the partitioning everytime you restart the job even if allowStartIfComplete's value is set to false.

I managed to solve my problem by just moving that partitioning bit of the code to its own Step and saved its output to the JobExecutionContext. I wouldn't want the partitioner to run this block of code everytime I restart the job as it will sometimes take a few minutes for it to finish executing.

[gerson721]

if the framework would skip that block only if allowStartIfComplete=false and the partition step is restarting would that help?

Yes, that feature will be really useful. :-)

[grossh]

Hi,
this feature would be very useful for us too.

Because I could not find a JIRA entry for this, I have created one.

http://jira.springframework.org/browse/BATCH-1531

thanks Harald

[gerson721]

Thanks for creating a JIRA entry for this Harald.

Hi Dave,
You mentioned a couple of posts back that we might even get lucky and have this feature available to us on version 2.1.1. What's needed for this feature to get a higher priority and be released as part of 2.1.1? The reason I'm asking is because my "put-the-data-that-the-partitioner-code-generates-in-the-job-execution-context-on-a-separate-step" workaround only sometimes work. The data sometimes get too big that it wouldn't fit the DB column.

Here's the exception I'm getting when the information that I've gathered becomes very very big.

org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; 
SQL [UPDATE BATCH_JOB_EXECUTION_CONTEXT SET SHORT_CONTEXT = ?, SERIALIZED_CONTEXT = ? WHERE JOB_EXECUTION_ID = ?]; 
Data truncation: Data too long for column 'SERIALIZED_CONTEXT' at row 1; 
nested exception is com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'SERIALIZED_CONTEXT' at row 1

I'll be able to avoid this issue is the data is split per Step Execution Context, ie. have the actual partitioner code executed in the Partitioner class.

Thanks,
Gerson