Passing an object between Listener and Writer

[springforever]

Hi,

Requirement-1
To create a unique id (load_id) before execution of the step begins.
Solution-1
Have used CustomListener class (implementing StepExecutionListener), beforeStep method to create load_id. Then, inserted this load_id and and job_instance_id in the table A. Also, stored load_id in the ExecutionContext.

Requirement-2
To get this load_id in the step, possibly in the writer class. (without accessing the db again... probably from the running ExecutionContext)

How do i get the load_id in the writer class? Can i access ExecutionContext for this step execution in the writer class?

Thanks in advance!
SpringForever

[dkaminsky]

1. Have your writer implement ItemStream (or extend AbstractItemStreamItemWriter if you are not already extending another class) if it doesn't already do so.

2. Add a private ExecutionContext member variable

3. Implement/Override/Add To the "open(ExecutionContext)" method to set the member execution context to the passed execution context. (If you are not extending the Abstract implementation, you may also need to add a no-op implementation of update(ExecutionContext) and close(ExecutionContext).

4. Add to / retrieve from the execution context whenever you want during the write method.

//QED

This is currently the best practice for making ItemReader/Writer collaborators "ExecutionContext aware"

Hope this helps

[springforever]

Thank you! It works.

[springforever]

Another Question (Related to Writer and Listener):

Writer_1 writes to the table A. What would be the best way to write to the table B, if there is a CommaDelimited error? (i think i would catch the error in "onErrorInStep" method in the listener and write to the table B)

Do i have to create writer_2 to write into the table B?

Thank you.

[dkaminsky]

No, you don't have to create a separate ItemWriter unless you really feel like it, since you will be manually invoking the logic yourself -- using the ItemWriter only buys you functionality when it's surrounded by framework code.

What you'll most likely want to do is to create a transaction inside your onErrorInStep method using spring-tx (this will require you to inject your transaction manager into your listener) that performs your write.

Take a look at how transactions are handled in ItemOrientedStep for more information.

You might also want to wrap it in a RetryTemplate callback, to give it several chances to write to table B.

The real concern is: what do you do when the write to table B fails?

[springforever]

Thanks dkaminsky!

Is there a way to implement the following scenario using Spring batch? (with respect to the table A and B above)

File has 4 records:
Rec-1 -> good
Rec-2 -> CommaDelimited error
Rec-3 -> good
Rec-4 -> CommaDelimited error

Output:
table A -> empty
table B -> Rec-2 and Rec-4 (all error records should be logged in table B)

Flow explanation:
-Job reads all the records.
-If all records are valid (no CommaDelimited error) then file data goes to table A.
-If one or more record is invalid then invalid records should be logged in table B, while table A will be left empty. (So, i think transaction will need to be applied to the entire file in one chunk)...In this case, transaction will be rolled back, job fails and job will be re-processed if a new file is provided.

Questions:
-Is there a way to implement this using Spring batch?
-What if i create a step, which does these validations and if all records are valid then the job will go to the next step, which will insert file data in table A? Can TaskletStep as the first step solve this problem?

Thank you for your help!
SpringForever

[dkaminsky]

Basically, you want your task of writing the file to the database to be all or nothing... You could tackle this a number of ways depending on your preferences - here are a couple of ideas

Option 1: "Simple" Approach - One Step with Listener

• item reader reads from file (FlatFileItemReader)
• no-op item writer (write an implementation with blank methods)
• add a step listener with
  
  • afterStep method defined to write whole file to database
  • onErrorInStep method defined to write bad records to database

Your listener would also have to count errors or keep a boolean flag as to whether or not any exceptions were encountered, since your exception handling method will not cause your job to halt. Your afterStep method will need to check to see if any errors were encountered, and only perform the write if none were.

Option 2: Modified "Simple" Approach - Two Steps with Listener
Exactly the same as option 1 except instead of defining your logic to write the whole file in an afterStep method you will create a 2nd step with the same item reader but an actual database-related (e.g. JDBC) item writer. The afterStep method would only contain the logic as to whether or not the job should terminate before moving on to the next step.

Option 2 is my preferred option of the two because although it duplicates the read, it puts more of the potentially failed operations (i.e. database writes) in framework-controlled code, allowing for more easily configurable repeat behavior, error handling, etc.

Option 3: Data Modeling Approach - One Step with Listener
Instead of making this a purely Spring Batch concern, model your database to stage the data as it is read and then just have Spring Batch "commit" it at the end of the step (or job). Most databases support the creation of session-scoped temporary tables, so you can leverage this to achieve your goal.

• run a query to create a temporary table in your beforeStep
• use the item reader (FlatFileItemReader) to read the file
• write error records in your onErrorInStep method in the listener
• use a database item writer (e.g. JDBC) to write good records to the temporary table
• run a query to copy the data from the temporary table to the real table in your afterStep (and clean up the temp table if your database doesn't do it for you) if no errors were encountered (i.e. onErrorInStep was never called)

All said and done, I'd say 3 is the way to go if you can swing it, 2 otherwise.

[springforever]

Option 2 works best for me.

However, onErrorInStep (1st step of 2 steps) throws an exception at Record-2 and terminates the step. How do i continue processing to the next records?

[Dave Syer]

You could set the skip limit really high?

What I would do is set up a pre-processing step that is less strict with the validation, and weed out the bad records there. Write bad records to your second database table, and good ones to another input file for the second step, so it only has good records to crunch.

[springforever]

I did try with the skip limit. In that case, onErrorInStep never gets called. Do i need to add SkipListner?
When i say CommaDelimited error it's that the file is not well formed.

How to use pre-processing? Is there any document where i can find "how to implement different things of Spring Batch"?