Sep 29th, 2011, 07:53 AM
Generic Slave for Remote chunking
I'm trying to design a batch architecture based on spring-batch and JMS. I have read a lot about remote chunking and I have a question.
I've seen that the slave that executes de ItemProcessor needs to create an specific context for each job. If you don't know the job beforehand, is there a way to create a generic slave, and along with the data pass the job definition (or the specific part of it)?
Nov 3rd, 2011, 02:53 AM
I have been investigating how to implement a generic slave. As I want the slave to be generic, the ChunkHandler shouldn't be initialized with a preconfigured ChunkProcessor. The ChunkProcessor must be passed to the ChunkHandler.
I have thought to pass the ChunkProcessor within the JMS message. To do this, ChunkProcessor must be Serializable.
One thing that I don't know is if this overhead of unmarshalling the ChunkProcessor is going to be expensive.
Is this a sensible way to go?
Nov 4th, 2011, 09:05 AM
I have seen that GridGain has done something similar:
Dec 1st, 2011, 04:46 PM
Finally I have make it work. I did 3 things:
- make all the objects serializable, specially the one's inside the ChunkProcessor
- use annotations for the custom objects
- enable loadtime-weaving in the remote slave to reconstruct the annotated objects
If I can, I will upload the code as an example
Dec 1st, 2011, 07:30 PM
do you know same way how to executed partitioned step remotely.
Dec 2nd, 2011, 01:12 AM
Take a look at this project. There are plenty of examples.
Also is worth having a look to the different integrations with grids.
I have done my example with JMS (JBoss). How are you planning to pass the data to the slaves?
Dec 2nd, 2011, 01:17 AM
I have followed the dave syers example , i am not passing data to remote nodes. The data is there in the database. It picks from database and executes.But i have problem running remote step it says with following error
step executions for given job execution are expected to be already saved
and i have posted in this forum.