Sep 26th, 2012, 10:06 AM
Design question - best way to read data
I wanted some input/suggestions on the best way to approach a problem: our input to the batch processor is a list of records from a third party web service. Unfortunately, their API doesn't support anything other than getting everything in one call. Furthermore, after the reading from the web service, we have to fetch some matching data from our system and see if the two can be linked. The union of that information (i.e. zero or one third party records and zero or one of our records) is what the reader sends to the processor.
I'm trying to figure out the best way to load the data. Right now, we have a reader bean which has a loop controller bean. The reader reads from the loop controller. The loop controller calls the web services in its constructor. Our application-context is setup so that when the reader bean is created, the loop controller bean is autowired (and thus makes the web services call). The problem with this approach is that all of the work to read the data happens before the job starts (i.e. it happens when Spring is instantiating all of the beans).
I'd like to link fetching data from the web service to the job and I'm wondering, from a design perspective, what's the best approach.
So in summary, I'm looking for some suggestions to:
When the job starts, assemble all of the data required for the job (i.e. make the web services call, query our system for matching data, and save data somewhere that the reader can access, currently our loop controller).
Is doing all of the setup work for the reader (the web services call and querying our system), an activity that belongs in the various bean constructors?
Sep 26th, 2012, 12:04 PM
Have you checked out org.springframework.batch.item.file.MultiResourceI temReader?
Maybe extended it to support additional behavior?
Sep 26th, 2012, 12:16 PM
Nope - I will though. Thanks for the tip.
Sep 27th, 2012, 12:47 PM
Have you looked at Spring Integration? You could have a component (endpoint) that polls/listens to the web service, subsequently calls a service activator to augment the data if it natches, and then invokes launches a batch job.
Alternatively, I supposed you could use a series of tasklets to initially obtain and then augment the data.