Chains steps together

[icephoenix]

I am just starting to use Spring Batch, so I apologize for possibly stupid question, but how do you chain(or rather pipe) job steps together.
Essentially I'd like to have something like:

step 1 reads data from db and puts it into some sort of a pipe
step 2 reads data from that pipe transforms it and writes it into xml file

Thank you!

[icephoenix]

so is it just not possible? or no one knows? or is it too stupid of a question? should I rtfm?

[Dave Syer]

Some people were quite busy last week when you posted. Your use case is also a little vague. Are you saying that sequential step execution doesn't work for your use case? Which is what exactly? Something about your decription of a "pipe" makes me think of some Spring Integration implementations that we have in the spring-batch-integration project in SVN. Anything there ring a bell (https://src.springframework.org/svn/...-integration)? There are some use cases written up in the site docs there.

[icephoenix]

I am sorry if I appeared mean or something, I didn't mean too.

Spring integration looks interesting, and I may certainly find it useful, though it doesn't specifically helps in this use case.

Let me try to describe it better.

I working on kind of ETL system. It has to read data from one database, do some lookup for auxiliiary data, transform it, map onto the other database, and persist transformed data in the other database. So my idea of process for this one is something like this:

Job:
step1: readSourceDataStep (reads data from the source data base and makes it available for the other steps to access)
step2: lookupAuxDataStep (traversing through the data read in step1, and doing various lookup for additional data)
step3: persistMappingStep (persisting relationship between source data and transformed data)
step4: persistTransformedDataStep (persisting transformed data )

So... I am at a step 1 trying to figure out what to use to save the list read from the database so it's accessible during other steps?..

Thank you for your help!

[Edouard]

if I can make a point, I think that it is not following the Spring Batch philosophy.

A step transforms a item "one by one" (not really but you can think like this).

So, you make one step with one ItemReader, several ItemProcessor and one CompositeItemWriter with several ItemWriter.

Is this solution solve your problem ?

I will have the same problem next week. If you have another ideas, I will be interested.

[icephoenix]

hm... makes sense! I guess I missed the point about what a step is.
Edouard, your solution sounds like it will work out! I will just have to partition my job a bit differently making each step deal with a specific type of data.

I guess my confusion is soming from idea of having transformation graph(like in other ETL frameworkds) where each node does very specific part of work. So I confused nodes with steps...