ParallelJob/Multithreading - Sample - Clarification

[amirvenkat]

Hi,

I went through the paralleljob sample and following is my understanding.
1. The "loading" step is multithreaded using "taskExecutor".
2. The "stagingReader", "stagingProcessor" & "tradeWriter" will be executed
sequentially in a given thread.
3. The primary keys are retrieved first using synchronization and then
read, process, write is performed.

The question that i have is:
1. Can reading the primary keys itself be multithreaded? I am not sure,
whether my question will be meaningful, but it just a thought in my
mind.
2. If there are million records to be processed, will there be any performance
hit while reading the primary keys in one single shot? Or can this be
read in chunks and then be multithreaded?
3. For flat file input, is multithreading possible?

Thanks.

--venkat

[amirvenkat]

Anyone can help on this thread?

--venkat

[Dave Syer]

Try to be patient. The best way to get an answer is actually to leave your post with 0 replies.

1. The "loading" step is multithreaded using "taskExecutor".

Yes.

2. The "stagingReader", "stagingProcessor" & "tradeWriter" will be executed
sequentially in a given thread.

Yes.

3. The primary keys are retrieved first using synchronization and then
read, process, write is performed.

Yes. There is also a synchronization in the read() method of the reader, so that multiuple threads don't get the same item.

1. Can reading the primary keys itself be multithreaded? I am not sure,
whether my question will be meaningful, but it just a thought in my
mind.

Anything is possible. You could load the keys in a separate thread and allow the processing to start before they are complete, for instance. I doubt you could process them much faster, but if you try it and it helps then be sure to let us know.

2. If there are million records to be processed, will there be any performance
hit while reading the primary keys in one single shot? Or can this be
read in chunks and then be multithreaded?

You could strategise the reading for sure. Our experience is that it isn't worth the effort until you get to the point where you are prepared to put some more effort into it, and then you probably want to use a PartitionHandler anyway (then the StagingItemReader is redundant). A million records is pretty small beer.

3. For flat file input, is multithreading possible?

In principle. In practice we don't support it (yet) because it is very difficult to store the restart data if items are not processed sequentially.

[amirvenkat]

Thanks a lot for your reply Dave.

And kindly apologize for replying to my own thread. I was expecting your
reply, but since there was a delay, i replied to the post. Anyway apologize.

Yes, sure i will implement this and will let you all know whether multithreading
in reading primary keys works out or not.

Thanks
venkat