Dynamic multi-threading

[luccotta]

Hi, I'm new to spring-batch and I don't the best way to do this.

My batch application generates some files to be downloaded later by the client, and I do all the processing in a class that I implemented org.springframework.batch.core.step.tasklet.Taskle t.

I need to make my application multi-thread, but the amount of thread to run each job instance will be defined by the client. So I'll have a pool with, say, 10 threads, and the client in his request will say how many threads he wants for his file generation.

What is the best way to accomplish this? I've already read the Chapter 7 form the spring-batch reference, but I still have no idea.

Thanks!!

[BjRi]

Hi,

I still hope to find a better solution, but it works - at least for me :-) I spent some time trying to use late binding for the property "concurrencyLimit" of the bean "taskExecutor", but I finally gave up.

<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTa skExecutor"/>

<batch:job id="myJob">
<batch:step id="myStep">
<batch:tasklet task-executor="taskExecutor" throttle-limit="10">
<batch:chunk reader="itemReader" writer="itemWriter" commit-interval="1"/>
</batch:tasklet>
</batch:step>
</batch:job>

I start the job like that:
final int concurrencyLimit = 7; // replace this value with whatever you like

ApplicationContext ctx = new ClassPathXmlApplicationContext("spring/test-config.xml");
JobLauncher launcher = (JobLauncher) ctx.getBean("jobLauncher");
SimpleAsyncTaskExecutor taskExecutor = (SimpleAsyncTaskExecutor) ctx.getBean("taskExecutor");
taskExecutor.setConcurrencyLimit(concurrencyLimit) ;
JobParameters jobParams = new JobParametersBuilder().toJobParameters();
Job job = (Job) ctx.getBean("myJob", Job.class);
launcher.run(job, jobParams);

[luccotta]

Hi,

I still hope to find a better solution, but it works - at least for me :-) I spent some time trying to use late binding for the property "concurrencyLimit" of the bean "taskExecutor", but I finally gave up.

<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTa skExecutor"/>
...

Thank you for the detailed answer!!
That already helped me a lot.
I see here that the setConcurrencyLimit will "set the maximum number of parallel accesses allowed".
Now, what if I want, in your example, to use exactly 7 threads, instead of using at most 7 threads? I would have to wait other jobs finish their execution... is it possible?

Thanks!

[BjRi]

Just to get it right:
You have a pool of 10 threads and there is currently one job running with 7 threads. If another job should be started with 4 threads, then the job with 7 threads has to finish first, so that there are four threads available for the new job?

That would be a tough task :-)

[luccotta]

Just to get it right:
You have a pool of 10 threads and there is currently one job running with 7 threads. If another job should be started with 4 threads, then the job with 7 threads has to finish first, so that there are four threads available for the new job?

That would be a tough task :-)

Exactly!
Well, if anyone has any idea it is most welcome!

Thanks!

[bmaduskar]

multithreading can come handy here, for java , each spring batch job is a process , if you use thread pool executor , which is with JDK 1.6 or further you can , execute 10 or more jobs in join and they will wait for resources.