Advice on Mutliple Batch Job Control

[chanley]

Hi,

Is it enough to rely on the application server thread pool to manage the execution of multiple concurrent jobs. In particular I mean jobs which are utilizing partitioning and could use anything from 5 to 20 threads for any given step. The concern is that if too many of these jobs are invoked in parallel then requests could be lost/aborted from the thread pool queue if the number of requests was large enough. They could be lost if the thread pool does not free up a thread within the timeout, or if the application server was brought down (and therefore lost due to non persistence of the thread pool)

I guess I am looking for some sort of reliable job queuing mechanism. JMS is one option for controlling the amount of work which supports persistence, but I would rather not introduce that at this time. Can people shed some light on what techniques they are using to queue work? Is there anything in Spring Batch now, or in future releases around job queuing functionality.

Some background: The application in question is a multi-tenant web application which has a number of spring batch jobs. Jobs can be invoked on demand via the UI or via a Quartz scheduler. Many jobs may start at the same time for each tenant/user. The majority of the jobs are importing/exporting files and they utilize concurrent processing (partitioning) for steps in order to optimize throughput. The application uses the JBossWorkManagerTaskExecutor for thread pool management delegation to JBoss.

Thanks

[raysuliteanu]

I don't know if this will help in your specific situation but Spring Batch has a throttling thread executor implementation: org.springframework.batch.admin.util.ThrottledTask Executor

[chanley]

Hi. Thanks for your reply.

From my reading of the ThrottledTaskExecutor implementation, the use of Semaphore is blocking threads until one becomes available. It might be a short term fix for limiting number of jobs but I would still have the problem of job launching threads being blocked and potentially being lost if the application server was halted. If the job launching thread was lost, I think there would be no record of it and no way to retry.

I suppose a database or JMS backed solution is what I need. Has anybody implemented something similar?

Thanks