Retry without blocking threads?

[ilmyggo]

We're using stateless retry with asynchronous task execution, and each failed job is blocking the corresponding thread while performing the retries. Occasionally a lot of jobs will fail (typically when some external system is unavailable), leading to thread starvation.

Any suggestions on how to implement Retry without blocking threads? We're considering JMS or Quartz; however, we prefer staying within Spring Batch if possible.

[Dave Syer]

What are the visible symptoms of starvation in this case (if an external resource is down, then presumably there is no way to make progress anyway)? How many threads are blocking? Is it one per job (which is ultimately unavoidable), or many (which maybe we can address somehow)?

[ilmyggo]

Thread pool monitoring reveals the starvation. We cannot afford blocking one thread per job for our retry requirements (can last hours, even days).

Our solution was to implement a scheduled timer to perform retry logic at a fixed interval (e.g. every hour), using the JobOberator to find failed jobs that should be retried.

Thanks for your reply, sorry for the delayed feedback.