Apr 16th, 2008, 10:00 PM
Spring Batch - EJB - ESB
I am trying to find out which of the technologies if used in our project will yield highest performance.
If I need to read millions of records and process them using a business logic then which one of the following should yield high performance:
=> Spring Batch : with its parallel job processing
=> EJB : I can read primary keys of the million records and send that as a key to stateless bean
=> ESB : message oriented enterprise bus, like MULE, read chunk of data and send them as asynchronous messages to process them
Also if anyone had used the combination of these and had good/bad experiences please share it.
Apr 18th, 2008, 01:32 PM
Spring Batch is a lightweight framework, which means it is not incompatible with EJB or ESB. I can think of a few ways to use those tools together with Spring Batch, and some of those might make a performance impact on some business problems. The answer probably is that is depends on the business problem. In particular, if you are reading your input records from a file, then it basically has to be single-threaded, and that might create a bottleneck in any scheme that you devise. On the other hand reading text from a file is fast, and if the business processing is reasonably complex or I/O intensive then it won't matter that the input stage is single-threaded because you can scale the workers. Then you have to implement a communication protocol between the dispatcher and the workers, and maybe an ESB could provide a transport for that communication. That's the value of an ESB in this scenario, as far as I can see, but be careful - the transport has to be durable and transactional, otherwise you will lose data in the case of failure. The conclusion is that in a typical ESB, the transport is going to be JMS, so you could just as well use vanilla JMS and forget the ESB. I guess that's where EJB (MDB) would probably come in. But you still need to drive the dispatcher and workers and collect and aggregate all the success / failure counts etc. That's what Spring Batch is going to do, working in conjunction with lower level enterprise features.
On the other hand, if all you want is to read a few million lines from a file and push them into a database, you can probably do that without all the hassle of setting up the ESB/ApplicationContainer, just with a single-threaded vanilla Spring Batch process. If you are reading from a database and writing to a database, then a multi-threaded approach would work (e.g. with process indicator on the input table). On a multi-core machine against a full-scale database you ought to get a hundred or more transactions per second per thread.