Nested item processing?

[adrianshum]

Hi all,

I am not sure if this is something Spring Batch can provide:
Currently we can use ItemReader to read items to be processed, and use a Composite Item Writer to include list of Item Writer, in order to have different handling on each of the item read. I am not sure if we can do something nested?

For example, I read list of accounts to be processed, and for each account, I do different kind of processsing, and one of them is, I read all trades performed by that account today, and for each trade I do several processing.

Is such kind of nested item-oriented processing capable in current or near future design of Spring batch?

[dkaminsky]

There are several ways you can handle this. e.g.
- You can hard-code a set of if/else conditions or a switch statement into your item writer's write method.
- You can use a single item writer that dispatches to different classes based on the characteristics of the item being processed.
- You can use a CompositeItemWriter that contains all of your writers and define each writer to do nothing if the item is of the incorrect type.

If these don't provide a solution, can you explain in more detail what you are trying to do?

[adrianshum]

There are several ways you can handle this. e.g.
- You can hard-code a set of if/else conditions or a switch statement into your item writer's write method.
- You can use a single item writer that dispatches to different classes based on the characteristics of the item being processed.
- You can use a CompositeItemWriter that contains all of your writers and define each writer to do nothing if the item is of the incorrect type.

If these don't provide a solution, can you explain in more detail what you are trying to do?

Just come back on my batch related work, sorry for the late reply
Maybe I expand my previous example in a bit more detail to see if this make sense.

For example, I am doing a order system, and in each day-end, I need to do something like this:

Foreach Account a in all active account {
  writeAccountCashBalanceToAuditTable(a);     - action1

  Foreach Order o in a.todayOrders()  {
    a.deductBalance(o.price);                 - action2
    writeOrderInfoInAuditFile(o);             - action3
    arrangeShipment(o);                       - action4
  }

  writeAccountEndingBalanceToAuditTable(a);   - action5
}

in order to mimic the above flow, as per your suggestion, shall I have a item reader that will return something like this:
Account Info (begin)
Order Info for this account
Order Info for this account
:
:
Account Info (end)

and action 1-5 are 5 item writers (chained by a composite writer), which 1 will do handling when the obj instance is Account Info (begin), 2-4 work on Order Info, and 5 work on Account Info (End)?

However, what if, as a normal situation, I want to commit after each account is processed?

[Dave Syer]

However, what if, as a normal situation, I want to commit after each account is processed?

Set commitInterval=1? I'm not sure if that is always a good idea, but you can try different settings and see where you get the highest throughput.

[adrianshum]

Set commitInterval=1? I'm not sure if that is always a good idea, but you can try different settings and see where you get the highest throughput.

In fact I am more concern on data consistency issue instead of throughput.
As I shall need all logics for each account being All-or-nothing. I have case that I cannot let my account being 'half-processed'.

So does it mean I gotta find some way to solve it myself ? it should be alright as I am thinking of making a special writer which in-turn take a reader and writers to do such work. Just wanna check if I am reinventing wheels before taking such consideration

Thanks a lot

[Dave Syer]

I'm not sure what you mean by "solve it yoursefl". Solve what? You are certainly going to have to write your own business logic - the framework can't guess what you want to do. Your use case is quite similar as Doug already pointed out to some we have already seen (and even one of the samples). There's nothing unusual about writing your own ItemWriter/Readers - probably borrowing some of ours as building blocks.

[adrianshum]

Hi Dave Syer,

Your reply seems a bit defensive and over-reacting
(I wish it is just some misunderstanding on your tone)

I raise this thread just because I am not sure if such nested item-oriented processing is catered in Spring Batch Design. Of course I know I need to write my business logic, but just as Spring batch generalized some batch processing pattern (e.g. item oriented processing), I just want to see if such kind of nested item oriented processing is also available in Spring Batch. dkaminsky did suggest a workaround (thanks a lot dkaminsky) but as I explained, such workaround may not always work for such kind of processing pattern. That's why I said I shall build that part myself simply means because such pattern is not provided by Spring Batch directly (what I mean by "solve it by myself"). I am not sure if I have said anything wrong on this...

[Dave Syer]

Sorry for any misunderstanding on my part, and if there is anything in my tone it was not intended (maybe caused by my need to keep it brief because I am typing on a mobile phone). I'm still not sure what you are really looking for. The way you describe your use case makes it look like an Account is the Item. From that it is natural to conclude that you will write some combination of Item* implementations that carry out your business logic. This is perfectly normal usage - no workarounds. Am I missing something? Is there a feature that you think might be provided by the framework and isn't?

[adrianshum]

Sorry for any misunderstanding on my part, and if there is anything in my tone it was not intended (maybe caused by my need to keep it brief because I am typing on a mobile phone). I'm still not sure what you are really looking for. The way you describe your use case makes it look like an Account is the Item. From that it is natural to conclude that you will write some combination of Item* implementations that carry out your business logic. This is perfectly normal usage - no workarounds. Am I missing something? Is there a feature that you think might be provided by the framework and isn't?

It's alright I was only worried that I speak something stupid that cause you guys trouble

As I described before, yup, Account is an item. However, my logic to process each Item (Account) is also some item-oriented logic, which is, e.g., get all orders of that account and process them one by one. Of course, I want the steps for processing each order being modular so that different subsystem can write their part to let me 'chain' them together in the batch job. (As I am doing a financial trading and settlement related system, such logic is quite normal). I just want to see if such nested item oriented processing pattern is available.

Adrian