Any way to publish events?

[adrianshum]

Dear all,

I am going to write a batch service that users send job requests, and my server will execute corresponding jobs asynchronously, and actively pushing any change in the job's status, including completion of individual step, completion of whole job, or even let steps publish their own message (reporting statistic or progress).

Such events may be sent to an JMS Topic to let interested parties to subscribe to.

Apart from notification to client, I may even use this as a distributed parallelism, where a mother job's task send out "child" job requests to other batch server, and wait until all child job to complete.

To achieve these, I need to subscribe to events in Job Instance (or Job Execution?), is there any way in Spring Batch that can let me do so?

To be more specific, is there any way I can "subcribe" to job instance, and being notified of:
1) Job start and completion
2) Start and completion of individual step
3) custom events published in tasklet/processor

Thanks a lot

[lucasward]

We actually had something like that with the Job originally, but removed it because we didn't see the value of it over just using Aop to surround Job.execute and/or Step.execute.

In the case of the step, you can use a RepeatListener, but I personally would prefer using AOP.

[adrianshum]

We actually had something like that with the Job originally, but removed it because we didn't see the value of it over just using Aop to surround Job.execute and/or Step.execute.

In the case of the step, you can use a RepeatListener, but I personally would prefer using AOP.

Hi Lucasward,

AOP does work for Job that part, thanks a lot

However, is there any way I can have custom statistic data and publish it actively? I am thinking of, in case I cannot use the item-reader-writer tasklet template, but I still wanna publish my custom progress or message, what can I do?

I am thinking of making a special JobParameter which include a observer for the execution; a JobExecution and StepExecution wrapper which provide method to set custom messages and method to notify observers; a JobRepository wrapper wrap the JobExecution and StepExeuction created by the original one, and plug the observer to the JobExecution wrapper. Our own tasklet can get the step execution (by StepContextAware) and set custom message on the step exeuction, and notify observers publish event to observers.

It seems to be a bit clumsy... is there any cleverer way to achieve what I want?

Thanks a lot.

[lucasward]

Couldn't you use the spring built in MessageListener?

http://static.springframework.org/sp...onality-events

All you would really need is a handle to the application context to toss events into.

[adrianshum]

That may be a choice... I shall try to see if I can make use of it.

My main concern is, as I will make use of one JobLauncher to serve multiple jobs concurrently, each incoming request shall only interest on its own event. However they can only distinguish which events are interested after they acquire the returned JobExecution. In order not to miss any interested event, different job may need to synchronized to wait for the caller received the JobExecution returned correctly and do its own registering. This may cause unnecessary contention.

Anyway, I shall just give it a try to see if the contention is significant

btw, I tried to look at StepExecution and I found Execution Attributes. I wonder if I can make use of that to store my customed step progress messages? However I tried to set attributes in it but seems it won't persists in DB. What should I do to tell Spring Batch to persist those attributes?

Thanks a lot

[lucasward]

I'm not sure I understand why you would have contention on events? They're generally 'fire and forget' by definition.

Regarding ExecutionAttributes, this is the correct place to store custom statistics about your job run. I'm not sure why you're having issues with it being persisted though, how are you accessing it? (You should be getting at it via the StepContext)

[adrianshum]

I'm not sure I understand why you would have contention on events? They're generally 'fire and forget' by definition.

Regarding ExecutionAttributes, this is the correct place to store custom statistics about your job run. I'm not sure why you're having issues with it being persisted though, how are you accessing it? (You should be getting at it via the StepContext)

For the first question, because I don't want the listener to miss any event on the job they submit. If I simply do in fire-and-forget way, before the invoker get the JobExecution from JobLauncher (so that invoker can know what event he is interested on by filtering of JobExecution ID), there may be plenty of events fired and missed. That's why I need to synchronize on that part, so that before I launch the job and register my interested events, no event is published. So if all job execution shares same 'channel' of publishing (via Application Event), that may cause contention.

For the second question, I just make my tasklet StepContextAware, and do something like:
stepContext.getStepExecution().getExecutionAttribu tes().put("foo", "bar");

However, after the job completed, it is not updated to DB. Shall I somehow ask StepExecution to persist the execution attribute once I updated it?

[lucasward]

yes, there's a property on the step 'saveExecutionAttributes' which should be set to true.

[adrianshum]

yes, there's a property on the step 'saveExecutionAttributes' which should be set to true.

it is weird that it still failed... trying to trace into the code to see what I have done wrong but not found the answer yet.

Here is what I am doing:

    <bean id="job" parent="simpleJob">
        <property name="steps">
            <list>
                <bean id="step1" parent="simpleStep">
                    <property name="tasklet">
                        <bean id="tradeTasklet"
                            class="com.foo.TradeMatchTasklet"
                            scope="step">
                            <aop:scoped-proxy />
                        </bean>
                    </property>
                    <property name="commitInterval" value="2" />
                    <property name="saveExecutionAttributes" value="true" />
                </bean>
            </list>
        </property>
    </bean>

and my tasklet:

public class TradeMatchTasklet implements Tasklet, StepContextAware {
    //... some other boring getters setters etc deleted
    public ExitStatus execute() throws Exception {
        stepContext.setAttribute("test" + status, "hello");
        if ((++(this.status)) > 5) {
          return ExitStatus.FINISHED;
        }
        return ExitStatus.CONTINUABLE;
    }
}

[lucasward]

You've found a bit of a bug. Try calling stepContext.getStepExecution.getExecutionAttribute s.put().

I know it's long, but I'll log the issue and we'll resolve it by either making the attributes easier to get at or making the setAttributes on step context the primary way to access it.