trouble with the MultiResourceItemReader...

[kmbarlow]

I am trying to get the MultiResourceItemReader working and having some problems applying it...

Here is the configuration for it (the input files sit in an "inputs" directory within the run directory):

<bean id="multiFileItemReader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
                <property name="resources" value="file:#{stepExecutionContext.fileName}/*.dat" />
                <property name="delegate" ref="itemReader" />
        </bean>

        <bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">
                <property name="lineMapper" ref="lineMapper" />
        </bean>

but on execution I get:

2009-02-23 17:45:14,485 [taskExecutor-1] WARN  org.springframework.batch.item.file.MultiResourceItemReader  No resources to read
2009-02-23 17:45:14,509 [taskExecutor-1] ERROR org.springframework.batch.core.step.AbstractStep  Exception while closing step execution resources
org.springframework.batch.item.ItemStreamException: Error while closing item reader
        at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.close(AbstractItemCountingItemStreamItemReader.java:94)
        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy9.close(Unknown Source)
        at org.springframework.batch.item.file.MultiResourceItemReader.close(MultiResourceItemReader.java:130)
        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy6.close(Unknown Source)
        at org.springframework.batch.item.support.CompositeItemStream.close(CompositeItemStream.java:86)
        at org.springframework.batch.core.step.tasklet.TaskletStep.close(TaskletStep.java:331)
        at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:245)
        at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:108)
        at org.springframework.batch.core.partition.support.TaskExecutorPartitionHandler$1.call(TaskExecutorPartitionHandler.java:107)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.NullPointerException
        at org.springframework.batch.item.file.FlatFileItemReader.doClose(FlatFileItemReader.java:214)
        at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.close(AbstractItemCountingItemStreamItemReader.java:91)
        ... 33 more

Also, if I might ask... which is the appropriate TaskExecutor to be using for the configuration proposed in section 7.4 of the docs http://static.springsource.org/sprin...alability.html?

org.springframework.core.task.SimpleAsyncTaskExecu tor

or

org.springframework.scheduling.concurrent.ThreadPo olTaskExecutor

Note: I am using local threads.

Thanks.

Keith

[Dave Syer]

The delegate itemReader doesn't need to be step scoped. Does it make any difference if you change that? I tried the same thing and it worked for me.

SimpleAsyncTaskExecutor is, well, "simple"; the ThreadPool* version has more properties. Look in the Spring user guide and Javadocs for more info.

[kmbarlow]

The problem was just a bad case of the "Mondays"....

[kmbarlow]

I did have a case of the mondays but the problem is not resolved... I am not getting substitution for some reason:

2009-02-24 13:10:32,843 [taskExecutor-1] DEBUG org.springframework.batch.core.scope.util.PlaceholderTargetSource  Replaced [file:#{stepExecutionContext.fileName}/*.psv] with [file:#{stepExecutionContext.fileName}/*.psv]

it seems to work if all files are in the run directory but not in the inputs directory... (i.e. my input variable is not getting substituted).

Also...

How can I validate that I am running correctly? if I put the files in the root directory it runs but I am having a little difficulty identifying the file:thread associations... is there anything I can look for to ensure I did things correctly?

Thanks.

Keith

[Dave Syer]

What do you mean by "file:thread associations"? You can try running in a single thread to make it simple?

BTW I think you might find that #{stepExecutionContext.fileName} has to be #{stepExecutionContext[fileName]} until we get Spring 3.0 and the full EL.

[kmbarlow]

Well... that was one problem... (using the wrong syntax) but that syntax is written that way in the doc in section 7.4... I didn't realize it was not valid yet. The other problem was that I was assuming the stepExecutionContext would inherit the jobParameters by default but that doesn't seem to be the case... when I changed it to jobParameters... it worked.

With regard to the file:thread associations... I am just trying to make sure I did everything correctly... I want to be sure I linked all the components properly... Its obvious that I have 16 partitions but I want to make sure each one is correctly processing a separate input file...

if you look at this log snippet... are does it appear as though 16 threads are running simultaneously? Every log entry appears under the taskExecutor-1 thread which I am assuming is the master thread. Is that normal?

Thanks.

Keith

[Dave Syer]

I fixed the error in the docos, thanks.

Your log isn't complete so I can't really tell what's happening. Is taskexecutor-1 the thread that you launched the Job in (through a JobLauncher)? How did you configure the TaskExecutor for the PartitionHandler?