Error using two FlatFileItemReaders in the same batch job - Simple batch template

Hi,

I have a batch application that is built on top of the simple batch template project in STS 2.9.0. I am using Spring 3.1 in my application.

In my application context I have defined two seperate FlatFileItemReader beans with different IDs as per the XML below.

The problem is that when I run the job, I get this error:

org.springframework.batch.item.file.FlatFileParseE xception: Parsing error at line: 2 in resource=[file [c:\Loader\sets\users.csv]], input=[111111,111111,Manager,Active]
at org.springframework.batch.item.file.FlatFileItemRe ader.doRead(FlatFileItemReader.java:182)
at org.springframework.batch.item.support.AbstractIte mCountingItemStreamItemReader.read(AbstractItemCou ntingItemStreamItemReader.java:85)
at org.springframework.batch.item.file.MultiResourceI temReader.readNextItem(MultiResourceItemReader.jav a:133)
at org.springframework.batch.item.file.MultiResourceI temReader.read(MultiResourceItemReader.java:108)
at org.springframework.batch.core.step.item.SimpleChu nkProvider.doRead(SimpleChunkProvider.java:90)
at org.springframework.batch.core.step.item.SimpleChu nkProvider.read(SimpleChunkProvider.java:148)
........
.........
org.springframework.batch.item.file.transform.Inco rrectTokenCountException: Incorrect number of tokens found in record: expected 3 actual 4
at org.springframework.batch.item.file.transform.Abst ractLineTokenizer.tokenize(AbstractLineTokenizer.j ava:123)
at org.springframework.batch.item.file.mapping.Defaul tLineMapper.mapLine(DefaultLineMapper.java:42)
at org.springframework.batch.item.file.FlatFileItemRe ader.doRead(FlatFileItemReader.java:179)
... 59 more

So, the tokenizer codeReader bean works as expected and maps the three fields defined in it's tokenizer "name" property but even though I have defined two different FlatFileItemReaders, it looks like the same DelimitedLineTokenizer is then being used for the userReader. To prove this, I changed the "names" property in the codeReader tokenizer so that it only had two values (value="REF-CODE-NAME,REF-CODE-FULL-DESCRIPTION"), and the stack trace changes to read

org.springframework.batch.item.file.transform.Inco rrectTokenCountException: Incorrect number of tokens found in record: expected 2 actual 4


So my question is this. Is this the expected behaviour in that both FlatFileItemReaders will share one tokenizer, even if these tokenizers are definied as seperate beans and if so how can I get around it? I tried to create a subclass of FlatFileItemReader and instantiate a new tokenizer within it's constructor but I still get the same problem.

Any help would be much appreciated. Also this is my first post so go easy on me if I have ommited anything ;-)


APP-CONTEXT Extract
BEAN 1
<bean id="codeReader" class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean
class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<property name="lineTokenizer">
<bean class="org.springframework.batch.item.file.transfo rm.DelimitedLineTokenizer">
<property name="names" value="REF-CODE-NAME,REF-CODE-FULL-DESCRIPTION,REF-CODE-HIGH-VALUE" />
</bean>
</property>
<property name="fieldSetMapper" ref="codeFieldSetMapper"></property>
</bean>
</property>
</bean>

<bean id="codeFieldSetMapper" class="mypackage.CodeFieldSetMapper">
<property name="multiItemReader" ref="codesetFileReader"/>
</bean>


BEAN 2
<bean id="userReader" class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="resource" value="${app.guiuserfile}" />
<property name="linesToSkip" value="1" />
<property name="lineMapper" ref="userLineMapper"/>
</bean>

<bean id="userLineMapper" class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<property name="lineTokenizer" ref="userLineTokenizer"/>
<property name="fieldSetMapper" ref="userFieldSetMapper"/>
</bean>

<bean id="userLineTokenizer" class="org.springframework.batch.item.file.transfo rm.DelimitedLineTokenizer">
<property name="names" value="BADGEID,USERNAME,ROLE,STATUS" />
</bean>

<bean id="userFieldSetMapper" class="mypackage.UserFieldSetMapper"/>



Module-context.xml
<batch:job id="job1" >
<batch:step id="step1" next="step2" >
<batch:tasklet transaction-manager="transactionManager" start-limit="100" >
<batch:chunk reader="codeReader" processor="codeProcessor" writer="writer" commit-interval="50" />
</batch:tasklet>
</batch:step>

<batch:step id="step2" >
<batch:tasklet transaction-manager="transactionManager" start-limit="100" >
<batch:chunk reader="userReader" processor="userItemProcessor" writer="xmlItemWriter" commit-interval="50" />
</batch:tasklet>
</batch:step>

</batch:job>

First, thank you for providing as much detail as you did. Any information you can provide is helpful!

To answer your question directly, no. You can have as many FlatFileItemReaders you want and each can have their own tokenizer. Looking at your configuration, there are a few things missing that I'm assuming you have in your environment (configuring the resource to be read on the codeReader for example). I'm assuming that from your description, the first step completes successfully prior to this error?

Also, would it be possible to post the entire app context and module context files?

Hi,

Firstly, thanks for getting back to me so promptly! Full app context is below. I'm using a MultiResourceItemReader reader for the codeReader.

When I run the test JobConfigurationTests that ships with the template, I get the error as per the stack trace I sent. Form the log it looks like step 1 is completing with a status of "Failed"

Thanks again
David


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schem...-beans-3.1.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schem...ontext-3.1.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.1.xsd">

<context:property-placeholder location="classpath:/app.properties" />

<context:component-scan base-package="mypackage"/>

<!-- CODESET BEANS -->

<bean id="codeReader" class="org.springframework.batch.item.file.FlatFil eItemReader"> <property name="linesToSkip" value="1" />
<property name="lineMapper">
<bean id="codesetLineMapper" class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<property name="lineTokenizer">
<bean id="codesetLineTokenizer" class="org.springframework.batch.item.file.transfo rm.DelimitedLineTokenizer">
<property name="names" value="REF-CODE-NAME,REF-CODE-FULL-DESCRIPTION,REF-CODE-HIGH-VALUE" />
</bean>
</property>
<property name="fieldSetMapper" ref="codesetFieldSetMapper"></property>
</bean>
</property>
</bean>


<bean id="codesetFieldSetMapper" class="mypackage.CodesetFieldSetMapper">
<property name="multiItemReader" ref="codeFileReader"/>
</bean>

<bean id="codeFileReader" class="org.springframework.batch.item.file.MultiRe sourceItemReader">
<property name="resources" value="${app.codesetfiles}" />
<property name="delegate" ref="codeReader"/>
</bean>


<bean id="codesetProcessor" class="mypackage.CodesetProcessor">
<property name="marshaller" ref="jaxbMarshaller"/>
</bean>

<bean id="fileWriter" class="mypackage.CodesetFileWriter">
<property name="resourceFactory" ref="resourceFactory"/>
</bean>

<bean id="writer" class="mypackage.CodesetWriter" init-method="init">
<constructor-arg name="dao" ref="restDao"/>
<property name="resourceFactory" ref="resourceFactory"/>
</bean>

<!-- GUIUSER BEANS -->

<bean id="userReader" class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="resource" value="${app.guiuserfile}" />
<property name="linesToSkip" value="1" />
<property name="lineMapper" ref="userLineMapper"/>
</bean>

<bean id="userLineMapper" class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<property name="lineTokenizer" ref="userLineTokenizer"/>
<property name="fieldSetMapper" ref="userFieldSetMapper"/>
</bean>

<bean id="userLineTokenizer" class="org.springframework.batch.item.file.transfo rm.DelimitedLineTokenizer">
<property name="names" value="BADGEID,USERNAME,ROLE,STATUS" />
</bean>

<bean id="userFieldSetMapper" class="mypackage.UserFieldSetMapper"/>


<bean id="fieldSetMapper" class="mypackage.UserFieldSetMapper"/>

<bean id="userItemProcessor" class="mypackage.UserItemProcessor">
<property name="marshaller" ref="jaxbMarshaller"/>
</bean>


<bean id="xmlItemWriter" class="mypackage.XmlItemWriter">
<constructor-arg name="dao" ref="restDao"/>
<property name="resourceFactory" ref="resourceFactory"/>
</bean>



<bean id="outputFile" class="org.springframework.core.io.FileSystemResou rce" scope="step">
<constructor-arg value="c:/delete/output.txt"/>
</bean>


<!-- HTTP CLIENT BEANS -->

<bean id="authenticatedRestClient" class="mypackage.AuthenticatedRestClient">
<constructor-arg name="loginUrl" value="${app.loginurl}"/>
<constructor-arg name="uname" value="${app.username}"/>
<constructor-arg name="passwd" value="${app.password}"/>
<property name="cookieWrapper" ref="cookieWrapper"/>
</bean>



<bean id="restDao" class="mypackage.RESTDao">
<constructor-arg name="restClient" ref="authenticatedRestClient"/>
<property name="server" value="${app.server}"/>
<property name="port" value="${app.port}"/>
</bean>


<!-- UTILITY BEANS -->

<bean id="resourceFactory" class="mypackage.ResourceFactory">

<property name="resources">
<util:map>
<entry key="MANUALDIR" value="${app.MANUALDIR}"/>
<entry key="EXTERNALSITES" value="${app.EXTERNALSITES}"/>
<entry key="SOURCEFILE" value="${app.SOURCEFILE}"/>
<entry key="WATCHLIST" value="${app.WATCHLIST}"/>
<entry key="PRICEFEED" value="${app.PRICEFEED}"/>
<entry key="PRICEFEEDREQUESTOR" value="${app.PRICEFEEDREQUESTOR}"/>
<entry key="PRICEMANUALLY" value="${app.PRICEMANUALLY}"/>
<entry key="REASON" value="${app.REASON}"/>
<entry key="GUIUSER" value="${app.GUIUSER}"/>
</util:map>
</property>
</bean>


<bean id="cookieWrapper" class="mypackage.CookieWrapper">
<property name="domain" value="${app.cookiedomain}"/>
<property name="path" value="/pl4edm"/>
</bean>

<bean id="jaxbMarshaller" class="org.springframework.oxm.jaxb.Jaxb2Marshalle r">
<property name="classesToBeBound">
<list>
<value>mypackage.Codeset</value>
<value>mypackage.GUIUser</value>
</list>
</property>
<property name="marshallerProperties">
<map>
<entry>
<key>
<util:constant static-field="javax.xml.bind.Marshaller.JAXB_FRAGMENT"/>
</key>
<value type="java.lang.Boolean">true</value>
</entry>
</map>
</property>
</bean>
</beans>



MODULE context

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:batch="http://www.springframework.org/schema/batch"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schem...-beans-3.1.xsd
http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd">

<description>Step 1 loads codesets and Step 2 loads users.</description>

<batch:job id="job1" >
<batch:step id="step1" next="step2" >
<batch:tasklet transaction-manager="transactionManager" start-limit="100" >
<batch:chunk reader="codeFileReader" processor="codesetProcessor" writer="writer" commit-interval="50" />
</batch:tasklet>
</batch:step>

<batch:step id="step2" >
<batch:tasklet transaction-manager="transactionManager" start-limit="100" >
<batch:chunk reader="userReader" processor="userItemProcessor" writer="fileWriter" commit-interval="50" />
</batch:tasklet>
</batch:step>

</batch:job>


</beans>

Hi,

Okay, so this has now been resolved and I am happy to report that it is my fault entirely. So firstly, apologies for wasting your time.

The problem was that I was using a MultiResourceItemReader for the codeReader and this was picking up a pattern of *.csv from a nominated directory. Then the userReader was picking up a file called users.csv which resided in the same directory. So, the codeReader was also picking up users.csv (from *.csv) and attempting to parse it with it's own tokenizer and this is why I was getting the error re. expected 3, found 4....

Your question about step 1 failing prompted me to check the logs again and it finally clicked with me what was happening. users.csv has now been moved to it's own directory and everything is working as expected.

Thanks for all your help with this. it was only with your feedback that I started to look at step1 instead of concentrating on step2.

Humbled and feeling slightly stupid
David

Glad you figured it out and thanks for posting the follow up!