Multi-line Multi record flat file

[AsalBanoo]

Hello,

I have a txt file which has multi line records and each of this record has its own records.

[Data.1]
"Header","2","2011-07-05 00:00:01",
"Price","2132467",35
"Price","1184067",35
"Price","4981716",25
"Price","4940699",70

[Data.2]
"Header","2","2011-07-03 00:00:11",
"Price","2551467",45
"Price","1184067",25
"Price","4981716",65
"Price","4940699",80
..

I have used "MultiLineProductRecordSeparatorPolicy" to read each [Data]. But I don't know how I can get the "Price" records in this Data. Also it seems that "linesToSkip" property does not work when I use multiLineRecordSeparatorPolicy.

<bean id="flatFileDataReader" class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="linesToSkip" value="31"/>
<property name="recordSeparatorPolicy" ref="multiLineRecordSeparatorPolicy"/>
<property name="lineMapper" ref="priceLineMapper"/>
</bean>

Thanks,
Asal

[michael.lange]

Multi Line Records

Did you try multi line records in general? (see multi line records en detail also)

like so

<bean id="orderFileLineMapper"
      class="org.spr...PatternMatchingCompositeLineMapper">
    <property name="tokenizers">
        <map>
            <entry key="Data*" value-ref="dataTokenizer" />
            <entry key="Header*" value-ref="headerTokenizer" />
            <entry key="Price*" value-ref="priceTokenizer" />
        </map>
    </property>
    <property name="fieldSetMappers">
        <map>
            <entry key="Data*" value-ref="dataFieldSetMapper" />
            <entry key="Header*" value-ref="headerFieldSetMapper" />
            <entry key="Price*" value-ref="priceFieldSetMapper" />
        </map>
    </property>
</bean>

Multi Line Records Examples

If you need to get one unified line, take a look at the Spring Batch Samples, e.g. the multi-line example(github source)

Possible Problem with your input file

If you use a DelimitedLineTokenizer i am almost sure the following lines will be count as one, due to the

,

at the end of the "Header" line.

"Header","2","2011-07-05 00:00:01",
"Price","2132467",35

[AsalBanoo]

Hey,

Thanks for your reply. Yes I have seen the PatternMatchingCompositeLineMapper. But that is used when you have different types of records. But in my case I have one multi-line record (Data) and I want to relate the Price and Header to one Data Object. If I read each Price as one record, then how can I know that each Price record relates to what Data record?

Cheers,
Asal

[michael.lange]

for your usecase i pointed to the example

Multi Line Records Examples

If you need to get one unified line, take a look at the Spring Batch Samples, e.g. the multi-line example(github source)

a AggregateItemReader is used there, it should be no problem to adjust the example to one, where you keep the data-id for each price record

one other solution could be:

• go with the "read each line" example
• in processor or writer decide - if header-line, keep data-id in StepExecutionContext, do not persist (do not call DAO layer, wrapped jdbcWriter, whatever)
• if price-line, extract data-id from StepExecutionContext, persist price
• next header-line - just overwrite data-id in context
• and so on

[AsalBanoo]

Thanks for your reply. I got just another question. How can I access StepExecutionContext in the ItemWriter? I read in the documents that I can have a StepListener for each Step. But I couldn't find out how I can access the StepExecutionContext in the writer.

Thanks,
Asal

[michael.lange]

to access the StepExecutionContext you can get the writer to be a listener too, just create a custom writer

• which implements the proper StepExecutionListenerinterface or annotations
• keep a reference from the StepExecutionContext from the

  void beforeStep(StepExecution stepExecution) { this.context=stepExecution.getExecutionContext()  }

  method, see StepExecution JavaDoc
• register it to the step

example job.xml

    <job id="skipJob" xmlns="http://www.springframework.org/schema/batch">
        <step id="skipJobStep">
            <tasklet>
                <chunk 
                    reader="itemReader" 
                    writer="itemWriter" 
                    commit-interval="5">
                </chunk>
            </tasklet>
            <listeners>
                <listener ref="itemWriter" />
            </listeners>            
        </step>
    </job>

its important on which level you use the listeners element, the spring batch doc example is for chunklisteners

[AsalBanoo]

Thanks for your reply. I got it to work finally!

Asal