OutOfMemoryError - File read & File write

[hailspring]

Hello All,

I am using a simple job that reads the file and writes the read line to another file (just doing a volume testing with spring batch). The input file is more than 7GB.
I have file reader (FlatFileItemReader) and a file writer (FlatFileItemWriter). Is there a limitation on the number of records that can be processed at a single time (I suppose not).

Here is the error, if someone can hint me know what might be wrong, I can check it out:

Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.
java:99)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518
)
        at java.lang.StringBuffer.append(StringBuffer.java:307)
        at java.io.BufferedReader.readLine(BufferedReader.java:345)
        at java.io.BufferedReader.readLine(BufferedReader.java:362)
        at org.springframework.batch.item.file.separator.ResourceLineReader$Stat
e.readLine(ResourceLineReader.java:225)
        at org.springframework.batch.item.file.separator.ResourceLineReader.read
Line(ResourceLineReader.java:134)
        at org.springframework.batch.item.file.separator.ResourceLineReader.read
(ResourceLineReader.java:120)
        at org.springframework.batch.item.file.FlatFileItemReader.readLine(FlatF
ileItemReader.java:103)
        at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFil
eItemReader.java:266)
        at org.springframework.batch.item.support.AbstractBufferedItemReaderItem
Stream.read(AbstractBufferedItemReaderItemStream.java:92)
        at org.springframework.batch.item.support.DelegatingItemReader.read(Dele
gatingItemReader.java:61)
        at org.springframework.batch.core.step.item.BatchListenerFactoryHelper$1
.read(BatchListenerFactoryHelper.java:67)
        at org.springframework.batch.core.step.item.SimpleItemHandler.doRead(Sim
pleItemHandler.java:88)
        at org.springframework.batch.core.step.item.SimpleItemHandler.read(Simpl
eItemHandler.java:80)
        at org.springframework.batch.core.step.item.SimpleItemHandler.handle(Sim
pleItemHandler.java:66)
        at org.springframework.batch.core.step.item.ItemOrientedStep$2.doInItera
tion(ItemOrientedStep.java:390)
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult
(RepeatTemplate.java:346)
        at org.springframework.batch.repeat.support.RepeatTemplate.executeIntern
al(RepeatTemplate.java:212)
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(Repea
tTemplate.java:143)
        at org.springframework.batch.core.step.item.ItemOrientedStep.processChun
k(ItemOrientedStep.java:383)
        at org.springframework.batch.core.step.item.ItemOrientedStep$1.doInItera
tion(ItemOrientedStep.java:251)
        at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult
(RepeatTemplate.java:346)
        at org.springframework.batch.repeat.support.RepeatTemplate.executeIntern
al(RepeatTemplate.java:212)
        at org.springframework.batch.repeat.support.RepeatTemplate.iterate(Repea
tTemplate.java:143)
        at org.springframework.batch.core.step.item.ItemOrientedStep.doExecute(I
temOrientedStep.java:231)
        at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep
.java:172)
        at org.springframework.batch.core.job.SimpleJob.execute(SimpleJob.java:1
00)
        at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run
(SimpleJobLauncher.java:86)
        at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecut
or.java:49)

I have a java object with one bytearray property. In my FieldSet Mapper I read first 100 bytes of the input line into this byte array and then in my writer (with PassThrough fieldset creator), I do a new String(byteArray) and write it out.

I also tried with that java object property as String. I am using Spring Batch 112.

Please let me know your thoughts. Also let me know whether there is a way I can speed this.

Thanks!!

[Dave Syer]

Spring Batch doesn't store much intermediate state, unless you are doing something crazy with ExecutionContext. It's practically impossible to diagnose OOM from a stack trace, so maybe you can share some more details?

[ameerb]

Try to set JVM memory size like java -Xms500m.

[Jeff Zhang]

I also encounter this problem recently, The input file is too large(maybe 2 million lines), and the OutOfMemory Error happens, Is any solution or findings about this problem?

[Dave Syer]

I don't have any problems with large files (as I said before Spring Batch is intentionally not storing very much intermediate data in memory). You need to share some more details of what you are doing.

[hailspring]

Here is what I encountered, even I was under the same impression that Spring batch is failing for huge files but only later to find out that somewhere in the middle of my file there was no "line feeder" or "next line character", and when I debugged through the code I found that Spring batch makes a call "readLine" internally to get the next line and the parse it.

Since the readLine relies on next line character, which was missing somewhere in the middle, it was trying to load the whole line (almost rest of the file) into memory as a single line which eventually caused it to fail and throw out of memory exception.

Again this was the experience I had and spend about couple of days to figure out the issue, hope you dont waste that much time. Additionally, this is what I encountered and I can not guarantee that you are also having the same issue, but worth checking it.!!! :-)

[Jeff Zhang]

Actually I found my problem is because of the bad design.I read the whole data into a list, this design result in the OutOfMemory error. Hence now I control the read line limit in one time read, the problem is never happened again. Thanks for you guys help.

[sansun2111]

Hi all ,
Even i am in the same bandwagon as you ppl have been in the past ,
I am not able to understand why the read method is throwing OutOfMemory Exception once the Reader reads the complete file .

MyReader

public  Object read() throws Exception {
String line;
		Object result = null;
		if (!i9CodesRead) {
			LoadI9List();
			// neutralizeSettings();
		}

		if (AF == null)
			AF = new AdditionalFile();
		FieldSet fieldSet;
 while((fieldSet = (FieldSet) itemReader.read()) != null) { 
			 line =	  fieldSet.readString(0);
			  
			 FieldSet tokenizedFieldSet = tokenizer.tokenize(line);
			  beforeProcess(tokenizedFieldSet);
			  if ( status !=false && valid)
			 if(status) { 
				 process(tokenizedFieldSet); 
				 } else {
			  process4010Files(tokenizedFieldSet); 
			  } }

.
.
.
.
.

return someObject;
}

My Application is throwing a exception only when the File Size is more than 15MB.

Can any one let me know where i am going .

[Dave Syer]

Maybe commit-interval="0" is the problem?

[anish555]

Having large value for commit-interval also will have same issue.