java heap problem in spring batch OutOfMemoryException )

[daggoluAnil]

Hi,
I am new to spring batch , iam facing problem with heap memory

requirement: i have flat file (500 MB (1 million lines)) , i need to write this file data into gemfire catch, and i set java max heap size 1GB .


Even i am geting OutOfMemoryException while i am procesing

1) how max heap size it requires to process 1 GB flat file( it may increase)
2) is it require flush or clean once wrien into cache ,if it how?
3)is it any problem with m code below.

kindly let know ans.




code:
======
<gfe:cache id="cacheWithXml" />
<batch:job id="toCacheFile">

<batch:step id="step1_1">
<batch:tasklet transaction-manager="jobRepository-transactionManager">
<batch:chunk reader="itemReader1" writer="DBToCache"
commit-interval="1"/>
</batch:tasklet>
</batch:step>

</batch:job>


<bean id="DBToCache" class="com.FeedToCache">
<property name="cache" ref="cacheWithXml" />
</bean>

cache.xml
=======

<cache>
<disk-store name="ds" auto-compact="true" max-oplog-size="1024" queue-size="10000" time-interval="10">
<disk-dirs>
<disk-dir >c:/filecache</disk-dir>
</disk-dirs>
</disk-store>

<region name="cacheRegion">
<region-attributes data-policy="persistent-replicate" scope="local"
refid="REPLICATE_PERSISTENT" statistics-enabled="true" disk-store-name="ds" disk-synchronous="false" >


</region-attributes>
</region>
</cache>

write metthod for cachig data(FeedToCache.java)
======================
public void write(List<? extends FeedData> items) throws Exception {
for (FeedData item : items) {
if (item!= null)
if(item.getAccountNumber()!=null)
cacheRegion.put(item.getAccountNumber()+Math.rando m(),item);

}

}


Thanks
Anil.

[Marten Deinum]

Please use [ code][/code ] tags when posting code, that way it remains readable ...

Post your configuration of the reader (maybe some code) and post the full stacktrace (remember use [ code][/code ] tags!!!!)

[bwawok]

The simple way to see how much memory your program needs to run.. is give it way more memory, watch how big your heap grows, then decide if you you can reduce memory usage

Run with a 64-bit JVM, and set your XMX to 16 GB.... watch your heap usage with jVisualVM... and see how high you get. Then decide if its cheaper to just leave XMX there, or work on using less memory.

[daggoluAnil]

my reader code:
=================

<bean id="itemReader1" class="org.springframework.batch.item.file.FlatFil eItemReader">
<property name="resource" value="file:///C:/anil.FEED3" />
<property name="encoding" value="ISO-8859-1" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping .DefaultLineMapper">
<property name="lineTokenizer" ref="accountTokenizer">
</property>
<property name="fieldSetMapper" >
<bean class="com.FeedDataFieldSetMapper" />
<property>
</bean>
</property>
</bean>

error log
========
SystemFailure Watchdog: [warning] failure detected
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethod(Unknown Source)

[error 2011/04/29 11:44:13.845 IST <UDP mcast receiver> tid=0x15] ThreadGroup:
Uncaught Exception in thread UDP mcast receiver
java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.getDeclaredMethod(Unknown Source)
at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
at java.io.ObjectStreamClass.access$1700(Unknown Source)
at java.io.ObjectStreamClass$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(Unknown Source)
at java.io.ObjectStreamClass.lookup(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at com.gemstone.gemfire.internal.InternalDataSerializ er.basicWriteObject
(InternalDataSerializer.java:1552)
at com.gemstone.gemfire.DataSerializer.writeObject(Da taSerializer.java:3
025)
at com.gemstone.gemfire.internal.util.BlobHelper.seri alizeToBlob(BlobHel
per.java:39)
at com.gemstone.gemfire.internal.cache.EntryEventImpl .serialize(EntryEve
ntImpl.java:1282)
at com.gemstone.gemfire.internal.cache.DiskEntry$Help er.writeToDisk(Disk
Entry.java:450)
at com.gemstone.gemfire.internal.cache.DiskEntry$Help er.doAsyncFlush(Dis
kEntry.java:1050)
at com.gemstone.gemfire.internal.cache.DiskStoreImpl$ FlusherThread.run(D
iskStoreImpl.java:1661)
at java.lang.Thread.run(Unknown Source)
at java.io.ObjectStreamClass.getPrivateMethod(Unknown Source)
at java.io.ObjectStreamClass.access$1700(Unknown Source)
at java.io.ObjectStreamClass$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(Unknown Source)
at java.io.ObjectStreamClass.lookup(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at com.gemstone.gemfire.internal.InternalDataSerializ er.basicWriteObject
(InternalDataSerializer.java:1552)
at com.gemstone.gemfire.DataSerializer.writeObject(Da taSerializer.java:3
025)
at com.gemstone.gemfire.internal.util.BlobHelper.seri alizeToBlob(BlobHel
per.java:39)
at com.gemstone.gemfire.internal.cache.EntryEventImpl .serialize(EntryEve
ntImpl.java:1282)
420393 [main] ERROR org.springframework.batch.core.step.AbstractStep - Encounte
red an error executing the step: class java.lang.OutOfMemoryError: Java heap space




Here i am sending reader code and error log what i am getting . kindly check and respond me.

[Marten Deinum]

Please as stated use [ code][/code ] tags so that we can actually (easily) read it...

However judging from the stacktrace it is a gemfire issue. I'm not that familiar with gemfire so I cannot help you there. But it is either a bug or some misconfiguration on your side.

[daggoluAnil]

Same issue i got for flat file writing . and even i tried with writing into queue also But i am facing same problem.

[Marten Deinum]

Then it has to do with either your configuration, environment, fieldsetmapper... You are keeping objects somewhere (at least that is what it then looks like).