Feb 1st, 2011, 03:15 AM
JdbcCursorItemReader throw out of memory in postgresql
JdbcCursorItemReader throw out of memory in postgresql.
i am using latest version(2.1.5.RELEASE).
it seem that all the data is loaded in the client side.
i am reading a lot of data (about 50 million rows), so increasing the memory will help in this case (not that is should work in this way...).
does any body encounter this problem in this version ?
how can i fix it ?
Feb 4th, 2011, 09:32 AM
The memory consumption of the JdbcCursorItemReader really depends on the database driver (how it efficiently "streams" the data). You can consider increasing the memory of the batch process, but perhaps the PostgreSQL driver isn't efficient enough for such datasets, period. Did you consider using the JdbcPagingItemReader instead?
Feb 10th, 2011, 09:30 AM
thanks for your replay.
Originally Posted by arno
well, performance will be problematic when reading large volumes of data.
Feb 10th, 2011, 10:12 AM
that's where the JdbcCursorItemReader can help. Instead of using one query and one ResultSet (and being dependant on the driver's cursor implementation), Spring Batch will handle the paging for you. You can use pages of 100 items and you shouldn't face memory exhaustion, at the price of issuing multiple queries (which is a fair trade-off).
Originally Posted by eranmiz133
Tags for this Thread