JdbcCursorItemReader throw out of memory in postgresql

[eranmiz133]

JdbcCursorItemReader throw out of memory in postgresql.

i am using latest version(2.1.5.RELEASE).
it seem that all the data is loaded in the client side.
i am reading a lot of data (about 50 million rows), so increasing the memory will help in this case (not that is should work in this way...).

does any body encounter this problem in this version ?
how can i fix it ?

thanks !

[arno]

The memory consumption of the JdbcCursorItemReader really depends on the database driver (how it efficiently "streams" the data). You can consider increasing the memory of the batch process, but perhaps the PostgreSQL driver isn't efficient enough for such datasets, period. Did you consider using the JdbcPagingItemReader instead?

[eranmiz133]

The memory consumption of the JdbcCursorItemReader really depends on the database driver (how it efficiently "streams" the data). You can consider increasing the memory of the batch process, but perhaps the PostgreSQL driver isn't efficient enough for such datasets, period. Did you consider using the JdbcPagingItemReader instead?

thanks for your replay.

well, performance will be problematic when reading large volumes of data.

[arno]

well, performance will be problematic when reading large volumes of data.

that's where the JdbcCursorItemReader can help. Instead of using one query and one ResultSet (and being dependant on the driver's cursor implementation), Spring Batch will handle the paging for you. You can use pages of 100 items and you shouldn't face memory exhaustion, at the price of issuing multiple queries (which is a fair trade-off).