Upload Files from web and Spring Batch?

[jilljuneja]

Our business needs is to upload files from web and insert into database. Files can have upto 50,000 records.

I have been looking into various solutions our the web and spring batch keeps popping up for processing large files.

I do understand spring batch is main intent is for batch processing. is spring batch good fit for external users to upload files in database or should I look into something else.

Thanks and Regards,
Jill

[snicoll]

To scale your batch processing should be asynchronous somehow. One solution would be to stream the uploaded file to a local storage and kick-in a new job with a parameter defining where the file is located.

This is fairly easy. But if you want to stream directly what you uploaded, that sounds like more complicated than it should and it probably breaks the chain of responsability

[syedmuneer]

Hi,
How do you need to insert files in DB, each line as a row in table. I dont know about other options, but spring batch is good fit for such requirement.
I have implemented almost similar requirement using spring batch.
A client (either thru program or web-ui provided) can send a huge file having huge amount of data (and each line represent a row in DB table) and I am validating both header and lines and storing validation success records (with if present update else insert logic) and for failed records writing to error table and also returning in failed line number details in http response.

[syedmuneer]

Hi,
You can do both in sync and async way (by chosing SyncTaskExecutor or any implementation of AsyncTaskExecutor) and dont have to store the file locally, can be read directly into inputstream and pass it as inputstreamresource to FlatFileItemReader.