Skipping advice on web service write failure
I have a job that reads from a folder of .xml files (each file is a reader item), does some transformation, and submits them to a vender webservice. Then an archive step moves the .xml files to an archive folder.
On occasion there is missing data and I get a SOAP exception. Right now the job fails on any error. This leaves all the .xml files in the processing folder (the successful ones, the failed one, and the unprocessed ones).
I have to manually archive the successful, remove the failed one for research, and then run the job again to process the rest of the files.
Obviously I'd like this to be automated by the job.
I have tried to use the skip functionality, however since I am writing to a webservice, rollback is not possible in a failed transaction. This results in multiple submits of the same data to the vender as the chunk is (re)processing to find the failing item. Even with a commit limit of 1, the failed record is submitted twice.
I have thought of two possible solutions.
One, keep using skips, but keep up with each filename written to the webservice and check against this list in the processor/writer to prevent multiple webservice writes. Move the failed file out to a research folder with a SkipListener.onSkipInWrite().
Two, don't use the skip feature and just trap the exceptions in the writer and move the failed file to a research folder and continue on processing.
Anyone have any thoughts on how to best accomplish this?