Aug 7th, 2011, 11:44 PM
reading from multiple files of different format
I have a scenario similar to the following example scenario:
Read a remote XML file with 2 columns with the first column as key.
Read a local flat file with 4 columns with one column as matcing key from above file.
Merge records from each file by matching key columns so that the pojo has now 5 properties (4 + 1 one key column common to both files).
Write the output to a database.
I am confused as to how the flow/job configuration will be and how will I be reading multiple files of different formats.
FootballJob exactly matches my requirement. But here I am talking about huge inputs (like 50000 records of 250 columns). So I am skeptic as to whether the approach taken in FootballJob will be efficient.
Last edited by nikunj_c; Aug 8th, 2011 at 04:00 AM.
Tags for this Thread