Flat File to DB mapping via config files?

[DavidC]

I need to import flat files into a database. I am using the sample fixedLengthImportJob as a starting point. So far, so good.

I would like to be able to add or delete fields without changing code. To add a field to fixedLengthImportJob it looks like you would have to change at least 3 java files: Trade.java, JdbcTradeWriter.java, and TradeFieldSetMapper.java in addition to the fixedLengthImportJob.xml. Not so good. Ideally, the behavior of these java classes would be driven from the job configuration file.

My questions are:
1. Is the paragraph above correct?
2. If yes, is it only because fixedLengthImportJob is simplified for example purposes?
3. Can anyone refer me to an example of what I want to do or give me a hint as to how to get started?
Thanks,
dc

[robert.kasanicky]

1. yes, you're right
2. I'd say contrary, it is more sophisticated than it needs to be
3. You probably don't want to map the fieldset to a domain object in your case, just use the fieldset directly (see PassThroughFieldSetMapper). Then you can write a generic FieldSetItemWriter which you can configure without changing java code. Maybe it would be worth to add such class to standard infrastructure as well.

[lucasward]

It is possible to do, but you would have to design for it, and be willing to sacrifice clarity and maintainability of your code in some respects.

What you could do, is create a FieldSetMapper that returns a list, rather than using a domain object (again, probably sacrificing a bit of maintainability/clarity here), and create a Dao that accepted an injected sql statement, and generated it's arguments list (Object [] as part of JdbcTemplate) by turning the list into an array and passing it to JdbcTemplate along with the injected Sql statement. Something like:

public class JdbcTradeDao extends JdbcDaoSupport{
  private final String sql;

  public JdbcTradeDao(String sql){
    this.sql = sql;
  }

  public void writeTrade(List trade){
    jdbcTemplate.update(sql, trade.toArray());
  }
}

The FieldSetMapper would have to look something like:

public class tradeMapper implements FieldSetMapper{
  public Object mapLine(FieldSet fs){
    return fs.getValues();
  }
}

(Please note that I typed this out by hand quickly without doing any compile checking, it's just a rough sketch)

The FieldSet values should return in the same order every-time, since fieldSet.get*(#) always returns the same value.

Personally, I would prefer an approach that used a domain object, since it feels more natural to me. If you used Hibernate to write the objects out, you would only need to update an hbm. You can also use the BeanWrapperFieldSetMapper to wire columns into a domain object using the JavaBean spec. However, it would still require that you update your domain object to include a new field. With all that being said, I can understand the appeal of a configuration only file load as well, and the above approach should work for that scenario.

[DavidC]

Thank you both. I'll give it a try and let you know how it goes.

[robert.kasanicky]

It is probably worth explicitly pointing out the key fact about the "no-mapping + fieldsetWriter" approach I suggested earlier: the flexibility you get is due to not having a domain model for the items you are processing.

Domain model can be an overkill for very simple jobs, but quicky becomes a necessity as complexity grows. So as usually it all comes down to making an informed decision what works best in particular scenario.