Filtering lines while reading from the files?

[ramkris]

Hi,
We need to filter out few lines from the files based on the pattern we specify in the configuration. I am not sure whether this feature exists in 1.0.0 release.
In M3, we overridden the DefaultFlatFileInputSource.readLine() method to do that. But were not able to do this 1.0.0 as this has become a private method in FlatFileItemReader. So we had to do this read() method which returns a FieldSet. I am converting this to string again to match the pattern which is not a good way.

It would be great if the framework provides this feature.

Also i attached the java file with this.

thanks in advance,
Ram

[Dave Syer]

DefaultFlatFileInputSource is from a very old version. Can you upgrade to 1.1.0.FINAL?

FlatFileItemReader has some basic support for ignoring lines that *begin* with one of a list of strings you give it (setComments(String[])). That works for most people (and it looks like it would work for the simple example you gave), but I like the filter idea as well. If you want to see it in the framework I suggest you open a JIRA.

[dkaminsky]

Off the top of my head: you could use an item transformer and a custom exception with a skip policy.

public abstract class FilteringItemTransformer implements ItemTransformer {
  protected abstract boolean accept(Object item);

  public Object transform(Object item) {
    if( ! accept( item ) ) {
      throw new ItemFilterException( item );
    }

    return item;
  }
}

public class ItemFilterException extends RuntimeException {
   /**
    * Generated
    */
   public static final Long serialVersionUID = -35153205L;

   private Object item;

   public ItemFilterException(Object item) {
     this.item = item;
   }
}

Then all you need is to configure your step with an ItemTransformerItemWriter using that transformer and your desired writer as its delegate.
Then just configure ItemFilterException as a skippable exception...

An alternate solution would be to change the ItemTransformerItemWriter (or create an alternate version) to not continue to the "write" phase if the returned transformed value is null - then the above-mentioned transformer looks like this:

public abstract class FilteringItemTransformer implements ItemTransformer {
  protected abstract boolean accept(Object item);

  public Object transform(Object item) {
    if( ! accept( item ) ) {
      return null;
    }

    return item;
  }
}

This way a separate exception / skippable policy is not required.

Created a JIRA issue --
http://jira.springframework.org/browse/BATCH-581