Parsing Java log files

[afborroy]

We're migrating a batch process to Spring Batch 2. One of the steps parses several Java log files from different servers and extract stack trace block for identified exceptions.

We're trying to use multiline pattern without success. Is there any suitable approach in order to achieve this behaviour?

Thanks in advance.

[DHGarrette]

You could create a wrapper that watches for the end of the stack trace. The delegate FFIR would use a PassThroughLineMapper.

public class LogItemReader implements ItemReader<List<String>> {
	private FlatFileItemReader<String> delegate; // setter omitted
	private String last = null;
	private boolean done = false;

	public List<String> read() throws Exception {
		if (done) { // last item already returned
			return null;
		}

		List<String> item = new ArrayList<String>();
		if (last != null) {
			item.add(last); // first line of item already read
		}

		for (String line = null; (line = this.delegate.read()) != null;) {
			if (line.startsWith(" ") || item.isEmpty()) { // part same item
				item.add(line);
			}
			else { // start of next item
				last = line; // store first line of next item
				return item;
			}
		}

		// no more lines
		done = true;
		return item;
	}

[lucasward]

It almost sounds like you need a regex reader or something similar, which wasn't what the FlatFilteItemReader was designed to do. Even the traditional 'multi-line' approach was really there to support lines with header - detail - footer, not the kind of parsing you're looking to do. I think you'll have to write a custom reader.

[afborroy]

Finally, we've implemented following schema.

<beans:bean id="multifileASReader" class="org.springframework.batch.item.file.MultiResourceItemReader" lazy-init="true">
    <beans:property name="resources" value="file://${loc.working.directory.appserver}/${appserver.file.1}.*" />
    <beans:property name="delegate" ref="proxyFlatFileASItemReader" />
</beans:bean>

<beans:bean id="proxyFlatFileASItemReader" class="foo.bar.LogItemReader">
    <beans:property name="delegate" ref="flatFileASItemReader" />
</beans:bean>

<beans:bean id="flatFileASItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
  <beans:property name="lineMapper">
    <beans:bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper" />
  </beans:property>
</beans:bean>

public class LogItemReader 
    implements ItemReader<List<String>>, 
               ResourceAwareItemReaderItemStream<List<String>> {
  
  public List<String> read() throws Exception, UnexpectedInputException, ParseException {
  
      // Logic...

  }

  public void setDelegate(FlatFileItemReader<String> delegate) {
    this.delegate = delegate;
  }

  public void setResource(Resource resource) {
    delegate.setResource(resource);
  }

  public void close() throws ItemStreamException {
    delegate.close();
  }

  public void open(ExecutionContext executionContext) throws ItemStreamException {
    delegate.open(executionContext);
  }

  public void update(ExecutionContext executionContext) throws ItemStreamException {
    delegate.update(executionContext);
  }
  
}

Maybe it's inelegant, but it works fine.

Thanks both.

[Dave Syer]

Have you tested this in a restart scenario? I think it might be broken because you are doing a look ahead into the delegate, so when the framework calls update() it will sometimes (often) be in the wrong place.

[afborroy]

Dave, you're right.

In order to allow restarts, update method must be modified. It's easy with our requirements, because we're always reading the whole file, but it could be dangerous in other scenarios.

  public void update(ExecutionContext executionContext) throws ItemStreamException {
    executionContext.putInt(ecSupport.getKey(READ_COUNT), 0);
    delegate.update(executionContext);
  }

[DHGarrette]

You can either:

a) set saveState=false to ensure that the reader always starts at the beginning, or
b) save the values of "done" and "last" in update() and restore them in open().