Problem with sample - generation of output file

[vijaynairis]

Hi,

I have been trying to spew the contents of my table into a file as part of a batch process...I tried to model the output generation using the sample provided for output file generation - Refer FlatFileCustomerCreditDao...While the file is indeed getting created, the contents seem to be empty...Was just wondering if its a problem with my FieldSetCreator ...Thanks in advance...

Here is the configuration

    <bean
		class="com.fcpb.integration.batch.advice.dao.OrderWriter"
		id="orderReportWriter">
		<property name="itemWriter">
			<bean
				class="org.springframework.batch.item.file.FlatFileItemWriter"
				id="orderFlatFileOutputSource">
				<property name="resource" ref="orderFileLocator" />
				<property name="fieldSetCreator">
					<bean
						class="org.springframework.batch.item.file.mapping.PassThroughFieldSetMapper" />
				</property>
			</bean>
		</property>
	</bean>

and here's the code within my writer

	public void write(Object data) throws Exception {
                log.debug("Order Writer is getting invoked");
                log.debug("Item Writer is"+itemWriter);
                Order order = (Order)data;
                log.debug("Order Id is"+order.getOrderId());
                log.debug("Order date is"+order.getOrderDate());
            	if (!opened) {
			open(new ExecutionContext());
		}
                String line = "" + order.getOrderId() + separator
				+ order.getOrderDate();

                log.debug("This line is"+line);
		itemWriter.write(line);
                
        }

[Dave Syer]

How did you implement flush() and clear() (looks like you should be delegating to the itemWriter, but maybe you ommitted to do that)?

[vijaynairis]

Hi dave,

Thanks for the reply. Injected ItemWriter as a dependency into this class

heres the code

package com.fcpb.integration.batch.advice.dao;

import com.fcpb.integration.batch.model.Order;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.batch.item.AbstractItemWriter;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStream;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.DisposableBean;


public class OrderWriter extends AbstractItemWriter implements DisposableBean {

        private static Log log = LogFactory.getLog(OrderWriter.class);
        private ItemWriter itemWriter;
        private String separator = "\t";
        private volatile boolean opened = false;

	public void write(Object data) throws Exception {
                log.debug("Order Writer is getting invoked");
                log.debug("Item Writer is"+itemWriter);
                Order order = (Order)data;
                log.debug("Order Id is"+order.getOrderId());
                log.debug("Order date is"+order.getOrderDate());
            	if (!opened) {
			open(new ExecutionContext());
		}
                String line = "" + order.getOrderId() + separator
				+ order.getOrderDate();

                log.debug("This line is"+line);
		itemWriter.write(line);
                
        }

	public void setSeparator(String separator) {
		this.separator = separator;
	}

	public void setItemWriter(ItemWriter itemWriter) {
		this.itemWriter = itemWriter;
	}

	public void open(ExecutionContext executionContext) throws Exception {
		if (itemWriter instanceof ItemStream) {
                    log.debug("Calling execution context");
			((ItemStream) itemWriter).open(executionContext);
		}
		opened = true;
	}

	public void close() throws Exception {
		if (itemWriter instanceof ItemStream) {
			((ItemStream) itemWriter).close(null);
		}
	}

	
	public void destroy() throws Exception {
		close();
	}
}

[lucasward]

Dave is right, you have to pass through the calls to flush and clear. The FlatFileItemWriter buffers output and won't actually write until flush is called. You need to either override them in your OrderWriter, or register the FlatFileItemWriter as a stream in the step.

[shaninga]

Just add two line in red

package com.fcpb.integration.batch.advice.dao;

import com.fcpb.integration.batch.model.Order;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.batch.item.AbstractItemWriter;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStream;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.DisposableBean;


public class OrderWriter extends AbstractItemWriter implements DisposableBean {

private static Log log = LogFactory.getLog(OrderWriter.class);
private ItemWriter itemWriter;
private String separator = "\t";
private volatile boolean opened = false;

public void write(Object data) throws Exception {
log.debug("Order Writer is getting invoked");
log.debug("Item Writer is"+itemWriter);
Order order = (Order)data;
log.debug("Order Id is"+order.getOrderId());
log.debug("Order date is"+order.getOrderDate());
if (!opened) {
open(new ExecutionContext());
}
String line = "" + order.getOrderId() + separator
+ order.getOrderDate();

log.debug("This line is"+line);

itemWriter.clear();
itemWriter.write(line);
itemWriter.flush();


}

public void setSeparator(String separator) {
this.separator = separator;
}

public void setItemWriter(ItemWriter itemWriter) {
this.itemWriter = itemWriter;
}

public void open(ExecutionContext executionContext) throws Exception {
if (itemWriter instanceof ItemStream) {
log.debug("Calling execution context");
((ItemStream) itemWriter).open(executionContext);
}
opened = true;
}

public void close() throws Exception {
if (itemWriter instanceof ItemStream) {
((ItemStream) itemWriter).close(null);
}
}


public void destroy() throws Exception {
close();
}
}

[vijaynairis]

Thanks guys...it worked...guess the samples need a bit of a documentation for explanation..a one liner perhaps

Thanks again

[Dave Syer]

You don't want to be calling flush() and clear() inside the write method. You should just delegate the calls down to the item writer you injected. The user guide has quite a lot of material on the ItemWriter interface. Have you checked that out?

[vijaynairis]

Hi Dave,

Just to confirm what you are saying....I do make a call to the itemwriter.flush() and itemwriter.clear() for the FlatFileOutputItemWriter that I have injected....

So are you saying taht I do not do that ?

I did go through the section on ItemWriter interface but mostly take hints from the samples

Cheers...VJ

[lucasward]

You definitely don't want to be calling flush and clear within the write method...you do want to pass them through, but not call them yourself. The framework should be completely controlling when flush and clear is called, if not, rollback won't work correctly. You should be overriding the flush and clear methods in AbstractItemWriter. Like so:

ackage com.fcpb.integration.batch.advice.dao;

import com.fcpb.integration.batch.model.Order;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.batch.item.AbstractItemWriter;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStream;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.DisposableBean;


public class OrderWriter extends AbstractItemWriter implements DisposableBean {

private static Log log = LogFactory.getLog(OrderWriter.class);
private ItemWriter itemWriter;
private String separator = "\t";
private volatile boolean opened = false;

public void write(Object data) throws Exception {
log.debug("Order Writer is getting invoked");
log.debug("Item Writer is"+itemWriter);
Order order = (Order)data;
log.debug("Order Id is"+order.getOrderId());
log.debug("Order date is"+order.getOrderDate());
if (!opened) {
open(new ExecutionContext());
}
String line = "" + order.getOrderId() + separator
+ order.getOrderDate();

log.debug("This line is"+line);

itemWriter.write(line);

}

public void clear(){
itemWriter.clear();
}

public void flush(){
itemWriter.flush()
}
public void setSeparator(String separator) {
this.separator = separator;
}

public void setItemWriter(ItemWriter itemWriter) {
this.itemWriter = itemWriter;
}

public void open(ExecutionContext executionContext) throws Exception {
if (itemWriter instanceof ItemStream) {
log.debug("Calling execution context");
((ItemStream) itemWriter).open(executionContext);
}
opened = true;
}

public void close() throws Exception {
if (itemWriter instanceof ItemStream) {
((ItemStream) itemWriter).close(null);
}
}


public void destroy() throws Exception {
close();
}
}

[vijaynairis]

Thanks guys for all the support and patient responses....much appreciated

Just one more quick question regarding the buffer size...Suppose I set a commit interval of 1000 , for a job which consists of a read from a table and output to a file (My output file is close to around a million records in production) how does the flushing mechanism work for the output file...Would a value of 1000 make sense only for database operations but does it play a factor for an output buffer/flush also ?

Thanks...Vijay