Multiple Output files

[prashu_spring]

Hi,

I have a requirement to read and write files in one-one fashion.

I.e. lets say below are the files to be read.

employee_Jan_2006.log
employee_Jan_2007.log
employee_Jan_2008.log

I can read them using MultiResourceItemReader but, for writing i need to create and write files as below after some transformation for each file.

employee_Jan_2006.log has to be written into employee_Jan_2006_done.log

employee_Jan_2007.log has to be written into employee_Jan_2007_done.log

employee_Jan_2008.log has to be written into employee_Jan_2008_done.log

How can i associate the output file names dynamically depending on input file name?


Thanks in advance.

Regards,
Prashant

[DHGarrette]

The MultiResourceItemReader is intended to seamlessly switch between input files, so using it wont tell your writer to switch files. The easiest way to solve this problem is probably to launch a single-file job for each file.

[prashu_spring]

Thanks for your reply. But I dint know how many files are there in a input folder. I have mentioned the resources as below for multi reader.

<bean id="multiResourceReader" class="org.springframework.batch.item.file.MultiRe sourceItemReader">
<property name="resources" value="classpath:emp*.txt"></property>
<property name="delegate" ref="reader" />
</bean>

So, for every file with emp*.txt (Example employee_Jan_2009.txt) format, i need to create its corresponding out put file with some transformation.

Please advise how it can be achieved in Spring Batch.

Thanks,
Prashant

[DHGarrette]

Prashant,

You should wrap the SimpleJobLauncher and add a property for resources that can be wired in in the same way as you do for MultiResourceItemReader. When you call run(), your job launcher will loop through these resources. For each iteration, it will create a job parameter with the particular resource and call run() on the SimpleJobLauncher, thus launching for the particular file.

[prashu_spring]

Thanks it is working fine. But only issue is JobExecution which i return only for last iteration.

[rinpoche]

Prashant:

I am also working on a similar scenario, where I need the output filename to be a variation of the input filename. And I have multiple input files that match a filename pattern, similar to what you have. Can you share some of your code?

Regards
Ajay

[prashu_spring]

We had written our own Job launcher which goes as below. But it has our own specific requirements, but it may help you for your problem.

public class BufJobLauncher extends SimpleJobLauncher
{
private String resources;

/**
* resources
* @return
*/
public String getResources()
{
return resources;
}

/**
* It customised for BUF run where for every input operation usage file, we
* need to create its corresponding BUF and billing file.
* @param job
* @param jobParameters
* @return
* @throws JobExecutionAlreadyRunningException
* @throws JobRestartException
* @throws JobInstanceAlreadyCompleteException
* @see org.springframework.batch.core.launch.support.Simp leJobLauncher#run(org.springframework.batch.core.J ob,
* org.springframework.batch.core.JobParameters)
*/
public JobExecution runBuf( final Job job, final JobParameters jobParameters ) throws JobExecutionAlreadyRunningException,
JobRestartException, JobInstanceAlreadyCompleteException
{
JobExecution jobExe = null;

// get the file pattern
String filePattern = resources.substring( resources.lastIndexOf( File.separator ) + 1 );
// Deleting existing dir is configurable in buf.properties.
if( Boolean.valueOf( BufProperties.getInstance().getProperty( BufConstants.DELETE_EXISTING_DIR ) ) )
{
deleteExistingDirectories();
}

File file = new File( resources );
file = new File( file.getParent() );
if( !file.exists() || !file.isDirectory() )
{
throw new JobRestartException( "Missing operation usage file(s) or directory" );
}

File[] operationFiles = file.listFiles();

Map<String, String> stringMap = new HashMap<String, String>();
Map<String, Date> dateMap = new HashMap<String, Date>();
stringMap.put( BufConstants.CLIENT_ID, jobParameters.getString( BufConstants.CLIENT_ID ) );
populateStartEndDates( dateMap, jobParameters );

for( File operationUsageFile : operationFiles )
{
if( !operationUsageFile.isDirectory() )
{
String filename = operationUsageFile.getName();
// match the file pattern
if( filename.contains( StringUtils.delete( filePattern, "*" ) ) )
{
stringMap.put( BufConstants.FILE_NAME, filename );
jobExe = super.run( job, new JobParameters( stringMap, new HashMap<String, Long>(),
new HashMap<String, Double>(), dateMap ) );
}
}
}
return jobExe;
}

/**
* resources
* @param resources
*/
public void setResources( final String resources )
{
this.resources = resources;
}

/**
* Deletes buf, billing, consolidated billing folders
*/
private void deleteExistingDirectories()
{
String folderNames[] = {
BufProperties.getInstance().getProperty( BufConstants.BUF_FOLDER ),
BufProperties.getInstance().getProperty( BufConstants.BILLING_FOLDER ),
BufProperties.getInstance().getProperty( BufConstants.CONSOLIDATED_BILLING_FOLDER )
};

for( String folderName : folderNames )
{
File folderToBeDeleted = new File( folderName );
if( folderToBeDeleted.isDirectory() )
{
File[] list = folderToBeDeleted.listFiles();
//First delete all the files in the specified folder.
for( File fileToBeDeleted : list )
{
fileToBeDeleted.delete();
}
//After deleting all the files inside the folder, now delete the folder itself
boolean deleted = folderToBeDeleted.delete();
if( !deleted )
{
logger.debug( "Unable to delete: " + folderToBeDeleted.getAbsolutePath() );
}
}
else
{
logger.error( "Invalid directory name: " + folderToBeDeleted.getAbsolutePath() + " given in properties file" );
}
}
}

private void populateStartEndDates( final Map<String, Date> dateMap, final JobParameters jobParameters )
{
SimpleDateFormat dateFormat = new SimpleDateFormat( BufProperties.getInstance()
.getProperty( BufConstants.INPUT_DATE_FORMAT ) );
try
{
dateMap.put( BufConstants.START_DATE, dateFormat.parse( jobParameters.getString( BufConstants.START_DATE ) ) );
dateMap.put( BufConstants.END_DATE, dateFormat.parse( jobParameters.getString( BufConstants.END_DATE ) ) );
}
catch( ParseException e )
{
logger.error( " Start and End Date not in 'yyyyMMdd' format: ", e );
}
}
}

[rinpoche]

Prashant:

Thank you. Can you please also share the corresponding job and associated XML configuration files? Mainly the job and the context files.

Regards
Ajay

[DHGarrette]

Prashant,

It would also be helpful if you could edit your post to use CODE tags. That would make it much more readable.

[rinpoche]

Dan: In this same scenario where there are multiple input files, and a corresponding output file with a similar name is written out for each input file, how can parallel processing or any other approach be employed to improve speed of processing?