BeanWrapperTransformer, anyone?

[kyrill007]

Basically I am looking for a way to do bi-directional transformation from an Object root (i.e. bean) to and from a mainframe/cobol fixed-length file.

Initially I thought that I will have to write it all myself. When I discovered BeanWrapperFieldSetMapper I was really excited. It looks like the "read from file into an object" part is done. I really like that it provides a nice way to register custom property editors to do transparent type conversion around common Cobol data types (that is, for example, if I need to convert 0000012345 into 12.345 BigDecimal, I can do it). THANK YOU, Dave, thank you, guys.

Well, then I started looking for a way to persist my bean (or more precisely any object can be handled with BeanWrapperImpl) back into such a file, and I didn't find anything clearly satisfactory.

It seems that FlatFileItemWriter can be used with a yet-to-be-written Transformer, say, BeanWrapperTransformer to achieve it.

I think such capability is an important feature and would add real value to the framework.

PROPOSAL:

1. Basically I think we need a better abstraction of a flat-file structure.

The problem with FixedLengthLineAggregator is that padding and alignment is often type specific. That is Numeric types are typically right-justified and padded with leading zeros, while textual types are left-justified with trailing spaces. So, this info needs to be specified on a per Range basis. I propose we add optional 'padding' and 'alignment' attribute to a Range class, and make it implement Comparable (where comparison is delegated to the min attribute, i.e. starting position), so we can sort them if we need to.

2. We need support for literal values, that is to tell declaratively 'Fill the Range 4:7 with 'XXX' value.

Presumably, we can reuse the same idea that the DefaultFieldSet uses, that is represent a record structure with an array of property names and an array of Ranges, then BeanWrapperTransformer can detect that if the property name starts with 'literal:', it should just use this value and not query the actual bean (that is passed as an argument to a transform method).

3. Implement the BeanWrapperTransformer (it will extend DefaultPropertyEditorRegistrar)

4. (Optional) Create proper PropertyEditors to convert to and from typical mainframe data types like BigDecimal (000012345 -> 12.345 where scale is configurable) and Boolean ('Y' and 'N' - the values should be configurable). The actual padding will be done inside the Transformer based on the individual Range configurations, but the type conversion will delegate to property editors.

I will be more than happy to write it (I am almost done actually). If you guys approve it of the idea, then I will submit a patch. I hope there is still time for it to make it into 1.0 release.

Let me know if you want to take the discussion off-line or into JIRA.

Thanks a lot for your consideration.

-Kyrill Alyoshin

[lucasward]

I would say we agree. In fact, there's already a jira issue for having separate alignment options:

http://jira.springframework.org/browse/BATCH-278

A similar BeanWrapper style approach does seem like it would make sense. There has actually been some changes to the Tokenizer recently to help with some of this. You might want to take a look at the latest snapshot. We weren't planning on getting to it in Release 1 though. Milestone 5 should be out next week, and we don't want to add many new features between then and RC1. However, please submit the patch and we'll take a look at soon as we can.

[kyrill007]

Excellent!

I've created a JIRA issue for this task and "verbally" assigned myself to it :-)

http://jira.springframework.org/browse/BATCH-379

I think I should have done over the weekend.

[Tony Falabella]

Guys,

I concur with Kyrill that something is needed to address all the points he's made. The mainframe data types example is a common thing people have to tackle. I've had to do it and I was going to propose that Spring Batch come up with a bunch of "canned" property editors for doing things just like that. I think there's a lot of value-add there.

I'd however take what's been discussed a step further by introducing some "meta-data" classes to the infrastructure package. Specifically a FieldMetaData and FieldSetMetaData class. The FieldMetaData would look similar to a ResultSetMetaData or RowSetMetaData but would have some additional properties added (i.e. PropertyEditor(s), alignment, padding, pattern, etc). There are a number of reasons that I have for introducing those classes:

• From the Spring Batch documention, [quote]a FieldSet is conceptually very similar to a Jdbc ResultSetMetaData or RowSetMetaData[quote]. Both of those classes however, offer a way to get at the metadata object for the row, via getMetaData().
• Sometime it's overkill to map a row into a domain object (i.e. when you're just displaying the rows of a file on a screen or directly dumping them into a database), or rows don't map easily into domain objects for a file. See http://forum.springsource.org/showthread.php?t=49347
• When you're importing a text file in Excel, it allows you to set metadata like type and format for each column/field.
• DefaultFieldSet is not consistent with overloads to methods regarding patterns and default values. Can't all objects have defaults and patterns applied?
• DefaultFieldSet already contains some "meta-data", though currently that is only the field names.
• Padding - can't it be applied to both delimited and fixed length tokenizers? What about padding when outputting?
• Offer an "allow null" for fields containing an empty string.
• Introducing those 2 classes would also allow you to perhaps introduce say an adapter class for the JSTL ResultSupport and
  ResultImpl classes that know how to display data columns on a JSP via their meta-data.
• Perhaps a class can then be introduced that maps a FieldSet to a database row (maybe via the BeanWrapperFieldSetMapper) merely using the FieldSetMetaData. See http://forum.springsource.org/showthread.php?t=49347.
• It would help with this issue: http://jira.springframework.org/browse/BATCH-261
• It would help with some of these issues: http://forum.springsource.org/showthread.php?t=47861
• Some nice Spring namespaces can then be introduced for easing the coding of the meta-data info describing the FieldSets.
• Companies I've worked at have implemented solutions that used metadata at the field and row level. Commericial ETL tools I'm familiar with also implement some sort of metadata at the field/row level.
  

Initially I see the AbstractLineTokenizer and DefaultFieldSet making the most use of these classes but a lot of possibilities open up if they were introduced.

I have a lot of this code written and would like to contribute it to the project. I was waiting until I had it a little more polished before I did so but perhaps I can share what I have sooner rather than later to see if it's something the project is interested in at all. I can probably clean it up enough for general consumption this weekend.

Sorry for the long post.

[Tony Falabella]

To clarify how the FieldMetaData might help with automatically being able to map fields to database columns or a JSP page, it would contain something like this:

private Class requiredType = String.class;

which would then allow the FieldSet to introduce some new methods like:

Object readObject(int index);
Object[] getValuesAsObjects();

[kyrill007]

1. We do a need a better abstraction for the fixed-length record element. But I'd say that all that is missing is alignment direction and padding character. For now I am going to modify range to include those.

2. Property Editors. Again, I have a feeling that Spring's built-in property editors do the trick. The only thing that is missing is CobolBigDecimalEditor (where you need to do fixed scaling, i.e. 1234.124 -> 1234124). CustomNumberEditor formatter will happily do "000123" into 123 int. CustomBooleanEditor can be configured for 'Y' and 'N'. CustomDateEditor can represent virtually any date format.

-Kyrill

[Tony Falabella]

Kyrill,

I agree that Spring's PropertyEditors will handle most of what people will need to do. Here's a valid use-case scenerio though.

Suppose I'd like the word "set" to represent true and "unset" to represent false. Out of the box, Spring registers a bunch of "default" editors for you within the PropertyEditorRegistrySupport when an application context starts. However, some assumptions are made for each editor it creates. For example, when Spring registers the default CustomBooleanEditor it assumes that true/false will be represented by "true/false, "on/off", "yes/no", or "1/0" and for booleans it does not allow null while for Booleans it does allow null. The FieldSet currently provides a limited workaround for handling this use-case via this method:

public boolean readBoolean(int index, String trueValue)

That method does not allow someone to set the value for false though and the underlying code assumes "if it's not true then it's false". That might not be valid for someone that wants to know if a value has been encountered that doesn't conform to the special values I've indicated in my above use-case example. Allowing people hooks to set a PropertyEditor on an individual field themselves solves that problem.

With regard to the CobolBigDecimalEditor you're working on, besides being able to handle the "implied decimal" will it handle things like this as well (or can you code it do this too)?
EBCDIC to ASCII Conversion of Signed Fields
Here's some sample code for that: Overpunch

Since I'm assuming it will be able to handle all types of numbers not just BigDecimal (i.e. BigInteger, decimal, long, float) perhaps you might consider naming it CobolNumberEditor or even better, EbcdicNumberEditor? That would be more consistent with the CustomNumberEditor.

[kyrill007]

Well, custom boolean values can easily be handled by registering own CustomBooleanEditor(String, String, boolean), which you obviously recognize.

As for EBCDIC editor... Well... EBCDIC is an encoding. Probably not the best word... We probably need a separate CobolSignedNumberEditor. It is likely that should not overwrite default property editors, because regular numbers are OK for Cobol too, and will need to be registered on a per bean property basis. (I am probably not going to write it right now... :-))

[kyrill007]

I've submitted a patch to BATCH-379 that implements my proposal. Waiting for the Batch Team feedback...

[rabueckers]

Is there now a common approach for handling of the Cobol data types and the Java types? Specifically: if I need to convert 12345678 into 123456.78 BigDecimal. I'm using Spring-Batch 2.0

Does anyone have a Spring or ideally a Spring-Batch config and associated unit tests where thay have this working? For some reason my config is using the default property editor CustomNumberEditor which doesn't handle the above mentioned conversion.

I'm attempting to read a fixed lenght file into a Java Object. I've setup my app context to populate the CustomEditorConfigurer.customEditors property but it still appears to call the default CustomNumberEditor for java.math.BigDecimal types.

Any help would be greatly appreciated!

Thanks!

Bob