spring hadoop, job properties questions

[vito]

Hi,

I have a working hadoop process.

I would like to migrate it to spring framework using spring hadoop.

I have a few questions.


1. How does hadoop child procesess instatiate spring application context, Is the xml uploaded to distributed
cache?

2. configureReducerTypesIfPossible() at JobFactoryBean is not implemented, how does reducerKey and
value types determined?

3. How can I determine job properties like org.apache.hadoop.mapreduce.JobContext.setNumReduc eTasks() ,
If they are not in spring-hadoop.xsd ?




4.I would like to determine parameter 'input-path' dynamically at runtime, What is the recommened
approcah?

[leejianwei]

1. Spring Hadoop is not used to instantiate spring application context in Hadoop child process, it is used to manage hadoop jobs from Spring. The xml which is used to define beans, jobs will not be uploaded to distributed cache. The xml is used by main application or spring batch to instantiate beans, the beans will call hadoop client API.
2.Hadoop can determine type dynamically. Anyway, configureReducerTypesIfPossible() should be implemented later for safe.
3.After getting the "Job" fro "JobFactoryBean", you can set properties on the job with Job api.
4.You can use SpEL. to input-path property.

[vito]

How do I access the the job before IOC submits the job?

[vito]

Thanks,
The input path is determined dynamically based on many conditions.
How can I make sure I access the job before it is submitted to processing.

If I subclass JobFactoryBean and implement afterPropertiesset(), how do I use my implementation with the hadoop namespace ?

[Costin Leau]

vito, if you want to stay entirely on the declarative side of things then using SpEL is the way to go - not only you can evaluate things directly through the expression but you can also call methods on other beans which, in effect, behave as factories.
Subclassing JobFactoryBean means using the programmatic approach which means you can't use the namespaces as these work only against JobFactoryBean - you can however still configure the JobFactoryBean through 'classic' beans XML - after all, a namespace, is just a predefined, concise declaration (basically an XML template).
Hope this helps,

[vito]

Thank you,
I have not had a chance to use spring in a few years, Last version I used was 2.0. Seems spring has added a feature or two since
Can you link me with examples of using SpEL with that approach , that is call methods on other beans.
Regarding the job namespace, how can I access/inject the job created ?

[Costin Leau]

There are plenty of posts on SpEL: http://lmgtfy.com/?q=spring+spel
The reference docs can be found here: http://j.mp/Hnsclq

As for the job - if you are using the namespace or not, you are still assigning a bean to the job - so simply inject that job/bean and start using it.