Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 27

Thread: Contribute some new adapter code to Spring integration.

  1. #11
    Join Date
    Oct 2011
    Location
    Mumbai, India
    Posts
    213

    Default

    Hello Mike, thanks a lot for your detailed feedback. Sorry to hear you are unable to use it for the reasons you have mentioned and i will work on those to sort the issues out.
    Regarding your concerns, please find my answer to them below.

    For various reasons, we cannot upgrade to Spring 3.1, but for some reason, your SQS adapter uses a ConversionService implementation only available in 3.1. I don't know why your SQS adapter needs to handle any conversion at all. Isn't that the job of Message Transformers?
    The messages sent to SQS can only contain string payload, where as, we cannot restrict the Spring integration message sent to the outbound adapter to have a payload of type String only. It can contain any type as long as the conversion service can convert the object to string and vice versa. This is where the conversion service comes into picture. In its absence, the user would have to convert the user provided object to a type string say using transformers as you suggested (which should also contain the original class type so that it can be converted back to the object ) and vice versa upon receiving the string message from the SQS queue. Also, when I developed this library, i had kept in mind spring integration version 2.1.x which makes use of conversion services as well.
    I will however go though this thing again and rethink on it to eliminate the use of conversion services.

    BTW, what version of spring and spring integration are you using ?

    While the XML configuration offered by your XSD makes dropping in the functionality very simple, it does not allow for overriding various components. It would be great if I could just point it to a reference of my sqsClient, which might be a slightly different implementation rather than the default (add encryption, etc.). Also, you use a default Json message transformer, but don't have the configuration in place to use a different message transformer, or even customize the configuration of the Json transformer. We have some Jackson modules that are necessary to serialize/deserialize our data, but we can't add that in with the XSD configuration.
    This shouldn't be too difficult to allow accepting an instance of sqs client and the transformer to use. There is an interface org.springframework.integration.aws.sqs.core.Amazo nSQSOperations which has methods to send, receive and delete messages from the SQS queue. Providing an implementation of this interface should override the default implementation that is used to send and receive messages.
    Allowing to accept a user provided (de)serializer is possible as well which should override the default JSON one. We will be accepting an implementation of the interface org.springframework.integration.aws.sqs.core.Amazo nSQSMessageTransformer

    The class org.springframework.integration.aws.sqs.core.Amazo nSQSMessage is a container for the various headers,payload and the original type of the payload received by the adapter. This instance needs to be serialized into a string value that is sent as the payload of the SQS queue and then deserialize it back to the AmazonSQSMessage instance.

    I'm not sure why you need to use your own interface to describe the AwsCredentials. AwsCredentials is already an interface for which there are multiple implementations. We have one that retrieves our credentials from a secure store, but can't use it with your library because it doesn't implement your interface
    The reason why AWS credentials are represented using the interface org.springframework.integration.aws.core.AmazonWSC redentials is because i didn't want any dependency for the end user on AWS SDK classes. For example, if we start allowing the user to provide an implementation of the AmazonSQSOperations, the implementation can very well be implemented using some propitiatory implementation of perhaps using the REST API but not using AWS SDK. In that case, just for the AWS credentials class we'll have to include the SDK classes. Can you give me some more details about the issue you are facing on this one. I am sure we can find some solution to it which may possibly need a new/custom implementation of the
    org.springframework.integration.aws.core.AmazonWSC redentials


    So to summarize,
    I can address your concern 2, on providing me with some details your concern 3 can also be addressed ( i'm hoping to )
    For the 1st concern, its a tough call. Eliminating the conversion service means effectively restricting the adapter to accept messages with string payload only. However, if that is the case, the adapter implementation gets simplified to a great extent eliminating the need of a conversion service and thus addressing your concern one. The problem is to decide a draw a line on what needs to be provided, or, is the responsibility of the end user and what needs to be handled by the adapter.

    Can you perhaps briefly give your use case that can help provide a better adapter implementation?
    Mark, Oleg, Gary, Gunnar
    Can you throw some light on this and share your thoughts/views on it? I'll need you'll help for sorting the issue faced by using the conversion service

  2. #12
    Join Date
    Apr 2012
    Posts
    3

    Default

    Hi Amol

    I did see the various interfaces and their default implementations. It's nice to be able to provide alternate implementations of those to plug in our own custom logic, but I don't see the point in providing interfaces to implement when there are already common interfaces that represent the various components. For instance, the Transformer interface already handles converting the payload from one type to another. But there are many implementations of the Transformer interface and other libraries are coded to deal with that interface, but not with your AmazonSQSMessageTransformer. Additionally, implementing your AmazonSQSMessageTransformer means if we want to plug in someone else's SNS library to route some messages to that channel, we have to implement the transformation logic twice. Meanwhile, there's a perfectly suitable JsonToObjectTransformer that is configurable and implements a common interface that all Spring Integration libraries will be able to deal with and can be plugged in anywhere. Similarly, we can implement your AmazonSQSOperations and add the logic to use the AmazonSQS client, but why bother implementing your interface when we could just implement the MessageHandler interface and reduce our dependencies? And you're correct that the ConversionService is common in the SpringIntegration framework, but everywhere, the conversion is invisible, rather than explicit. So my message endpoint can implement a method, processMessage(MyDataType data) and just registering a ConversionService that knows how to go from String to MyDataType is enough to make it all happen.

    I'll give you a brief overview of our use case- we're processing a number of feeds from various feed sources in the cloud. During feed validation and processing we enqueue messages to SQS for the processors/validators to pick up and process. We use an encrypting SQS client so that our data isn't transmitted in plain text and we use a custom implementation of the AWSCredentials interface that retrieves our credentials from a secure store (so we don't have to store our credentials in code/configuration and credential rotation can happen without deployment). Our implementations are also wrapped in retrying decorators so short network blips don't short circuit the entire processing. In the future we want to be able to route messages for different feeds to different channels, whether different SQS queues, SNS notifications or e-mails via SES so each of our customers can be notified of their own feeds in whatever manner is appropriate so their systems can, for instance, read messages from their queue or subscribe to an SNS channel for a feed they depend on, etc.

    Your library is very appealing as it provides two of the notification mechanisms and SNS seems like an obvious choice for a future extension. However, in your effort to decouple your library from the Spring abstractions and the AWS library, you're forcing users to couple themselves to you instead. Most of us are perfectly happy binding our systems to Spring code (we use @Component and @Autowired throughout our codebase knowing that Spring is ubiquitous within our systems). But coupling our components to your library isolates those components from the rest of the system, resulting in a lot of duplication. Spring is about simplification and pluggability. Don't force someone to implement interfaces when they can wire together loosely coupled components declaratively instead.

    Best of luck with your library.

    Mike

  3. #13
    Join Date
    Oct 2011
    Location
    Mumbai, India
    Posts
    213

    Default

    the Transformer interface already handles converting the payload from one type to another. But there are many implementations of the Transformer interface and other libraries are coded to deal with that interface, but not with your AmazonSQSMessageTransformer.
    The transformer here is used by the AmazonSQSOperationsImpl to transform the AmazonSQSMessage instance to a JSON string and back to the AmazonSQSMessage instance. Trouble with using the spring's transformer interface it has the following method

    Message<?> transform(Message<?> message);


    If i do need to use this transformer, then the transformation needs to be done before the message is being given to the AmazonSQSService. I will have a look at this one though and see if spring integration's transformers can be used eliminating the use of conversion services and avoid implementing the transformer interface of the framework. Will need to rework on the design a bit, will update you on that

    If you choose to provide your own implementation of AmazonSQSOperations, you might not even need to have a transformer as that implementation might be doing the conversion by itself

    Similarly, we can implement your AmazonSQSOperations and add the logic to use the AmazonSQS client, but why bother implementing your interface when we could just implement the MessageHandler interface and reduce our dependencies?
    If you have the MessageHandler handler implementation already in place, then the framework has nothing left in it. That is the reason why i feel providing a generic interface makes sense. Also, MessageHandler is used for outbound adapter, what for inbound channel adapter? The user then will have to also provide an implementation of MessageSource.

    And you're correct that the ConversionService is common in the SpringIntegration framework, but everywhere, the conversion is invisible, rather than explicit. So my message endpoint can implement a method, processMessage(MyDataType data) and just registering a ConversionService that knows how to go from String to MyDataType is enough to make it all happen.
    Isn't it transparent here? Nowhere, you need to explicitly use the conversion service. If an appropriate converter is found, your payload will be converted to string and vice versa transparently.

    Your library is very appealing as it provides two of the notification mechanisms and SNS seems like an obvious choice for a future extension. However, in your effort to decouple your library from the Spring abstractions and the AWS library, you're forcing users to couple themselves to you instead.
    Your application's other components do not depend on this interface. If you already have an implementation of the class that does these SQS operations for you, you just need to be implementing this interface and delegate calls to your implementation. You would then need to just define this in your config and pass the adapter a reference to it. If you choose not to use the library all you need you to do is delete this class and the corresponding configurations. This is IMHO

    SNS is definitely on the radar and am planning to include support for it, but i am deferring it till the existing adapter implementations become relatively stable.
    Last edited by Amol Nayak; Apr 5th, 2012 at 10:41 AM. Reason: Added missing details

  4. #14
    Join Date
    Apr 2012
    Posts
    3

    Default

    Hi Amol

    If you have the MessageHandler handler implementation already in place, then the framework has nothing left in it. That is the reason why i feel providing a generic interface makes sense.
    Here, I think, is where the discrepancy lies. It seems like you want to provide a framework for implementing Amazon SQS messages. But we don't want a framework. We've already chosen a framework with the appropriate abstractions: Spring Integration itself satisfies all of the abstractions and common interfaces. What we need are simple adapters to make SQS, SNS and SES available channels to send/receive messages using existing AWS APIs. Additionally, we need common logic to be shared between all channels, such as type transformations, filtering, etc. By adding another abstraction layer beneath Spring Integration for SQS message transformations, you're making it necessary to implement that transformation for every adapter we want to include.
    If you choose to provide your own implementation of AmazonSQSOperations, you might not even need to have a transformer as that implementation might be doing the conversion by itself
    Again, going this route, our Amazon SNS implementation would then have to implement the same conversion logic.
    Your application's other components do not depend on this interface. If you already have an implementation of the class that does these SQS operations for you, you just need to be implementing this interface and delegate calls to your implementation. You would then need to just define this in your config and pass the adapter a reference to it.
    Again, this is the exact same work that would be required to implement the MessageHandler and MessageSource interfaces. The difference being that the MessageHander and MessageSource interfaces are common interfaces in the Spring Integration framework, therefore plug in directly, giving us more flexibility and making the components cooperative with other components in the framework.

    Here is the configuration we would like to have in place.
    Code:
        <!-- inbound -->
        <aws-sqs:inbound-channel-adapter
            sqsClient="myClientImpl"
            channel="inboundChannel"
            sqs-queue="https://queue.amazonaws.com/myworkqueue"
            transactional="false"
            max-redelivery-attempts="3">
            <int:json-to-object-transformer type="com.foo.MyDataClass" object-mapper="objectMapper"/>
            <int:transformer expression="payload.dataField" />
            <int:filter ref="myMessageFilterBean" discard-channel="notReadyChannel" />
            
            <int:service-activator ref="dataProcessor" method="processData" />
        </aws-sqs:inbound-channel-adapter>
    
        <!-- outbound -->
        <int:chain output-channel="myCustomerOutput" id="customerOutputChannel">
            <int:transformer ref="myCustomersCustomDataTransformer" />
            <aws-sqs:outbound-channel-adapter
    		sqsClient="myClientImpl"
    		channel="inboundChannel"
    		sqs-queue="https://queue.amazonaws.com/myCustomersQueue" />
        </int:chain>
    Here we have the SQS messaging set up as two message chains, including filters and transformers. Note that the input channel uses a different transformer from the output channel. The difference is that we produce the input messages being consumed here, therefore always use the same message format. Our customers, however, have different serialization formats- some use JSON, but other use a binary serialization format. The goal is to make it easy to route the right messages to the right customer queue using the right transformer. As mentioned, some messages will go out via SQS, but others will use SNS or SES. Message transformers are common among several output channels, so it does not make sense to implement the conversion in the SQS Operations impl, then implement it again in SNS Operations Impl. Additionally, any new message solutions that might be added in the future (e.g., AMQP, etc.) should also be able to utilize the existing tools.

    Hope this helps to clarify the issues for you.

    Mike

  5. #15
    Join Date
    May 2012
    Posts
    3

    Default

    Hi Amol,

    We are not able to use the sqs adapter for queues which are in different endpoint than queue.amazonaws.com
    The reason being amazon sqs client sets as default endpoint queue.amazonaws.com and then ignores the endping information from the queue urls.

    May be adding an attribute to adapter xml for specifying the endpoint would be useful.

    This is very useful library, thanks a lot
    Bilgin

  6. #16
    Join Date
    Oct 2011
    Location
    Mumbai, India
    Posts
    213

    Default

    Hello Bilgin,
    Thanks for evaluating the library. Can you perhaps give me more specific information about the queue endpoints? Do you mean to say the endpoints that are AWS region specific?
    We are in a process of enhancing these adapters for usability purpose and feedback from the end users is very useful indeed. Will keep you updated in it.

    Cheers
    Amol.

  7. #17
    Join Date
    May 2012
    Posts
    3

    Default

    Yes, exactly. For example it is not possible to use a queue with url like https://eu-west-1.queue.amazonaws.com/477...

    Bilgin

  8. #18
    Join Date
    Oct 2011
    Location
    Mumbai, India
    Posts
    213

    Default

    Ok, I'll take a look at this one and have a fix for this one included

  9. #19
    Join Date
    Oct 2011
    Location
    Mumbai, India
    Posts
    213

    Default

    Hello Bilgin,
    I have resolved the issue you mentioned about using the region specific URL. Please take the latest from here and try if it works. Thanks for bringing this up.

    Cheers,
    Amol.

  10. #20
    Join Date
    May 2012
    Posts
    3

    Default

    Hi Amol,

    I noticed another bug in s3 outbound adapter:

    User metadata (which can be set to the message with header name user_meta_data) is actually never set to PutObjectRequest, so it is not possible to add any user specific meta data.

    Bilgin

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •