Starting Multiple Consumers

[tsure]

I have been wondering about what would be the best way to go about implementing creating multiple consumers with Spring Integration Framework.

So to elaborate on the question, I am more interested in scaling the consumers that are consuming messages from a RabbitMQ broker. In the amqp example that I could run, there is a producer and a consumer. Now lets say, I am just interested in the consumer part of it. That just makes up one consumer with a context.xml file where I can define the wiring required for that consumer. What if I have the need to incrementally add more consumers as and when needed from my application.

Since I am a newbie, if I think of the layman approach, it would be having multiple context files for each consumer, but in that case I would need to know the number of consumers before hand which I do not know for my application. My application should be able to add consumers by mere clicks. Is this possible with Spring Integration? I am stuck since in the approach that I plan to take I would have to write the configuration for each consumer separately which I is not what I am looking for.

Any thoughts or suggestions ??

Thanks.

[Mark Fisher]

If you simply want more consumers within a single consumer-side process, then you can provide the 'concurrent-consumers' on your inbound adapter (e.g. amqp:inbound-channel-adapter).

[tsure]

Thanks for the quick reply Mark. I did look at concurrent-consumers attribute as an option but then all my consumers would be getting their messages from one single adapter. What if I want to shut down one of the consumer out of all I have? I don't think that would be possible with this, since currently I have modeled a single consumer with the rabbitmq adapter and if i use concurrent-consumer for that adapter, and then with my control bus shutdown the rabbit adapter, that would imply all the remaining consumers on that adapter would also not receive message. I want to be able to scale and have full control over each consumer.

[Mark Fisher]

Well, you can create multiple adapters that listen to the same queue(s). If you want each of those to have a single consumer, that's the default.

That said, can you describe your rationale a bit more?

[tsure]

Yes exactly what my current option is. But here is why it does not work for me. So I want to create a monitoring and management configuration such that I should be able to start/pause/resume and stop any consumer as and when required. Also, I require that I should be able to add consumers or in other words want to scale the consumers though my application. Now lets say, there application is a simple GUI that has a button to add consumers. Now if I need control over each of them, I need to create consumers the way you replied in the post above ( the default way each having their own adapter) But now this is not possible since the wiring would still need to be hard coded correct? I mean the xml will not get generated on the fly by mere click of add consumer. I will have to manually add these consumers each having their own rabbit adapter listening to the queue for messages eventually leading me to hard code the configuration/wiring for each consumer.

Let me know if this is still not clear enough and I can may be try and explain better. Thanks and your help is greatly appreciated !

[Tomcruise]

If you want to consume concurrently from a queue, then you must use a different session for each consumer.
This is because you must have a session per thread. The JMS contract is that only 1 session is used by one thread at once - which if you're using consumers means that only 1 consumer can receive messages at once if using the same session. So if you want concurrent consumption of messages, you need to use different sessions for each consumer.
However its your call on how many connections you have. In a normal network based mode, sharing the same connection across sessions will essentially multiplex traffic across a single socket (and so reduce resource usage at the cost of more synchronization). So sometimes there's a need to use more connections, as typically using different sockets concurrently tends to be a bit faster than using a single one).
Incidentally in the JCA specifications in J2EE 1.4, Resource Adapters will typically create 1 connection for each session, so there's a 1-1 mapping between the two.

[Gary Russell]

I you want to control the life cycle of multiple consumers in the way you describe, one technique would be to use parent/child application contexts; the main (root) application would contain the bulk of your integration flow, it would start with a channel. Declare the inbound adapter in a separate application context and, when you create it, set the main context as the parent. This will give the child context access to the beans in the parent context, including the inbound channel, rabbit infrastructure, etc. The inbound contexts can then send messages into the main context.

Now, you can create/start/stop/destroy child contexts as you please, with each one having a separate listener container/consumer.

[tsure]

Ok. This looks like something I can start looking into...I do not have much experience with parent/child application context since I never had the need to use one, but I will dig into it and see what I can extract. Do you have any particular samples/links that you think may be helpful to me?

[Gary Russell]

There's a sample here https://github.com/SpringSource/spri...ed/dynamic-ftp that dynamically creates application contexts on the outbound side (for dynamic ftp). However, it doesn't need parent/child contexts. For that, just use this constructor...

	/**
	 * Create a new ClassPathXmlApplicationContext with the given parent,
	 * loading the definitions from the given XML files and automatically
	 * refreshing the context.
	 * @param configLocations array of resource locations
	 * @param parent the parent context
	 * @throws BeansException if context creation failed
	 */
	public ClassPathXmlApplicationContext(String[] configLocations, ApplicationContext parent) throws BeansException {
		this(configLocations, true, parent);
	}

Just remember that beans in the child context can reference beans in the parent, but not vice versa.

[tsure]

Thanks a lot Gary. Looking into this ..