Achieving fast consumption rate

[stephanion2002]

Hi,

With RabbitMQ and Spring AMQP, I have to achieve is a faster consumption of each message. The scenario is like this:

I have to send emails/SMSes to different people after consuming the messages internally. I want to just post a message and consume it faster by having multiple consumers ready. The message should be consumed only by one consumer…

Should I make a no. of consumers using a shared rabbitmq queue in which each consumer consumes the message posted on the exchange, deletes it from the queue and then after passes that message to a new thread for processing ?

But having a single queue might hamper the performance.

Is there any other way which can cater my requirement ? any suggestions?

Regards,
Kshitiz

[Mark Fisher]

You should probably begin by looking at the support for "Message-driven POJOs" that is part of Spring AMQP. It is modeled after Spring's JMS support and provides the same general options for supporting multiple consuming threads per actual Consumer instance.

Here's a quick link:
http://static.springsource.org/sprin...single/#d0e288

You will want to explore the various properties that can be configured on the MessageListener container.

[stephanion2002]

Thanks Mark,

I am using a message Handler which is just a POJO, thanks to Spring AMQP , but now I am facing a situation. I want to hold acknowledging the message so that it stays live on the rabbitMQ till my POJO's handle message gets completed successfully.

On rabbitMQ mailing list , I have been told to consume with no_ack=false and then send a basic.ack after the processing is complete. How can we achieve that from this POJO or its configuration ?

Regards,
Kshitiz Garg

[Dave Syer]

We have transactions covered fully in the RabbitTemplate now, but the asynch use case (message listeners) is not finished yet. However, I think if you set autoAck=false in the message listener container you should see the acks going after the POJO listener has completed.

[stephanion2002]

Thanks Dave, I will certainly try that, please let me know later if u come to know about its confirmation

[stephanion2002]

Hi,

Is there any provision that even if I use a POJO message Handler, the queue name property of the associated listener is inserted programatically?

Regards,
Kshitiz Garg

[stephanion2002]

You should probably begin by looking at the support for "Message-driven POJOs" that is part of Spring AMQP. It is modeled after Spring's JMS support and provides the same general options for supporting multiple consuming threads per actual Consumer instance.

Here's a quick link:
http://static.springsource.org/sprin...single/#d0e288

You will want to explore the various properties that can be configured on the MessageListener container.

Hi Mark,

we have a message handler POJO associated with a listener container, If we want different instances/thread of this POJO to handle each new message , is it an automatic process?
if not, what option I should set on the associated listener..

[Mark Fisher]

The listener container has a 'concurrentConsumers' property. Here's the JavaDoc from that setter:

/**
 * Specify the number of concurrent consumers to create. Default is 1.
 * <p>
 * Raising the number of concurrent consumers is recommendable in order to
 * scale the consumption of messages coming in from a queue. However, note
 * that any ordering guarantees are lost once multiple consumers are
 * registered. In general, stick with 1 consumer for low-volume queues.
 */

[stephanion2002]

Thanks a lot mark, that's exactly the thing I am looking for as the order of messages is not important and that will enhance the consumption rate ....

Now I have 20 exchanges, and 4 output queues having 20 concurrent consumers consuming messages coming from each of these 4 queues. Do you think that this mismatch between no. of exchanges and queues, (exchanges are many and queues are few) is not good?

Actually initially I thought that I will make 4 different type of queues(email/http/sms/ftp) and 4 different types of handlers(email/http/sms/ftp) for each one of those 20 exchanges....

This design is trying to scale the no. of POJO handler instances vertically , i.e. across different queues (instead of horizontal scaling in one queue, achieved through concurrentConsumers property).

But it seems that this design is not possible as we bind the handlers with queues while configuring the associated listeners beforehand......Can you please suggest any configuration/design changes to achieve this design, if it's possible?

Regards,
Kshitiz Garg

[Mark Fisher]

Can you explain the "bindings" you have set up between those Exchanges and Queues?

By the way, are you using Spring Integration for the various adapters (mail, ftp, etc)?