Scheduling messages with custom priority and custom algo

[rock_star]

Hi Guys,

I have a scenario like --

I have 100 messages coming on queue Q1. I have configured a Message Driven Adaptor on queue Q1, which feeds the message to a router.
Router stamps the message priority and routes to two queues High-Q, Low-Q for high and low priority queues.

Let us say router classified them as 60 higher priority, rest 40 Low priority as per some rules and hence, sent to High-Q, Low-Q respectively.

Now I want to have JMS listeners to High-Q and Low-Q, so that they put the messages on a final output queue Q-final such that High-Q messages are always taken off first and processed.


Questions:

1. People will say that why not use priorty queue and put the output of route to priority-channel or queue. I would be happy but
   
   • Let us take a case, when message producers send messages on Queue Q1, in a pattern like 10 messages which will be classified by router as High-Proority and send 5 which will be classified as low-priorty kind, at time T1 and then another 200 messages which will be of type High-Q at T1 + 5 milliseocond. In this case, the final consumer might end up taking lower priority 5 ahead of 200 messages. I want to avoid this for some good reasons.
   
   
   • Second, is I can specify priority channel size as 5 but I dont want to loose message in case of server crash, as I think the messages in priorty-channel are in memory.

1. Is there a possibility that I can customize the message listeners on queues High-Q and Low-Q so that they can cooridnate that low-queue listeners take off message only when High-Q has nothing to do and with a added delay.

Also, I am not keen to use pollers as there is polling interval which essentially adds to latency. Not sure if I have no other option but to use poller.

Please let me know if I am thinking right or there is already something in SI which I can use.

Any pointer will be greatly appreciated.

[oleg.zhurakousky]

Priority channel is the right solution because it guarantees that if message with the higher priority is present , then release it before the message with the lower priority. Yes that does mean that if at the given time queue the only messages on such channel are the messages with the lower priority they will be released even if messages with higher priority will come later. If this is not OK, then i would rethink the entire strategy since you would have to take a batch of messages, split them in high/low and send high before low.

Also, I am not keen to use pollers as there is polling interval which essentially adds to latency

Poller don't add latency, you do
Seriously, it really up to the poller configuration. You can configure very small interval trigger (1 - 10) with high number of max-messages-per-poll. . . .
The bottom line is that poller allow you to throttle the load where messages might be coming into the system at one rate, but you can now process then at your own pace.

Also, you were mentioning that you don't want to loose messages on the queue during the server crash. For that you can simply inject MessageStore to the queue channel via message-store attribute.

[rock_star]

Thanks Oleg. Yes, I am worried about the situation of releasing low priority when high priority can come after fraction of milliseconds. This is one of the situation that I am trying to deal with. And, splitting the messages to how/low is the option that I am working as POC.

Basically, I am doing POC on the lines that I will have two queues high, low and then release messages from high first. And, before I take off messages from low queue I will check the High queue again if something has arrived. If nothing arrived then release the low priority message if there is anything then do not release.
I think to achieve this I will have to customize the message listener, No? Or something else?

Yes, I thought of poller interval of 1 milliseocnd as that would not be much latency but that again depends on volume.

MessageStore- I skimmed through SI M7 manual, does this need to be configured to store the message to Database( this is what I could make out ) Or there are some other options? I would imgaine this store to be persistent somewhere, no?

[oleg.zhurakousky]

Lets compare your statements:

I am worried about the situation of releasing low priority when high priority can come after fraction of milliseconds

vs.

before I take off messages from low queue I will check the High queue again if something has arrived. If nothing arrived then release the low priority message

The first one describes the situation where you have a batch of 100 messages and you know it but somehow some high-priority messages were delayed, but you know they are there.

The second one describes the semantics of PriorityQueue. In other words the two statements contradict one another.

MessageStore is a strategy that would allow you to provide a persistent store for any component that has capability to buffer messages (e.g, queue channel, aggregator etc.). THis way the messages will be written to such store on arrival and read into on the start of the AC providing guarantee that messages will not be lost (similar to the way JMS queue backed by the db)
We have JdbcMessageStore which points to the dataSource configuration, but you can also provide your own implementation

[Mark Fisher]

As far as latency is concerned, the best option for a poller that makes calls to a blocking queue is to use a very short interval between polls but a long receive timeout. That is generally described as "long polling" and it provides the illusion of being event-driven while also avoiding busy spinning.

[rock_star]

Oleg,

The first scenario is not a definite but could be one of the scenario that can happen at any time in the system. And it could be 100 messages or 10. And, this is not gurantteed to happen at some known time.

Second, part-- I was trying to put a solution for the first. The reason that priority queue does not solve my case completely. I will have priorityQueue of certain size lets say 20. But with High and Low queues( tibs) I can use the router to stamp and put the message on them. I know same can be achieved with priortyQueue of unlimited size but with a messageStore on Db( which team-people think is not a nice idea ).

Let me add a bit. Why did I thought of two queues which will store messages of two different priorities.

A source queue Q1 have multiple producers say 10 and each of them can send a message at any time, some may send higher and some few but all of them send to queue Q1.

Now, my component will read them off queue Q1 and decides which should be processed first and which should be put on hold, lets say by stamping priorities High and Low.

1. If i use priority queue with capacity 20 then there is very much possible case that consumer of priority queue will take low priority messages of the priority channel/queue even though there are many messages left on queue Q1( which could be classified as High priority ). So the prioirtyQueue helps me but not 100% of the cases

2. I thought why not have a router, which classify the priority of messages and put them on two separate queues named High and Low.

3. Then rest of flow as I mentioned in my previous message.

Please let me know If I am clear now.

On persistent store, I think I will make High Queue as Tib queues. Mark Thanks for hint.

[rock_star]

May be I am being nuts on begining of the week. I will have to see that essentially the Tibco queue can act as priority queue with big persistent storage. I will customize the poller to pick the message of higher priority and if the next message in queue is of Low priority then I will wait before taking it off queue.