Performance Improvements

[mural]

Hello,

We are using Spring-Integration heavily, and while profiling our application we found a couple of improvements that will increase the overall performance of the framework. In our particular case, we jumped from 16K msgs/sec to 170K msgs/sec (1000% performance gain) which is a huge gain.

Some of critical points we found and that can be improved
- Message ID: UUID generation is really slow and in addition, UUID.randomUUID() is synchronized, which makes message instantiation a bottleneck. Making the ID Generation mechanism pluggable would enable the implementation of different strategies for ID generation. IMHO, there is no need to force the Message ID to be an UUID, wouldn't Serializable be enough?
- MessageHistory: MessageHistory is a feature that we consider very useful for testing environments, but we won't use it in production. It would be nice if it could be turned off.
- AbstractMessageChannel#convertPayloadIfNecessary was another HotSpot. Checking datatype is costly and by default Object is the base type, which makes the check really unnecessary. Maybe, different strategies could be used depending of the current "datatype" array value.
- Type checking using iAssignableFrom (in MessageHeaders) was another HotSpot. Here, we just decided to make a cast and avoid doing the call to that method.

There are other improvements we have made, one of them is already in JIRA: https://jira.springsource.org/browse/INT-1141. Some others have less impact so I left them out of this thread.

What do you think about moving some of these improvements into the framework?

Best,
Diego

[iwein]

Pure gold this. Each point is worth it's own JIRA. These should make it in the next milestone imo.

[Mark Fisher]

I don't think we'll want to move back to non-UUIDs. For one thing, we need to enforce unique IDs for an increasing range of reasons, and secondly, since Messages can be created via constructor calls at runtime, an IdGenerator strategy doesn't really fit (would require some kind of MessageFactory).

UUID.randomUUID() is not itself synchronized actually. It does invoke a single synchronized method internally (nextBytes() on SecureRandom).

I have a couple questions regarding this:
1) Do you have some benchmark test results for the randomUUID() invocation itself?
2) You mentioned that you had major performance increases after making the changes (which of course I'm very interested in), so I'm wondering what you did specifically for the ID generation case (given the 2 limitations I mentioned above).

Thanks very much for this valuable info!
-Mark

[mural]

Hi,

The main bottleneck was synchronization inside the randomUUID method (SecureRandom as you mentioned). Our testing app is using 13 threads, which publish messages in different channels (all in one VM) and we are testing on a Xeon 5620 (2cpu 4 core, 16 threads).

We have not tested UUID generation explicitly, but just removing it caused the initial jump from 16k to 120k (from the top of my head). When we were profiling the app, we had noticed there was a monitor contention, we found later it was caused by ID generation.

I am not saying to remove UIDs as default ID Type for messages. I just want to be able to plug a different implementation (of any type) because in some environments UUID is not strictly necessary.

We were thinking of a way to generate IDs using other mechanism and plug that in the MessageHeader or in some new intermediate class (MessageIDGeneratorProvider) that could be configured using Spring. I am not an expert on Spring but I believe it is possible to add a bean (PostProcessor) that could configure this static class with the proper implementation after application context is loaded.

Thanks!
Diego

[Mark Fisher]

in some environments UUID is not strictly necessary.

In 2.0, we are building a number of features for which UUIDs really are necessary. For example, we are supporting Aggregation with a backing MessageStore implementation. If we cannot be assured that IDs are unique in that MessageStore, then the Aggregator functionality would be easily broken. Therefore, the interface itself has been designed to expect UUID-based keys. This is extremely important as we consider the portability of applications built with Spring Integration from a single developer machine to a data-grid-backed cloud deployment. I am a firm believer in "make it right, then make it fast", and I do not want to sacrifice correctness for performance. That said, what I do want to explore is any alternative to UUID.randomUUID() where we can possibly find significant performance improvements by avoiding synchronization while still ensuring uniqueness in an environment where concurrent throughput may exceed the clock resolution.

[Dave Syer]

I think it might have less to do with synchronizing in UUID (SecureRandom) and more to do with what happens inside the synchronized method. On a Linux platform the default SecureRandom actually goes out to the OS to /dev/random and does a blocking call there.

I did some experiments with MessageBuilder, and if I artificially switch off ID generation I can create 180K messages/second (comparable with the figure above on a similar platform, but with no message processing at all). I doesn't depend much on the number of threads once the system is saturated.

With the default UUID.random() I get 60K messages/second, so only a factor of 3 worse and quite a lot faster than anything else could happen in my system. If I change the SecureRandom algorithm from native (the default on Linux) to SHA1PRNG I can get 120K messages/second, not as fast as not generating ids at all, but as Mark pointed out: we need UUID ids for enough use cases that it would be a bad idea to disable them completely. I'm guessing, but I didn't have time to confirm, that 60K is probably close to the limit for all Linux platforms with native random UUIDs, while 120K might be beaten on a faster system than mine using a custom generator.

I'm guessing the additional factor of 4 performance penalty described in the original post can be ascribed to the other (more optional) stuff like message history etc.

[mural]

We didn't go to low in the stack, we just detected it was slow and additionally it was synchronized, so it was the major bottleneck. In fact, it causes the throughput to decrease when more threads are added.

I've just run a simple Test of UUID generation, which shows this behavior. The tests were run in the same box I mentioned in my previous post.

Threads | Total UUIDs generated 
------------------------------
1       |  249400
2       |  207798
3       |  176305
4       |  164070
5       |  161442
10      |  158110

Diego

[mural]

Mark, I actually agree with you, correctness before performance, but I don't see why you think using a different interface for IDs is not correct. I guess proper documentation of what an ID should be, and the uniqueness that is required should be enough. If a developer changes the default ID implementation from UUID (which would be the default) to anything else, he/she should ensure that ID requirements are met by the new implementation and that proper "translators" are provided to the framework in order to allow usage of MessageStores, or any other functionality which requires the actual representation of the ID.

Nevertheless, I guess I won't be able to convince you ... so, if it is not possible to relax the ID type requirement, it would be useful if the framework allows plugging the UUID factory. Is this something you are willing to consider?

Best,
Diego

[Mark Fisher]

We're still investigating some alternatives for UUID-generation.

Let me be clear about the reason I want to require the actual UUID type though. The Message ID is used by strategy interfaces where IDs *must* be unique if we are to guarantee proper functionality of the framework (e.g. MessageStore), and in terms of "documenting" the requirement to be a unique ID, it seems that having the declared type as UUID is in fact the best way to do that. Anything less would be risky IMO. Of course, there are alternative approaches to generating a UUID, and that's what we're exploring now.

In terms of a pluggable UUID "factory", it's a bit tricky since these calls are made in the constructor of MessageHeaders, i.e. not in a place where there is access to a Spring-managed bean.

[justinmiller]

I'm running into the same issue as the OP in terms of UUID (really SecureRandom) causing a lot of lock contention.

Nearly a year later, have there been any updates to this?

Is it possible, without recompiling the spring integration source, to change the default generation algorithm to "SHA1PRNG" like Dave did?

Thanks,
Justin