MBean remoting causes connection reset

I am attempting to get JSR-160 remote MBeans working over RMI in an existing piece of code, but I am having little success.

Here is the scenerio. We have a JBoss implementation using their MBean server and their custom RMI remoting for MBeans. I would like to start conversion away from the JBoss-specific stuff. The portion of code that calls the remote MBean server is being re-written, using Spring heavily, so I see this as an opportunity to start a switch to the more standard JSR-160 way of doing business.

What I have done is implemented a Spring context within the old app, adding the following bean definitions:

Code:

<bean id="registry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean"> <property name="port" value="1099"/> </bean> <bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean" /> <bean id="serverConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean"> <property name="server"> <ref local="mbeanServer"/> </property> <property name="objectName" value="connector:name=rmi"/> <property name="serviceUrl" value="service:jmx:rmi://localhost/jndi/rmi://localhost:1099/myconnector"/> </bean>

This "seems" to work ok, or at least I dont get an error. I then wrote unit test class that attempts to grab a reference to a remote MBean using the proxy generated MBeanProxyFactory as such

Code:

<bean id="mbeanServerConnection" class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean"> <property name="serviceUrl"> <value>service:jmx:rmi://localhost/jndi/rmi://localhost:1099/myconnector</value> </property> </bean> <bean id="proxy" class="org.springframework.jmx.access.MBeanProxyFactoryBean"> <property name="objectName"> <value>ESM:service=monitor,type=Central Application Monitor</value> </property> <property name="proxyInterface"> <value>com.xxx.remote.monitor.ApplicationMonitorMBean</value> </property> <property name="server"> <ref local="mbeanServerConnection"/> </property> </bean>

And the code to grab the proxy

Code:

ApplicationMonitorMBean mbean = (ApplicationMonitorMBean)ctx.getBean("proxy"); System.out.println("MBean: "+mbean);

All looks good so far, but when I try to run the JUnit, I get a big nasty error:

Code:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mbeanServerConnection' defined in file [C:\dev\Projects\Work\virtualHub\apps\node\src\conf\spring\spring_soapclient_example.xml]: Initialization of bean failed; nested exception is java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset] java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:317) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.connect(MBeanServerConnectionFactoryBean.java:146) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.afterPropertiesSet(MBeanServerConnectionFactoryBean.java:114) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1059) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:363) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:226) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:147) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:269) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:320) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:89) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:74) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:65) at com.xxx.node.test.SimpleTaskExecution.setUp(SimpleTaskExecution.java:57) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:436) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:311) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:110) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203) at javax.naming.InitialContext.lookup(InitialContext.java:361) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1807) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1777) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259) ... 23 more

Searching the web and this forum for info has turned a bust, so I now turn to the wisdom of my peers.

Thanks in advance for any help, and please let me know if there is any other information you might need.

Can you change the protocol and rerun the tests - it appears that the communication breaks at some point (maybe the server is not started)?
Turn on logging and see what is going on - see if the server maintains the communication.

Well, I attempted to do some work to use MX4J's SOAP JSR-160 implementation in JBoss, but that looks like it wont happen. Apparently there are some clashes between the MX4J and the JBoss.

So, I started looking at IIOP instead of RMI. In searching though my jboss-services.xml, I noticed port 1099 was JNDI, not RMI which is set at 1098, so I tried switching my configs to use the 1098. The Connection Reset has dissapeared, but I got another error:

Code:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mbeanServerConnection' defined in file [C:\dev\Projects\Work\virtualHub\apps\node\src\conf\spring\spring_soapclient_example.xml]: Initialization of bean failed; nested exception is java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table] java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:317) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.connect(MBeanServerConnectionFactoryBean.java:146) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.afterPropertiesSet(MBeanServerConnectionFactoryBean.java:114) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1059) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:363) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:226) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:147) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:269) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:320) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:89) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:74) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:65) at com.xxx.node.test.SimpleTaskExecution.setUp(SimpleTaskExecution.java:58) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:436) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:311) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) Caused by: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:110) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203) at javax.naming.InitialContext.lookup(InitialContext.java:361) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1807) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1777) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259) ... 23 more

This looks way more promissing, but I still can't get it right. The error doesn't seem to be associated with the actual proxy, as I still get it when I remove the bean definition for the proxy in the Spring XML file.

My new server XML looks like:

Code:

<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean" /> <bean id="serverConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean"> <property name="server"> <ref local="mbeanServer"/> </property> <property name="objectName" value="connector:name=rmi"/> <property name="serviceUrl" value="service:jmx:rmi://localhost/jndi/rmi://localhost:1098/myconnector"/> </bean>

And the client is:

Code:

<bean id="mbeanServerConnection" class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean"> <property name="serviceUrl"> <value>service:jmx:rmi://localhost/jndi/rmi://localhost:1098/myconnector</value> </property> </bean> 

I am in the process of trying IIOP again, but I am not 100% sure what port to send it too. I feel like I am just missing something, like a required JAR or a setting or something.

I haven't worked in your environment but IIRC, I've seen some topics about mx4j remoting not working with jboss some time ago. Again, I haven't investigated the issue - this may be a real issue or one solved already but you should find more details on the mx4j site - simone is very helpful.
Btw, you can try using the JMX-RI jars - you shouldn't have any problems with that one.. I think.

I'm sorry, I thought I mentioned it, but I ditched MX4J because I found the same thing...incompatability with JBoss. The above error was using Sun's reference implementation of JSR-160 (jmx-remote.jar). The error in the last post was what happenes when I try RMI. If I switch to IIOP I get:

Code:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mbeanServerConnection' defined in file [C:\dev\Projects\Work\virtualHub\apps\node\src\conf\spring\spring_soapclient_example.xml]: Initialization of bean failed; nested exception is java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ConfigurationException: COS Name Service not registered with ORB under the name 'NameService' [Root exception is org.omg.CORBA.ORBPackage.InvalidName: NameService:org.omg.CORBA.COMM_FAILURE: purge_calls:1451 reason=1 state=5 vmcid: IBM minor code: 306 completed: Maybe] java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ConfigurationException: COS Name Service not registered with ORB under the name 'NameService' [Root exception is org.omg.CORBA.ORBPackage.InvalidName: NameService:org.omg.CORBA.COMM_FAILURE: purge_calls:1451 reason=1 state=5 vmcid: IBM minor code: 306 completed: Maybe] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:317) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.connect(MBeanServerConnectionFactoryBean.java:146) at org.springframework.jmx.support.MBeanServerConnectionFactoryBean.afterPropertiesSet(MBeanServerConnectionFactoryBean.java:114) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1059) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:363) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:226) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:147) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:269) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:320) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:89) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:74) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:65) at com.bofa.esm.node.test.SimpleTaskExecution.setUp(SimpleTaskExecution.java:58) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:436) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:311) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) Caused by: javax.naming.ConfigurationException: COS Name Service not registered with ORB under the name 'NameService' [Root exception is org.omg.CORBA.ORBPackage.InvalidName: NameService:org.omg.CORBA.COMM_FAILURE: purge_calls:1451 reason=1 state=5 vmcid: IBM minor code: 306 completed: Maybe] at com.sun.jndi.cosnaming.CNCtx.setOrbAndRootContext(CNCtx.java:403) at com.sun.jndi.cosnaming.CNCtx.initUsingIiopUrl(CNCtx.java:330) at com.sun.jndi.cosnaming.CNCtx.initUsingUrl(CNCtx.java:285) at com.sun.jndi.cosnaming.CNCtx.createUsingURL(CNCtx.java:100) at com.sun.jndi.url.iiop.iiopURLContextFactory.getUsingURLIgnoreRest(iiopURLContextFactory.java:73) at com.sun.jndi.url.iiop.iiopURLContext.getRootURLContext(iiopURLContext.java:61) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:200) at javax.naming.InitialContext.lookup(InitialContext.java:361) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1807) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1777) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:259) ... 23 more

I could be wrong, but I am interpreting these two errors as basically saying the same thing: Whatever object its looking for on the remote side is not registered. My question is, what IS it looking for.

So, I thought about this, and it "seems" like the Spring file points to something being registered as "connector:name=rmi", as per the ObjectName property. I started poking around JMX Console on JBoss looking for it, but could not find anything. This could explain why all subsequent calls from the client come back with "name not in table". Am I interpretting this correctly?

My next question goes to spring. My understanding is that putting the MBeanServerFactoryBean and ConnectorServerFactoryBean definitions in my spring.xml file, and bootstrapping that file somehow, will "magically" create the necessary infrestructure to get the MBean remoting to work. I currently bootstrap my Spring environment using an MBean that is set to launch at start, and I have absolutely confirmed that the file is at least working partially.

Is there some other piece I am missing here? Do I need to actually get reference to the for mentioned spring beans to get them registered correctly?

I see two issues with your configuration:

1. You are using not jboss mbean server but one you are creating by yourself through Spring. If you want to use Jboss mbean server then you can use some of the classes inside Jboss and Spring which can detect the mbean server id and pass this to Spring mechanism - search this forum for more information and examples.

2. No matter what mbean server you are using the connector should be already registered. Make sure your beans are initialized at startup by marking them as non-lazy and turn on logging to see that the connector is in fact attached. You can do a simple test (inside your unit tests) and create your own mbean server and then connect to it locally without jboss or anything else. AFAIK, there are some similar tests inside the Spring distribution. MX4j might help here not just by implementation but through documentation since it provides some nice examples on how to set up your rmi registry and so on.

First of all, thanks for all the help thus far...I honestly feel that I am close to something...what that something is I am not sure of yet.

I have some confusion. In reading the docs online, and the Pro Spring book by APress, I was under the assumption that

Code:

<bean id="mbeanServer" class="org.springframework.jmx.support.MBeanServerFactoryBean" />

line in the config was the "magic bullet" that grabs ahold of the MBean server, or creates on if it can't find it. Your comment now makes me think I misinterpreted this, however. In searching on the forums I honestly found nothing explaining how to instruct Spring to use JBoss's own MBean server. There are several talks discussing JBoss 4.0.x, but nothing in relation to using the MBean server with remoting, and nothing on 3.2.5, which is the version I am stuck with.

At this point I am considering some other options. The basic problem is this:

We have an old piece of code that is an MBean that make heavy use of the JMX Notification object (javax.management.Notification) and uses JBoss's own remote MBean stuff (org.jboss.jmx.adaptor.rmi.RMIAdaptor) to allow remote calls to this MBean. There is also some early Spring-ized code in the existing codebase, which is where I am placing the "server" code I pasted above.

The code that is calling this remote MBean is going through a transistion...basically we are getting rid of JBoss, and moving to a more light-weight SOAP implementation using embedded Tomcat and Apache Axis, an of course Spring to tie everything together.

Our environment is large enough that just converting everything at once is totally out of the question. So, the thought was to use JSR-160, or some other remotting technology, to connect to the remote MBean for the new code place of the JBoss-specific code currently in place now. This way, the old stuff can work over the JBoss-specific route and the new stuff can work over the route, which allows us to start phasing the old code out.

What I have control over and/or can alter: The "new" connection protocol and code surrounding, the Spring config file on the server, and any "new" code needed on the server (as long as existing code is left untouched).

What I have no control over and/or cannot touch: The "old" connection protocol and code surrounding, the heavy usage of the JMX Notification object (which is why I didn't try SOAP first) or any other "supporting" objects, or the JBoss server version (currently 3.2.5).

I am totally open to ANY suggestions at this point.

MBeanExporter is the silver bullet (not quite magical) - which either uses the given mbean server or auto-detects one. The MBeanServerFactoryBean creates an mbean server no matter if there is already one. I think this part is explained by the "Exporting your Beans to JMX" chapter.
To make sure you use an already defined jmx server you can use the advices present on this thread: http://forum.springframework.org/sho...t=20760&page=2
IIRC you can give a name to the mbean server created by jboss through some configuration file and I think the Jboss JMX util classes mentioned there are also available in the 3.2.x branch.
I haven't worked with Jboss for quite a while but AFAIK, the 3.2.x branch uses an old jmx specification with some specific jboss extensions in it. You can find more about it by looking into their jar, tests and documentation.
Considering your environment, you can start by finding out exactly what jmx version your server support especially in regards with jsr-160 (jmx remoting) which at that time I think was not fully release or just an initial version. You should probably use only RMI even though it might worth trying to get mx4j to get with jboss since you can then use SOAP, Hessian and Burlap transparent and a lot more reliable then RMI.
In the mean time make some simple tests just with Spring and either JMX-RI or mx4j support to make sure your configuration is correct and then try creating a remote connector on Jboss side either through Spring or JBoss directly.