OtpIOException: failed to connect from 'rabbit-spring-monitor' to peer node

[digitalsanctum]

I'm working on some code to send heartbeat messages from consumers. Getting the following exception:

org.springframework.erlang.OtpIOException: failed to connect from 'rabbit-spring-monitor' to peer node 'rabbit@localhost'
	at org.springframework.erlang.connection.SimpleConnectionFactory.createConnection(SimpleConnectionFactory.java:65)
	at org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangAccessor.java:43)
	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:114)
	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:46)
	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:98)
	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getQueues(RabbitBrokerAdmin.java:148)
	at spg.poly.consumer.monitoring.HeartbeatPublisher.sendHeartbeat(HeartbeatPublisher.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
	at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)
	at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: Cannot connect to peer node
	at com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.java:998)
	at com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:205)
	at com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
	at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
	at org.springframework.erlang.connection.SimpleConnectionFactory.createConnection(SimpleConnectionFactory.java:62)
	... 15 more

Here's the code causing the exception:

public void sendHeartbeat() {                                 
    log.info("sending heartbeat...");

    ConnectionFactory connFactory = config.connectionFactory();
    RabbitBrokerAdmin admin = new RabbitBrokerAdmin(connFactory);

    StringBuilder msg = new StringBuilder();
    msg.append("[Heartbeat] ");

    List<QueueInfo> queues = admin.getQueues();
    for (QueueInfo qi : queues) {

      msg.append("consumers=").append(qi.getConsumers());

    }

    config.rabbitTemplate().convertAndSend("heartbeat-ex", null, msg);
  }

The broker is version 2.0 and has the default settings.
Any ideas?

[Mark Pollack]

Hi,

The issue is that JInterface requires that the string for the host name matches what the rabbit broker reports as its node name. Look at the console output when starting rabbit for 'node ='.

JInterface actually hides the underlying exception related to this handshake protocol, I'll see how to file an issue so the stack trace and underlying exception can better describe what is happening.

In my case the node name is called rabbit@MARK6500 and not rabbit@localhost.

I'm guessing that you are passing 'rabbit@localhost' into the constructor of the connection factory (SingleConnectionFactory or CachingConnectionFactory). If you use the zero arg constructor it will call InetAddress.getLocalHost().getHostName() which will typically match what the rabbit broker is using for its node name.

The node name for rabbit itself can be configured in rabbitmq.conf should you want to use something else.

You can experiment with this yourself with the following lower level code that demonstrates the problem.

OtpSelf self = new OtpSelf("rabbit-monitor");
OtpPeer peer = new OtpPeer("rabbit@localhost");
self.connect(peer);

will fail while

OtpSelf self = new OtpSelf("rabbit-monitor");
OtpPeer peer = new OtpPeer("rabbit@" + InetAddress.getLocalHost().getHostName());
self.connect(peer);

will pass.

Cheers,
Mark

[digitalsanctum]

Mark, thanks for the reply. I did as you suggested and passed a zero arg constructor to the connection factory. I also confirmed that the hostname matches what my local broker reports. However, I'm still seeing the same exception (with the changed hostname):

1:40:16 [ConsumerTimerFactory] ERROR
timer.MethodInvokingTimerTaskFactoryBean  - Invocation of method
'sendHeartbeat' on target class [class
spg.poly.consumer.monitoring.HeartbeatPublisher] failed
org.springframework.erlang.OtpIOException: failed to connect from
'rabbit-spring-monitor' to peer node 'rabbit@dbjaxwslkhgzmy'
       at
org.springframework.erlang.connection.SimpleConnectionFactory.createConnec
tion(SimpleConnectionFactory.java:65)
       at
org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangA
ccessor.java:43)
       at
org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java
:114)
       at
org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemp
late.java:46)
       at
org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(Erlang
Template.java:98)
       at
org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getQueues(RabbitBr
okerAdmin.java:148)
       at
spg.poly.consumer.monitoring.HeartbeatPublisher.sendHeartbeat(HeartbeatPub
lisher.java:33)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
pl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at
org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)
       at
org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodIn
vokingRunnable.java:65)
       at
org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTim
erTask.java:70)
       at java.util.TimerThread.mainLoop(Timer.java:512)
       at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: Cannot connect to peer node
       at
com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.ja
va:998)
       at
com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:
205)
       at
com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
       at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
       at
org.springframework.erlang.connection.SimpleConnectionFactory.createConnec
tion(SimpleConnectionFactory.java:62)
       ... 15 more

[ndpar]

In Rabbit server the node name does not necessarily have the form of 'rabbit@yourhostname'. To find out yours, run

cat /var/log/rabbitmq/startup_log

In the output you will see the node name. Providing correct name fixes 'Cannot connect to peer node' error, but you also need to provide correct cookie to pass authentication. Just to prove the concept I made quick hack (see attachment), after which 'admin.getQueues()' successfully passed.

Now we need to think how to implement it properly.

[digitalsanctum]

I was never able to get this working on windows but I did get it working on mac. Is it possible this is a windows specific bug?

I'm setting host to 'DBJAXWSLKHGZMY' which leads to the exception below. Setting it to 'dbjaxwslkhgzmy' results in the previous exception. Rabbit reports node: 'rabbit@DBJAXWSLKHGZMY' on startup.

Exception in thread "main" org.springframework.erlang.UncategorizedOtpException: com.ericsson.otp.erlang.OtpAuthException: Peer authentication error.
	at org.springframework.erlang.support.ErlangUtils.convertOtpAccessException(ErlangUtils.java:51)
	at org.springframework.erlang.core.ErlangTemplate.convertOtpAccessException(ErlangTemplate.java:142)
	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:121)
	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:46)
	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:98)
	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:292)

[digitalsanctum]

At the prompting of Dave Syer, I tried again using a recent snapshot (1.0.0.BUILD-20101101.013057). The following is the result:

Exception in thread "main" org.springframework.amqp.rabbit.admin.RabbitAdminAuthException: Could not authorise connection to Erlang process. This can happen if the broker is running, but as root or rabbitmq and the current user is not authorised to connect. Try starting the broker again as a different user.
	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:358)
	at spg.dexc.integ.BrokerAdminTest.stopApp(BrokerAdminTest.java:32)
	at spg.dexc.integ.BrokerAdminTest.main(BrokerAdminTest.java:22)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: com.ericsson.otp.erlang.OtpAuthException: Peer authentication error.
	at com.ericsson.otp.erlang.AbstractConnection.recvChallengeAck(AbstractConnection.java:1303)
	at com.ericsson.otp.erlang.AbstractConnection.doConnect(AbstractConnection.java:990)
	at com.ericsson.otp.erlang.AbstractConnection.<init>(AbstractConnection.java:205)
	at com.ericsson.otp.erlang.OtpConnection.<init>(OtpConnection.java:86)
	at com.ericsson.otp.erlang.OtpSelf.connect(OtpSelf.java:219)
	at org.springframework.erlang.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:211)
	at org.springframework.erlang.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:150)
	at org.springframework.erlang.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:133)
	at org.springframework.erlang.support.ErlangAccessor.createConnection(ErlangAccessor.java:43)
	at org.springframework.erlang.core.ErlangTemplate.execute(ErlangTemplate.java:115)
	at org.springframework.erlang.core.ErlangTemplate.executeErlangRpc(ErlangTemplate.java:47)
	at org.springframework.erlang.core.ErlangTemplate.executeAndConvertRpc(ErlangTemplate.java:99)
	at org.springframework.amqp.rabbit.admin.RabbitBrokerAdmin.getStatus(RabbitBrokerAdmin.java:352)
	... 7 more