Transactional POP3 Mailbox access with Spring Integration

[MachineCode]

That seems to have solved the issue thanks. I gave up waiting for RC2 and have tried 2.2.0.BUILD-SNAPSHOT from 20121012 instead, which I got from the artifactory repositories.

[MachineCode]

I couldn't get the transactional handling of emails (pop3 in my case) working with 2.2.0 Build Snapshot 20121012, despite the fix to the stuck poller being in that version. I looked at the source code and the changes are not in that version or RC1 version. However, as reported by another user in the thread, the 2.2.0 M4 build works well.

My only outstanding issue is that the filter does not work (emails still disappear from the inbox) - maybe it's a pop3 issue - or my set up.

[Gary Russell]

RC2 was released Friday - sorry for the delay, but we had to wait for another dependency to be released.

The transaction stuff for mail was changed between M4 and RC2 - we had to make it orthogonal to the existing delete and mark read options - in order to maintain backwards compatibility for folks currently using transactions but expecting the delete to occur immediately.

If you want to synchronize message deletion with a transaction, you need to set should-delete-messages to false, and add some code to do the deletion as part of transaction synchronization...

http://static.springsource.org/sprin.../#mail-tx-sync

[MachineCode]

Hi. Thanks for the advice on RC2 and transactions. I've ditched M4 based on your advice. Trouble is I've hit a problem already and am wondering whether to raise as a Jira bug...

With RC2, should-delete-messages=false and a trans sync element, I get this problem.

[] DEBUG 113050 [24 Oct 2012 14:30:22,111] AbstractPlatformTransactionManager - Creating new transaction with name [org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT,timeout_10000
[] DEBUG 113050 [24 Oct 2012 14:30:22,111] TransactionSynchronizationManager - Initializing transaction synchronization
[] DEBUG 113050 [24 Oct 2012 14:30:22,111] TransactionAspectSupport - Getting transaction for [org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call]
[] DEBUG 113050 [24 Oct 2012 14:30:22,111] TransactionSynchronizationManager - Bound value [org.springframework.integration.transaction.IntegrationResourceHolder@3a0c2cb] for key [org.springframework.integration.mail.MailReceivingMessageSource@33e4034] to
thread [task-scheduler-5]
[] DEBUG 113050 [24 Oct 2012 14:30:22,111] TransactionSynchronizationManager - Retrieved value [org.springframework.integration.transaction.IntegrationResourceHolder@3a0c2cb] for key [org.springframework.integration.mail.MailReceivingMessageSource@33e4034]
 bound to thread [task-scheduler-5]
DEBUG POP3: connecting to host "nn.nnn.n.nn", port 110, isSSL false
S: +OK Microsoft Exchange Server 2003 POP3 server version 6.5.7638.1 (xxxxxxxxxx.xxxxxxxxx.xxxxx.xxxx.xxxx) ready.
C: USER xxxxxxx
S: +OK
C: PASS xxxxxxxxxxxxxx
S: +OK User successfully logged on.
[] DEBUG 113081 [24 Oct 2012 14:30:22,142] AbstractMailReceiver - opening folder [pop3://xxxxxxxxx:*****@nn.nnn.n.nn/inbox]
C: STAT
S: +OK 1 46952
[] INFO  113081 [24 Oct 2012 14:30:22,142] AbstractMailReceiver - attempting to receive mail from folder [inbox]
C: NOOP
S: +OK
[] DEBUG 113081 [24 Oct 2012 14:30:22,142] AbstractMailReceiver - found 1 new messages
C: TOP 1 0
S: +OK
Received: by xxx99999
        id <01CDB1EB.A960F132@xxx99999>; Wed, 24 Oct 2012 14:29:58 +0100
Content-class: urn:content-classes:message
Subject: HTML, one attachment, RC2 test 14:29
Date: Wed, 24 Oct 2012 14:29:58 +0100
Message-ID: <51E87F6C6074754DB9999999A293CBF3518183@xxx99999>
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="----_=_NextPart_001_01XXX1EB.A960F132"
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
Thread-Topic: HTML, one attachment, RC2 test 14:29
Thread-Index: Ac2w8BmTZ/2ABx6ZQPqvedXnPVyVDQ==
X-MimeOLE: Produced By Microsoft Exchange V6.5
From: "xxxxxxx" <xxxxxxx@xxxx.xxx.xxxx.xx>
To: "xxxxxxxx" <xxxxxxxx@xxxx.xxxx.xxx.xx>

.
C: LIST 1
S: +OK 1 46952
[] DEBUG 113128 [24 Oct 2012 14:30:22,189] AbstractMailReceiver - Received 1 messages
[] DEBUG 113143 [24 Oct 2012 14:30:22,204] AbstractMailReceiver - USER flags are not supported by this mail server. Flagging message with system flag

This next three lines are key to the problem I think - it is quitting before the email attachment is read - there is no "C: RETR 1" entry like with the M4 version:

C: NOOP
S: +OK
C: QUIT
S: +OK Microsoft Exchange Server 2003 POP3 server version 6.5.7638.1 signing off.
[] DEBUG 113143 [24 Oct 2012 14:30:22,204] MailReceivingMessageSource - received mail message [com.sun.mail.pop3.POP3Message@3a186fb]
[] DEBUG 113159 [24 Oct 2012 14:30:22,220] SourcePollingChannelAdapter - Poll resulted in Message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1351085422220, id=d6fba356-a6d4-4124-9bf2-f62833e64d02}]
[] DEBUG 113159 [24 Oct 2012 14:30:22,220] AbstractMessageChannel$ChannelInterceptorList - preSend on channel 'receiveEmailChannel', message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1351085422220, id=d6fba356-a6d4-4124-9bf2-f6283
3e64d02}]
[] DEBUG 113159 [24 Oct 2012 14:30:22,220] AbstractMessageChannel$ChannelInterceptorList - preSend on channel 'logger', message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1351085422220, id=d6fba356-a6d4-4124-9bf2-f62833e64d02}]
[] DEBUG 113175 [24 Oct 2012 14:30:22,236] AbstractMessageHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1351085422220, id=d6fba356-a6d4-4124-9bf2-f628
33e64d02}]
[] DEBUG 113190 [24 Oct 2012 14:30:22,251] LoggingHandler - com.sun.mail.pop3.POP3Message@3a186fb
[] DEBUG 113190 [24 Oct 2012 14:30:22,251] AbstractMessageChannel$ChannelInterceptorList - postSend (sent=true) on channel 'logger', message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1351085422220, id=d6fba356-a6d4-4124-9bf2-f6283
3e64d02}]
[] DEBUG 113190 [24 Oct 2012 14:30:22,251] AbstractMessageHandler - ServiceActivator for [org.springframework.integration.handler.MethodInvokingMessageProcessor@22ef823] received message: [Payload=com.sun.mail.pop3.POP3Message@3a186fb][Headers={timestamp=1

This results in an exception when trying to retreive the input stream of the attachment :

javax.mail.MessagingException: No inputstream from datasource;
  nested exception is:
        java.lang.IllegalStateException: Folder is not Open
        at javax.mail.internet.MimeMultipart.parsebm(MimeMultipart.java:732)
        at javax.mail.internet.MimeMultipart.parse(MimeMultipart.java:466)
        at javax.mail.internet.MimeMultipart.getCount(MimeMultipart.java:242)
        ...

With should-delete-messages set to true in integration.xml the stream is retrieved however, but the message gets deleted and not by any transaction sync code :-(

[] DEBUG 358193 [24 Oct 2012 14:18:09,695] AbstractMailReceiver - USER flags are...
C: RETR 1
S: +OK

[oleg.zhurakousky]

I am looking into that, but could you please post your configuration to make sure we are looking t the same problem.

[MachineCode]

Thanks for the quick response!

Hopefully this is the key part of the configuration you are after from integration.xml. I added the trans sync parts based on the recent reply to using RC2 instead of M4 but with or without the sync I get the same issue. Let me know if you need any more info.

<util:properties id="javaMailProperties">
    <prop key="mail.pop3.socketFactory.fallback">false</prop>
    <prop key="mail.debug">true</prop>
    <prop key="mail.pop3.rsetbeforequit">false</prop>
</util:properties>

<mail:inbound-channel-adapter id="mailAdapter"
	store-uri="pop3://xxxxxx:xxxxxxxxxxxxxxxx@99.999.9.99/inbox"
	channel="receiveEmailChannel"
	should-delete-messages="false"
	auto-startup="true"
	java-mail-properties="javaMailProperties">
	<int:poller fixed-delay="20" max-messages-per-poll="20"
		time-unit="SECONDS">
		<int:transactional
			synchronization-factory="syncFactory"
			isolation="DEFAULT"
			propagation="REQUIRED"
			read-only="false"
			timeout="10000"/>
	</int:poller>
</mail:inbound-channel-adapter>

<int:channel id="receiveEmailChannel">
	<int:interceptors>
		<int:wire-tap channel="logger" />
	</int:interceptors>
</int:channel>

<int:logging-channel-adapter id="logger"
	level="DEBUG" />

<int:service-activator input-channel="receiveEmailChannel"
	ref="emailAdapterService" method="processEmail" />
	
<int:transaction-synchronization-factory id="syncFactory">
	<int:after-commit expression="@syncProcessor.markMimeMessageAsDeleted(payload)"/>
</int:transaction-synchronization-factory>

<bean id="syncProcessor"
	class="xx.xxx.xxxxxxxxxxx.xxxx.MailTransactionSynchronizationProcessor" />

[oleg.zhurakousky]

Yeah, this is enough, give me few hours

[MachineCode]

OK, thanks for looking at it. I finish work in 30 mins, at 4pm here in the UK so no rush to reply. Thanks again for responding quickly.

[oleg.zhurakousky]

I just remembered; The javax.mail.Message arrives with a handle to its Folder which is indeed closed which means you have to open it (if you need and in your case you do), do whatever you need and close it again.
Here is my example:

public void postProcess(Message mailMessage) throws Exception{
		mailMessage.getFolder().open(Folder.READ_WRITE);
		mailMessage.setFlag(Flags.Flag.DELETED, true);
		mailMessage.getFolder().close(true);
		System.out.println("Deleted");
}

[oleg.zhurakousky]

Also, look at the underlying JIRA for this one https://jira.springsource.org/browse/INT-1819 you can see a sample code there as well.
Let us know how it goes