file is not get created

[SARAL SAXENA]

Hi ,

I have two queues that are montired by seprate queue manager Now when I put 50000 messages those 5000 messages are internally divided into half means each queue holding 2500 messages each..the queue's data looks like these

RODS.DATA.IN 2500 messages (having diiferent queuee manger)
RODS.DATA.IN 2500 messages (having different queue manager)


Now when I try to read the data from both the queues two seprate files get created ,each file having entry of the all 2500 records of a single queue.so this works perfectly, but the issue comes when I put 10,000 records those are internally divided into 5000 but the moment when I try to read them only 1 file of 5000 records get created and the secod file is not created ...!! please guide me ,,,wats wrong..!!


below is the configuration that is the xml I am using ....

		
			
			
	<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> 
		<property name="location">
			<value>config.properties</value>
		</property>
	</bean>			
			
		
	<!-- Websphere MQ connection factory config-->
	<bean id="jmsQueueWriteConnectionFactory"
		class="com.ibm.mq.jms.MQQueueConnectionFactory">
		<property name="queueManager">
			<value>${MQ.QueueManagerWrite}</value>
		</property>
		<property name="hostName">
			<value>${MQ.HostNameWrite}</value>
		</property>
		<property name="port">
			<value>${MQ.Port}</value>
		</property>
		<property name="channel">
			<value>${MQ.Channel}</value>
		</property>
		<property name="transportType">
			<value>${MQ.TransportType}</value>
		</property>
	</bean>
	
	
	<bean id="jmsQueueReadConnectionFactory"
		class="com.ibm.mq.jms.MQQueueConnectionFactory">
		<property name="queueManager">
			<value>${MQ.QueueManagerRead}</value>
		</property>
		<property name="hostName">
			<value>${MQ.HostNameRead}</value>
		</property>
		<property name="port">
			<value>${MQ.Port}</value>
		</property>
		<property name="channel">
			<value>${MQ.Channel}</value>
		</property>
		<property name="transportType">
			<value>${MQ.TransportType}</value>
		</property>
	</bean>	
	
	<bean id="mqConnectionFactory"
		class="org.springframework.jms.connection.SingleConnectionFactory">
		<property name="targetConnectionFactory">
			<ref bean="jmsQueueWriteConnectionFactory" />
		</property>
	
	</bean>
	
	<!-- Jms Template to send messages. Used for Unit Testing -->
	<!--<bean id="jmsTemplate"
		class="org.springframework.jms.core.JmsTemplate">
		<property name="connectionFactory">
			<ref bean="mqConnectionFactory" />
		</property>
		<property name="defaultDestinationName">
			<value>${MQ.DefaultDestinationName}</value>
		</property>
	</bean>

	
	
	-->
	
	
	<!-- Websphere MQ queue config. This config is used only in the absence of jndi  -->
	<bean id="odsRequestQueue_dev1" class="com.ibm.mq.jms.MQQueue">
		<property name="baseQueueManagerName" value="${MQ.BaseQueueManagerName}" />
		<property name="baseQueueName" value="${MQ.BaseQueueName}" />
		<property name="targetClient" value="${MQ.TargetClient}" />
	</bean>
				
		<!-- putting data into the queue commented by saral  -->	
				

   <si:channel id="input">
		
	</si:channel>
			
		 <jms:outbound-channel-adapter id="channel-to-mq"
			channel="input" destination-name="${queueNameWrite}" connection-factory="mqConnectionFactory"  />  
							
			<!-- Reading from the queue commented by saral -->	 		

 	  <jms:message-driven-channel-adapter
			id="mq-message-listner_QManager_4" channel="aggregator-input-channel_1" connection-factory="jmsQueueReadConnectionFactory"
			destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}"   />	
		
		<jms:message-driven-channel-adapter
			id="mq-message-listner_QManager_3" channel="aggregator-input-channel_2" connection-factory="jmsQueueWriteConnectionFactory"
			destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}"   /> 

		

	<!--   <si:gateway id="gateway" service-interface="com.sample.agg.Gateway"
		default-request-channel="aggregator-input-channel">
	</si:gateway> -->
	
	<si:channel id="aggregator-input-channel_1">
		<si:queue capacity="${queueCapacity}" />
	</si:channel>
	
	<si:channel id="aggregator-input-channel_2">
		<si:queue capacity="${queueCapacity}" />
	</si:channel>	
	
	<si:aggregator id="aggregator_1" input-channel="aggregator-input-channel_1" output-channel="aggregator-output-channel_1"
		
		ref="sampleAggregator" method="aggregateMessagaes"  		
		correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy" 
		release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy" 
		message-store="messageStore"
		order="1" 
		send-partial-result-on-expiry="true"		
		send-timeout = "5000"		>
	</si:aggregator>
	
	<si:aggregator id="aggregator_2" input-channel="aggregator-input-channel_2" output-channel="aggregator-output-channel_2"
		
		ref="sampleAggregator" method="aggregateMessagaes"  		
		correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy" 
		release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy" 
		message-store="messageStore"
		order="1" 
		send-partial-result-on-expiry="true"		
		send-timeout = "5000"
		>
	</si:aggregator>
	
	
	<bean id="sampleAggregator" class="com.walgreens.ods.producer.Aggregator" scope="prototype" ></bean>   
	
	<bean id="correlationBean" class="com.walgreens.ods.producer.CorrelationBean" scope="prototype" />
	
	<bean id="releaseStrategyBean" class="com.walgreens.ods.producer.ReleaseStrategyBean" scope="prototype" > 
	<property name="recordLength" value="${recordLength}" />	
	</bean>
	
	 <si:channel id="throwAwayChannel"	>		
	<si:queue capacity="${queueCapacity}" />
	</si:channel> 
	

		<file:outbound-channel-adapter channel="aggregator-output-channel_1"
			directory="${aggregatorOutputDirectoryPathtemp1}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
			   />	
		<file:outbound-channel-adapter channel="aggregator-output-channel_2"
			directory="${aggregatorOutputDirectoryPathtemp2}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
			   />				   

		<!--  <file:outbound-channel-adapter channel="throwAwayChannel"
			directory="${aggregatorOutputDirectoryPath}" filename-generator-expression="'discard-'+headers.getTimestamp()+'.xml'"
			temporary-file-suffix="_swp"   />	-->  
	
	
	<si:channel id="aggregator-output-channel_1">
		<si:queue capacity="${queueCapacity}" />
	</si:channel>
	
	<si:channel id="aggregator-output-channel_2">
		<si:queue capacity="${queueCapacity}" />
	</si:channel>
	
	
	<task:scheduler id="scheduler" />
	
	<!--  
	<task:scheduled-tasks scheduler="scheduler">
		<task:scheduled ref="reaper" method="run" fixed-rate="${reaperSchedulerFixedRate}" />
	</task:scheduled-tasks> 
	
       --> 		

	<bean id="reaper" class="org.springframework.integration.store.MessageGroupStoreReaper">
			<property name="messageGroupStore" ref="messageStore"  />
		 	
		<property name="timeout" value="${reaperTimeOut}" />
		
	</bean>
	
 <bean id="messageStore" class="org.springframework.integration.store.SimpleMessageStore" />
	
 	
	<!--<si:service-activator input-channel="aggregator-output-channel" ref="printerService" method="printString"  ></si:service-activator>-->
	
	<!--  <bean id="printerService" class="com.sample.agg.PrinterService"></bean>-->
 	

	<si:poller max-messages-per-poll="1" id="defaultPoller" default="true">
		<si:interval-trigger interval="${pollerInterval}" />
	</si:poller>
	
<!-- <si:inbound-channel-adapter id="test" ref="exampleReleaseStrategy" method="releaseStrategy" >
<si:poller fixed-rate="5000"/>
</si:inbound-channel-adapter> --> 	
	

</beans>

and the properties file is .....

# Spring bean values

#explicitly reading from seprate , queue name is decided after observing the messages  in queuezee queue by saral
queueNameWrite=RODS.DATA.OUT
queueNameRead=RODS.DATA.IN
#queueNameRead=RODS.DATA.OUT


MQ.QueueManagerWrite=DCC2_QM_3
MQ.QueueManagerRead=DCC2_QM_4

MQ.HostNameWrite=dinfmq03
MQ.HostNameRead=dinfmq04

MQ.Port=11421
MQ.Channel=RODS.APP.SVRCONN
MQ.TransportType=1
MQ.DefaultDestinationName=RODS.DATA.OUT
#MQ.DefaultDestinationName=RODS.APP.DATA.IN
MQ.BaseQueueManagerName=RODS.APP.SVRCONN
MQ.BaseQueueName=RODS.DATA.OUT
MQ.TargetClient=1

concurrent-consumers=10
queueCapacity=100
recordLength=5000
#aggregatorOutputDirectoryPath=C:\\ODS_DEV_DIR\\temp\\
aggregatorOutputDirectoryPathtemp1=C:\\ODS_DEV_DIR\\temp1\\
aggregatorOutputDirectoryPathtemp2=C:\\ODS_DEV_DIR\\temp2\\

outputFileNameExpression='output-'+headers.getTimestamp()+'.xml'
reaperSchedulerFixedRate=10000
#reaperTimeOut=30
reaperTimeOut=500
pollerInterval=50
Header.Source.Key=eiaHdrSource
Header.Destination.Key=eiaHdrDest
Header.DataId.Key=eiaHdrDataId
Header.Route.Key=eiaHdrRoute
	
Header.Source.Value=RODS
Header.Destination.Value=RODS
Header.DataId.Value=DATA
Header.Route.Value=LOCAL

pLesae guide me why the second file is not created....!!

[SARAL SAXENA]

Hi Guys,

Please guide me ...As I am stuck up..I am doing R&D on this but couldn't reach to the satisfication point..!!

[Mark Fisher]

Like I said yesterday, do not reply to your own posts because you didn't receive a reply yet.

[Gary Russell]

I suggest you turn on debug logging; make sure you include the thread (%t) and timestamp (%d) in your log spec; you should be able to see what's going on.

I suggest you first do it with a few (one or two) messages in each queue, to get familiar with what the logs show you.

Then run it with your failing test and look at the logs.

[SARAL SAXENA]

I suggest you turn on debug logging; make sure you include the thread (%t) and timestamp (%d) in your log spec; you should be able to see what's going on.

I suggest you first do it with a few (one or two) messages in each queue, to get familiar with what the logs show you.

Then run it with your failing test and look at the logs.

Hi Gary Russell,
Yeah I have configured log4j and logging channel adapter and regularly checking the logs , in the case when I put 5000 records it works perfectly and 2 files containing 2500 records each is being created but in case when I put 10,000 records then In that case only one file of 5000 records is created but the second file containing the remaining 5000 records is not created..!!!

[Gary Russell]

Right; you already said that; but can you not figure out the difference between the two tests when looking at the logs?

[SARAL SAXENA]

Right; you already said that; but can you not figure out the difference between the two tests when looking at the logs?

well the logs for the second are not being created since the messages from the queue for the second one , that is all the messages are written in file and queue is being emptied but the file is not created...messages are lost ...and this is the case of worry...!!..please check my configuration it mite give any hint ...

<bean class="org.springframework.beans.factory.config.Pr opertyPlaceholderConfigurer">
<property name="location">
<value>config.properties</value>
</property>
</bean>


<!-- Websphere MQ connection factory config-->
<bean id="jmsQueueWriteConnectionFactory"
class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="queueManager">
<value>${MQ.QueueManagerWrite}</value>
</property>
<property name="hostName">
<value>${MQ.HostNameWrite}</value>
</property>
<property name="port">
<value>${MQ.Port}</value>
</property>
<property name="channel">
<value>${MQ.Channel}</value>
</property>
<property name="transportType">
<value>${MQ.TransportType}</value>
</property>
</bean>


<bean id="jmsQueueReadConnectionFactory"
class="com.ibm.mq.jms.MQQueueConnectionFactory">
<property name="queueManager">
<value>${MQ.QueueManagerRead}</value>
</property>
<property name="hostName">
<value>${MQ.HostNameRead}</value>
</property>
<property name="port">
<value>${MQ.Port}</value>
</property>
<property name="channel">
<value>${MQ.Channel}</value>
</property>
<property name="transportType">
<value>${MQ.TransportType}</value>
</property>
</bean>

<bean id="mqConnectionFactory"
class="org.springframework.jms.connection.SingleCo nnectionFactory">
<property name="targetConnectionFactory">
<ref bean="jmsQueueWriteConnectionFactory" />
</property>

</bean>

<!-- Jms Template to send messages. Used for Unit Testing -->
<!--<bean id="jmsTemplate"
class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory">
<ref bean="mqConnectionFactory" />
</property>
<property name="defaultDestinationName">
<value>${MQ.DefaultDestinationName}</value>
</property>
</bean>



-->


<!-- Websphere MQ queue config. This config is used only in the absence of jndi -->
<bean id="odsRequestQueue_dev1" class="com.ibm.mq.jms.MQQueue">
<property name="baseQueueManagerName" value="${MQ.BaseQueueManagerName}" />
<property name="baseQueueName" value="${MQ.BaseQueueName}" />
<property name="targetClient" value="${MQ.TargetClient}" />
</bean>

<!-- putting data into the queue commented by saral -->


<si:channel id="input">

</si:channel>

<jms:outbound-channel-adapter id="channel-to-mq"
channel="input" destination-name="${queueNameWrite}" connection-factory="mqConnectionFactory" />

<!-- Reading from the queue commented by saral -->

<jms:message-driven-channel-adapter
id="mq-message-listner_QManager_4" channel="aggregator-input-channel_1" connection-factory="jmsQueueReadConnectionFactory"
destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}" />

<jms:message-driven-channel-adapter
id="mq-message-listner_QManager_3" channel="aggregator-input-channel_2" connection-factory="jmsQueueWriteConnectionFactory"
destination-name="${queueNameRead}" concurrent-consumers="${concurrent-consumers}" />



<!-- <si:gateway id="gateway" service-interface="com.sample.agg.Gateway"
default-request-channel="aggregator-input-channel">
</si:gateway> -->

<si:channel id="aggregator-input-channel_1">
<si:queue capacity="${queueCapacity}" />
</si:channel>

<si:channel id="aggregator-input-channel_2">
<si:queue capacity="${queueCapacity}" />
</si:channel>

<si:aggregator id="aggregator_1" input-channel="aggregator-input-channel_1" output-channel="aggregator-output-channel_1"

ref="sampleAggregator" method="aggregateMessagaes"
correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy"
release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy"
message-store="messageStore"
order="1"
send-partial-result-on-expiry="true"
send-timeout = "5000" >
</si:aggregator>

<si:aggregator id="aggregator_2" input-channel="aggregator-input-channel_2" output-channel="aggregator-output-channel_2"

ref="sampleAggregator" method="aggregateMessagaes"
correlation-strategy="correlationBean" correlation-strategy-method="correlationStrategy"
release-strategy="releaseStrategyBean" release-strategy-method="releaseStrategy"
message-store="messageStore"
order="1"
send-partial-result-on-expiry="true"
send-timeout = "5000"
>
</si:aggregator>


<bean id="sampleAggregator" class="com.walgreens.ods.producer.Aggregator" scope="prototype" ></bean>

<bean id="correlationBean" class="com.walgreens.ods.producer.CorrelationBean" scope="prototype" />

<bean id="releaseStrategyBean" class="com.walgreens.ods.producer.ReleaseStrategyB ean" scope="prototype" >
<property name="recordLength" value="${recordLength}" />
</bean>

<si:channel id="throwAwayChannel" >
<si:queue capacity="${queueCapacity}" />
</si:channel>


<file:outbound-channel-adapter channel="aggregator-output-channel_1"
directory="${aggregatorOutputDirectoryPathtemp1}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
/>
<file:outbound-channel-adapter channel="aggregator-output-channel_2"
directory="${aggregatorOutputDirectoryPathtemp2}" filename-generator-expression="${outputFileNameExpression}" temporary-file-suffix="_swp"
/>

<!-- <file:outbound-channel-adapter channel="throwAwayChannel"
directory="${aggregatorOutputDirectoryPath}" filename-generator-expression="'discard-'+headers.getTimestamp()+'.xml'"
temporary-file-suffix="_swp" /> -->


<si:channel id="aggregator-output-channel_1">
<si:queue capacity="${queueCapacity}" />
</si:channel>

<si:channel id="aggregator-output-channel_2">
<si:queue capacity="${queueCapacity}" />
</si:channel>


<task:scheduler id="scheduler" />

<!--
<task:scheduled-tasks scheduler="scheduler">
<task:scheduled ref="reaper" method="run" fixed-rate="${reaperSchedulerFixedRate}" />
</task:scheduled-tasks>

-->

<bean id="reaper" class="org.springframework.integration.store.Messa geGroupStoreReaper">
<property name="messageGroupStore" ref="messageStore" />

<property name="timeout" value="${reaperTimeOut}" />

</bean>

<bean id="messageStore" class="org.springframework.integration.store.Simpl eMessageStore" />


<!--<si:service-activator input-channel="aggregator-output-channel" ref="printerService" method="printString" ></si:service-activator>-->

<!-- <bean id="printerService" class="com.sample.agg.PrinterService"></bean>-->


<sioller max-messages-per-poll="1" id="defaultPoller" default="true">
<si:interval-trigger interval="${pollerInterval}" />
</sioller>

<!-- <si:inbound-channel-adapter id="test" ref="exampleReleaseStrategy" method="releaseStrategy" >
<sioller fixed-rate="5000"/>
</si:inbound-channel-adapter> -->


</beans>

[Gary Russell]

Please STOP posting the same stuff over and over again - you already posted your config above.

If the messages are being consumed, they are going somewhere.

Please CAREFULLY look through the log again, and follow a message from each queue from start to finish be sure to have %t in your log spec,

Is it possible that ...PathTemp1 and ...PathTemp2 point to the same directory and the second file overwrites the first one?

[SARAL SAXENA]

Please STOP posting the same stuff over and over again - you already posted your config above.

If the messages are being consumed, they are going somewhere.

Please CAREFULLY look through the log again, and follow a message from each queue from start to finish be sure to have %t in your log spec,

Is it possible that ...PathTemp1 and ...PathTemp2 point to the same directory and the second file overwrites the first one?

Hi Gary,

Yeah I will take care of that, well I have tried this option also by creating two seprate directories temp1 and temp2 and in that case file inside temp1 file is created containing 5000 records but in the folder temp2 the remaining 5000 records file is not created ....this doesn't works when you put 10,000 messages and when you try to consume them and writing them into the file , but the thing that I am amazed is that it works perfectly in the case when you put only 5000 messges so some sort of load balancing is required when you increase the quantity of messages and then try to read them..!!